# Neural Network Compression Framework (NNCF)

[Key Features](#key-features) •
[Installation](#installation-guide) •
[Documentation](#documentation) •
[Usage](#usage) •
[Tutorials and Samples](#demos-tutorials-and-samples) •
[Third-party integration](#third-party-repository-integration) •
[Model Zoo](./docs/ModelZoo.md)

[![GitHub Release](https://img.shields.io/github/v/release/openvinotoolkit/nncf?color=green)](https://github.com/openvinotoolkit/nncf/releases)
[![Website](https://img.shields.io/website?up_color=blue&up_message=docs&url=https%3A%2F%2Fdocs.openvino.ai%2Flatest%2Fopenvino_docs_model_optimization_guide.html)](https://docs.openvino.ai/latest/openvino_docs_model_optimization_guide.html)
[![Apache License Version 2.0](https://img.shields.io/badge/license-Apache_2.0-green.svg)](LICENSE)
[![PyPI Downloads](https://static.pepy.tech/badge/nncf)](https://pypi.org/project/nncf/)

Neural Network Compression Framework (NNCF) provides a suite of post-training and training-time algorithms for neural networks inference optimization in [OpenVINO™](https://docs.openvino.ai) with minimal accuracy drop.

NNCF is designed to work with models from [PyTorch](https://pytorch.org/), [TensorFlow](https://www.tensorflow.org/), [ONNX](https://onnx.ai/) and [OpenVINO™](https://docs.openvino.ai/latest/home.html).

NNCF provides [samples](#demos-tutorials-and-samples) that demonstrate the usage of compression algorithms for different use cases and models. See compression results achievable with the NNCF-powered samples at [Model Zoo page](./docs/ModelZoo.md).

The framework is organized as a Python\* package that can be built and used in a standalone mode. The framework
architecture is unified to make it easy to add different compression algorithms for both PyTorch and TensorFlow deep
learning frameworks.

## Key Features

### Post-Training Compression Algorithms

| Compression algorithm                                                       |OpenVINO|PyTorch|   TensorFlow   |     ONNX       |
|:----------------------------------------------------------------------------| :---: | :---: |:--------:|:------------------:|
| [Post-Training Quantization](./docs/usage/post_training_compression/post_training_quantization/Usage.md) | Supported | Supported |Supported| Supported |
| [Weights Compression](./docs/usage/post_training_compression/weights_compression/Usage.md) | Supported | Supported |Not supported| Not supported |

### Training-Time Compression Algorithms

|Compression algorithm|PyTorch|TensorFlow|
| :--- | :---: | :---: |
|[Quantization Aware Training](./docs/usage/training_time_compression/quantization_aware_training/Usage.md) | Supported | Supported |
|[Mixed-Precision Quantization](./docs/usage/training_time_compression/other_algorithms/LegacyQuantization.md#mixed-precision-quantization) | Supported | Not supported |
|[Sparsity](./docs/usage/training_time_compression/other_algorithms/Sparsity.md) | Supported | Supported |
|[Filter pruning](./docs/usage/training_time_compression/other_algorithms/Pruning.md) | Supported | Supported |
|[Movement pruning](./nncf/experimental/torch/sparsity/movement/MovementSparsity.md) | Experimental | Not supported |

- Automatic, configurable model graph transformation to obtain the compressed model.
  > **NOTE**: Limited support for TensorFlow models. The models created using Sequential or Keras Functional API are only supported.
- Common interface for compression methods.
- GPU-accelerated layers for faster compressed model fine-tuning.
- Distributed training support.
- Git patch for prominent third-party repository ([huggingface-transformers](https://github.com/huggingface/transformers)) demonstrating the process of integrating NNCF into custom training pipelines
- Seamless combination of pruning, sparsity and quantization algorithms. Please refer to [optimum-intel](https://github.com/huggingface/optimum-intel/tree/main/examples/openvino) for examples of
joint (movement) pruning, quantization and distillation (JPQD), end-to-end from NNCF optimization to compressed OpenVINO IR.
- Exporting PyTorch compressed models to ONNX\* checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use with [OpenVINO™ toolkit](https://docs.openvino.ai).
- Support for [Accuracy-Aware model training](./docs/usage/training_time_compression/other_algorithms/Usage.md#accuracy-aware-model-training) pipelines via the [Adaptive Compression Level Training](./docs/accuracy_aware_model_training/AdaptiveCompressionLevelTraining.md) and [Early Exit Training](./docs/accuracy_aware_model_training/EarlyExitTraining.md).

## Documentation

This documentation covers detailed information about NNCF algorithms and functions needed for the contribution to NNCF.

The latest user documentation for NNCF is available [here](https://docs.openvino.ai/latest/openvino_docs_model_optimization_guide.html).

NNCF API documentation can be found [here](https://openvinotoolkit.github.io/nncf/autoapi/nncf/).

## Usage

### Post-Training Quantization

The NNCF PTQ is the simplest way to apply 8-bit quantization. To run the algorithm you only need your model and a small (~300 samples) calibration dataset.

[OpenVINO](https://github.com/openvinotoolkit/openvino) is the preferred backend to run PTQ with, and PyTorch, TensorFlow and ONNX are also supported.

OpenVINO

```python
import nncf
import openvino.runtime as ov
import torch
from torchvision import datasets, transforms

# Instantiate your uncompressed model
model = ov.Core().read_model("/model_path")

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)

# Step 1: Initialize transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
```

PyTorch

```python
import nncf
import torch
from torchvision import datasets, models

# Instantiate your uncompressed model
model = models.mobilenet_v2()

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset)

# Step 1: Initialize the transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)

```

**NOTE**  In case the Post-Training Quantization algorithm could not reach quality requirements you can fine-tune the quantized pytorch model. Example of the Quantization-Aware training pipeline for a pytorch model could be found [here](examples/quantization_aware_training/torch/resnet18/README.md).

TensorFlow

```python
import nncf
import tensorflow as tf
import tensorflow_datasets as tfds

# Instantiate your uncompressed model
model = tf.keras.applications.MobileNetV2()

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = tfds.load("/path", split="validation",
                        shuffle_files=False, as_supervised=True)

# Step 1: Initialize transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(val_dataset, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
```

ONNX

```python
import onnx
import nncf
import torch
from torchvision import datasets

# Instantiate your uncompressed model
onnx_model = onnx.load_model("/model_path")

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)

# Step 1: Initialize transformation function
input_name = onnx_model.graph.input[0].name
def transform_fn(data_item):
    images, _ = data_item
    return {input_name: images.numpy()}

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(onnx_model, calibration_dataset)
```

[//]: # (NNCF provides full  [samples](#post-training-quantization-samples), which demonstrate Post-Training Quantization usage for PyTorch, TensorFlow, ONNX, OpenVINO.)

### Training-Time Quantization

Below is an example of Accuracy Aware Quantization pipeline where model weights and compression parameters may be fine-tuned to achieve a higher accuracy.

PyTorch

```python
import nncf
import torch
from torchvision import datasets, models

# Instantiate your uncompressed model
model = models.mobilenet_v2()

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset)

# Step 1: Initialize the transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)

# Now use compressed_model as a usual torch.nn.Module
# to fine-tune compression parameters along with the model weights

# Save quantization modules and the quantized model parameters
checkpoint = {
    'state_dict': model.state_dict(),
    'nncf_config': model.nncf.get_config(),
    ... # the rest of the user-defined objects to save
}
torch.save(checkpoint, path_to_checkpoint)

# ...

# Load quantization modules and the quantized model parameters
resuming_checkpoint = torch.load(path_to_checkpoint)
nncf_config = resuming_checkpoint['nncf_config']
state_dict = resuming_checkpoint['state_dict']

quantized_model = nncf.torch.load_from_config(model, nncf_config, example_input)
model.load_state_dict(state_dict)
# ... the rest of the usual PyTorch-powered training pipeline
```

### Training-Time Compression

Below is an example of Accuracy Aware RB Sparsification pipeline where model weights and compression parameters may be fine-tuned to achieve a higher accuracy.

PyTorch

```python
import torch
import nncf.torch  # Important - must be imported before any other external package that depends on torch

from nncf import NNCFConfig
from nncf.torch import create_compressed_model, register_default_init_args

# Instantiate your uncompressed model
from torchvision.models.resnet import resnet50
model = resnet50()

# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_imagenet_rb_sparsity.json")

# Provide data loaders for compression algorithm initialization, if necessary
import torchvision.datasets as datasets
representative_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
init_loader = torch.utils.data.DataLoader(representative_dataset)
nncf_config = register_default_init_args(nncf_config, init_loader)

# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)

# Now use compressed_model as a usual torch.nn.Module
# to fine-tune compression parameters along with the model weights

# ... the rest of the usual PyTorch-powered training pipeline

# Export to ONNX or .pth when done fine-tuning
compression_ctrl.export_model("compressed_model.onnx")
torch.save(compressed_model.state_dict(), "compressed_model.pth")
```

**NOTE (PyTorch)**: Due to the way NNCF works within the PyTorch backend, `import nncf` must be done before any other import of `torch` in your package _or_ in third-party packages that your code utilizes, otherwise the compression may be applied incompletely.

Tensorflow

```python
import tensorflow as tf

from nncf import NNCFConfig
from nncf.tensorflow import create_compressed_model, register_default_init_args

# Instantiate your uncompressed model
from tensorflow.keras.applications import ResNet50
model = ResNet50()

# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_imagenet_rb_sparsity.json")

# Provide dataset for compression algorithm initialization
representative_dataset = tf.data.Dataset.list_files("/path/*.jpeg")
nncf_config = register_default_init_args(nncf_config, representative_dataset, batch_size=1)

# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)

# Now use compressed_model as a usual Keras model
# to fine-tune compression parameters along with the model weights

# ... the rest of the usual TensorFlow-powered training pipeline

# Export to Frozen Graph, TensorFlow SavedModel or .h5  when done fine-tuning
compression_ctrl.export_model("compressed_model.pb", save_format="frozen_graph")
```

For a more detailed description of NNCF usage in your training code, see [this tutorial](docs/usage/training_time_compression/other_algorithms/Usage.md).

## Demos, Tutorials and Samples

For a quicker start with NNCF-powered compression, try sample notebooks and scripts presented below.

### Jupyter* Notebook Tutorials and Demos

A collection of ready-to-run Jupyter* notebooks tutorials and demos are available to explain and display NNCF compression algorithms for optimizing models for inference with the OpenVINO Toolkit.

| Notebook Tutorial Name                                                                                |                                  Compression Algorithm                                  |  Backend   |               Domain                |
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------:|:----------:|:-----------------------------------:|
| [BERT Quantization](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/language-quantize-bert)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/language-quantize-bert/language-quantize-bert.ipynb) |                               Post-Training Quantization                                |  OpenVINO  |                 NLP                 |
| [MONAI Segmentation Model Quantization](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/ct-segmentation-quantize)
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2Fct-segmentation-quantize%2Fct-scan-live-inference.ipynb)     |                               Post-Training Quantization                                |  OpenVINO  |            Segmentation             |
| [PyTorch Model Quantization](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pytorch-post-training-quantization-nncf)                                                                                |                               Post-Training Quantization                                |  PyTorch   |        Image Classification         |
| [Quantization with Accuracy Control](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/quantizing-model-with-accuracy-control)                                                                                |                    Post-Training Quantization with Accuracy Control                     |  OpenVINO  | Speech-to-Text,
Object Detection |
| [PyTorch Training-Time Compression](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/pytorch-quantization-aware-training)                                                                                |                                Training-Time Compression                                |  PyTorch   |        Image Classification         |
| [TensorFlow Training-Time Compression](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/tensorflow-quantization-aware-training)                                                                                |                                Training-Time Compression                                | Tensorflow |        Image Classification         |
| [Joint Pruning, Quantization and Distillation for BERT](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/sparsity-optimization)                                                                                |                      Joint Pruning, Quantization and Distillation                       |  OpenVINO  |                 NLP                 |

Below is a list of notebooks demonstrating OpenVINO conversion and inference together with NNCF compression for models from various domains.

| Demo Model                                                                                |               Compression Algorithm               |  Backend  |                                Domain                                |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------:|:---------:|:--------------------------------------------------------------------:|
| [YOLOv8](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov8-optimization)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/yolov8-optimization/yolov8-object-detection.ipynb)            |            Post-Training Quantization             | OpenVINO  |  Object Detection,
KeyPoint Detection,
Instance Segmentation   |
| [YOLOv7](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov7-optimization)                                                                                |            Post-Training Quantization             | OpenVINO  |                           Object Detection                           |
| [EfficientSAM](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/efficient-sam)                                                                                |            Post-Training Quantization             | OpenVINO  |                          Image Segmentation                          |
| [Segment Anything Model](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/segment-anything)                                                                                |            Post-Training Quantization             | OpenVINO  |                          Image Segmentation                          |
| [OneFormer](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/oneformer-segmentation)                                                                                |            Post-Training Quantization             | OpenVINO  |                          Image Segmentation                          |
| [InstructPix2Pix](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/instruct-pix2pix-image-editing)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Image-to-Image                            |
| [CLIP](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/clip-zero-shot-image-classification)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Image-to-Text                             |
| [BLIP](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/blip-visual-language-processing)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Image-to-Text                             |
| [Segmind-VegaRT](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/stable-diffusion-xl/segmind-vegart.ipynb)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Text-to-Image                             |
| [Latent Consistency Model](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/latent-consistency-models-image-generation)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Text-to-Image                             |
| [Würstchen](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/wuerstchen-image-generation)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Text-to-Image                             |
| [ControlNet QR Code Monster](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/qrcode-monster)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Text-to-Image                             |
| [SDXL-turbo](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/sdxl-turbo)                                                                                |            Post-Training Quantization             | OpenVINO  |                   Text-to-Image,
Image-to-Image                   |
| [ImageBind](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/image-bind)                                                                                |            Post-Training Quantization             | OpenVINO  |                        Multi-Modal Retrieval                         |
| [Distil-Whisper](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/distil-whisper-asr)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Speech-to-Text                            |
| [Whisper](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/whisper-subtitles-generation)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/whisper-subtitles-generation/whisper-convert.ipynb) |            Post-Training Quantization             | OpenVINO  |                            Speech-to-Text                            |
| [MMS Speech Recognition](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/mms-massively-multilingual-speech)                                                                                |            Post-Training Quantization             | OpenVINO  |                            Speech-to-Text                            |
| [Grammar Error Correction](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/grammar-correction)                                                                                |            Post-Training Quantization             | OpenVINO  |                       NLP, Grammar Correction                        |
| [LLM Instruction Following](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-question-answering)                                                                                |                Weight Compression                 | OpenVINO  |                      NLP, Instruction Following                      |
| [Dolly 2.0](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/dolly-2-instruction-following)                                                                                |                Weight Compression                 | OpenVINO  |                      NLP, Instruction Following                      |
| [Stable-Zephyr-3b](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/stable-zephyr-3b-chatbot)                                                                                |                Weight Compression                 | OpenVINO  |                            NLP, Chat Bot                             |
| [LLM Chat Bots](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-chatbot)                                                                                |                Weight Compression                 | OpenVINO  |                            NLP, Chat Bot                             |

### Post-Training Quantization Examples

Compact scripts demonstrating quantization and corresponding inference speed boost:

| Example Name                                                                                |              Compression Algorithm               |  Backend   |         Domain         |
|:-----------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------:|:----------:|:----------------------:|
| [OpenVINO MobileNetV2](./examples/post_training_quantization/openvino/mobilenet_v2/README.md)                                            |            Post-Training Quantization            |  OpenVINO  |  Image Classification  |
| [OpenVINO YOLOv8](./examples/post_training_quantization/openvino/yolov8/README.md)                                                       |            Post-Training Quantization            |  OpenVINO  |    Object Detection    |
| [OpenVINO YOLOv8 QwAС](./examples/post_training_quantization/openvino/yolov8_quantize_with_accuracy_control/README.md)                   | Post-Training Quantization with Accuracy Control |  OpenVINO  |    Object Detection    |
| [OpenVINO Anomaly Classification](./examples/post_training_quantization/openvino/anomaly_stfpm_quantize_with_accuracy_control/README.md) | Post-Training Quantization with Accuracy Control |  OpenVINO  | Anomaly Classification |
| [PyTorch MobileNetV2](./examples/post_training_quantization/torch/mobilenet_v2/README.md)                                                |            Post-Training Quantization            |  PyTorch   |  Image Classification  |
| [PyTorch SSD](./examples/post_training_quantization/torch/ssd300_vgg16/README.md)                                                        |            Post-Training Quantization            |  PyTorch   |    Object Detection    |
| [TensorFlow MobileNetV2](./examples/post_training_quantization/tensorflow/mobilenet_v2/README.md)                                        |            Post-Training Quantization            | TensorFlow |  Image Classification  |
| [ONNX MobileNetV2](./examples/post_training_quantization/onnx/mobilenet_v2/README.md)                                                    |            Post-Training Quantization            |    ONNX    |  Image Classification  |

### Training-Time Compression Examples

These examples provide full pipelines including compression, training and inference for classification, detection and segmentation tasks.

| Example Name                                                                                |   Compression Algorithm   |  Backend   |        Domain         |
|:-----------------------------------------------------------------------------------------------------------|:-------------------------:|:----------:|:---------------------:|
| [PyTorch Image Classification](./examples/torch/classification/README.md)                                  | Training-Time Compression |  PyTorch   | Image Classification  |
| [PyTorch Object Detection](./examples/torch/object_detection/README.md)                                    | Training-Time Compression |  PyTorch   |   Object Detection    |
| [PyTorch Semantic Segmentation](./examples/torch/semantic_segmentation/README.md)                          | Training-Time Compression |  PyTorch   | Semantic Segmentation |
| [TensorFlow Image Classification](./examples/tensorflow/classification/README.md)                          | Training-Time Compression | TensorFlow | Image Classification  |
| [TensorFlow Object Detection](./examples/tensorflow/object_detection/README.md)                            | Training-Time Compression | TensorFlow |   Object Detection    |
| [TensorFlow Instance Segmentation](./examples/tensorflow/segmentation/README.md)                           | Training-Time Compression | TensorFlow | Instance Segmentation |

## Third-party repository integration

NNCF may be straightforwardly integrated into training/evaluation pipelines of third-party repositories.

### Used by

- [OpenVINO Training Extensions](https://github.com/openvinotoolkit/training_extensions)

  NNCF is integrated into OpenVINO Training Extensions as model optimization backend. So you can train, optimize and export new models based on the available model templates as well as run exported models with OpenVINO.

- [HuggingFace Optimum Intel](https://huggingface.co/docs/optimum/intel/optimization_ov)

  NNCF is used as a compression backend within the renowned `transformers` repository in HuggingFace Optimum Intel.

## Installation Guide

For detailed installation instructions please refer to the [Installation](./docs/Installation.md) page.

NNCF can be installed as a regular PyPI package via pip:

```bash
pip install nncf
```

NNCF is also available via [conda](https://anaconda.org/conda-forge/nncf):

```bash
conda install -c conda-forge nncf
```

### System requirements

- Ubuntu\* 18.04 or later (64-bit)
- Python\* 3.8 or later
- Supported frameworks:
  - PyTorch\* >=2.2, <2.4
  - TensorFlow\* >=2.8.4, <=2.12.1
  - ONNX\* ==1.16.0
  - OpenVINO\* >=2022.3.0

This repository is tested on Python* 3.8.10, PyTorch* 2.3.0 (NVidia CUDA\* Toolkit 12.1) and TensorFlow* 2.12.1 (NVidia CUDA\* Toolkit 11.8).

## NNCF Compressed Model Zoo

List of models and compression results for them can be found at our [Model Zoo page](./docs/ModelZoo.md).

## Citing

```bi
@article{kozlov2020neural,
    title =   {Neural network compression framework for fast model inference},
    author =  {Kozlov, Alexander and Lazarevich, Ivan and Shamporov, Vasily and Lyalyushkin, Nikolay and Gorbachev, Yury},
    journal = {arXiv preprint arXiv:2002.08679},
    year =    {2020}
}
```

## Contributing Guide

Refer to the [CONTRIBUTING.md](./CONTRIBUTING.md) file for guidelines on contributions to the NNCF repository.

## Useful links

- [Documentation](./docs)
- Example scripts (model objects available through links in respective README.md files):
  - [PyTorch](./examples/torch)
  - [TensorFlow](./examples/tensorflow)
- [FAQ](./docs/FAQ.md)
- [Notebooks](https://github.com/openvinotoolkit/openvino_notebooks#-model-training)
- [HuggingFace Optimum Intel](https://huggingface.co/docs/optimum/intel/optimization_ov)
- [OpenVINO Model Optimization Guide](https://docs.openvino.ai/latest/openvino_docs_model_optimization_guide.html)

## Telemetry

NNCF as part of the OpenVINO™ toolkit collects anonymous usage data for the purpose of improving OpenVINO™ tools.
You can opt-out at any time by running the following command in the Python environment where you have NNCF installed:

`opt_in_out --opt_out`

More information is available at https://docs.openvino.ai/latest/openvino_docs_telemetry_information.html.


reddit2telegram
===============

[![](https://img.shields.io/badge/Telegram-Group-blue.svg)](https://t.me/r_channels)
[![](https://img.shields.io/badge/Telegram-Channel-blue.svg)](https://t.me/reddit2telegram)

Hi there!

These bots just send latest hot submissions from subreddit **/r/%subreddit%** to telegram channel **@r_%subreddit%** hourly. For example [/r/gifs](https://www.reddit.com/r/gifs/) → [@r_gifs](https://t.me/r_gifs).

If you want to obtain your own channel for `/r/%subreddit%`:

1. Create new public telegram channel `@r_%subreddit%`.
2. Add [`@reddit2telegram_bot`](https://t.me/reddit2telegram_bot) as administrator to this channel.
3. Make a pull request to this repo with new script to make posts (use [`reddit2telegram/channels/r_funnny/app.py`](https://github.com/Fillll/reddit2telegram/blob/master/reddit2telegram/channels/r_funny/app.py) as draft), and the tags related (create a tags.txt containing at least 3 tags, separated by spaces and starting with #).
4. Then I will make it alive :)
5. For any questions do not hesitate to contact me in [@r_channels](https://t.me/r_channels) group.

If you appreciate it
--------------------

Patreon: [donate](https://www.patreon.com/reddit2telegram)

BTC: `3DydQjRPaykiapdeab6ABAhB5f2JazwaRD`

ETH: `0xc313B7bEbFc605cC5B50fa97902506404D58df80`

LTC: `MT3gDG9VZvSDmsRexQ3k19JLxhDSCjf45T`

Currently alive channels
------------------------

Complete list of all active channels (more than 800) is available at our channel [@reddit2telegram](https://t.me/reddit2telegram).

This is a pet-project 🐶
------------------------

Which means you really shouldn't expect much from it. I wrote it over the weekend to solve my own pain. No state-of-art kubernetes bullshit, no architecture patterns, even no tests at all. It's here just to show people what a pet-project might look like.

This code has been written for fun, not for business. There is usually a big difference. Like between riding a bike on the streets and cycling in the wild for fun :)

_© from [infomate.club](https://github.com/vas3k/infomate.club)_

Running it
----------

Easiest way is to ask for new channel at [@r_channels](https://t.me/r_channels). Or follow the manual as stated above. :)

But if you are geek enough then install mongodb, ffmpeg, python and setup cron:

```cron
46 * * * * ~/reddit2telegram/auto_update.sh
* * * * * ~/reddit2telegram/reddit2telegram/cron_job.sh
```


![ivadomed Overview](https://raw.githubusercontent.com/ivadomed/doc-figures/main/index/overview_title.png)

[![DOI](https://joss.theoj.org/papers/10.21105/joss.02868/status.svg)](https://doi.org/10.21105/joss.02868)
[![Coverage Status](https://coveralls.io/repos/github/ivadomed/ivadomed/badge.svg?branch=master)](https://coveralls.io/github/ivadomed/ivadomed?branch=master)
[![test status](https://github.com/ivadomed/ivadomed/workflows/Run%20tests/badge.svg)](https://github.com/ivadomed/ivadomed/actions?query=workflow%3A%22Run+tests%22)
[![publish package](https://github.com/ivadomed/ivadomed/workflows/Publish%20Package/badge.svg)](https://github.com/ivadomed/ivadomed/actions?query=workflow%3A%22Publish+Package%22)
[![Documentation Status](https://readthedocs.org/projects/ivado-medical-imaging/badge/?version=latest)](https://ivadomed.org/en/latest/?badge=latest)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.md)
[![Twitter Follow](https://img.shields.io/twitter/follow/ivadomed.svg?style=social&label=Follow)](https://twitter.com/ivadomed)

`ivadomed` is an integrated framework for medical image analysis with deep learning.

The technical documentation is available [here](https://ivadomed.org).  The more detailed installation instruction is available [there](https://ivadomed.org/installation.html)

## Installation

``ivadomed`` requires Python >= 3.7 and < 3.10 as well as PyTorch == 1.8. We recommend working under a virtual environment, which could be set as follows:

```bash
python -m venv ivadomed_env
source ivadomed_env/bin/activate
```

### Install from release (recommended)

Install ``ivadomed`` and its requirements from `Pypi <https://pypi.org/project/ivadomed/>`__:

```bash
pip install --upgrade pip
pip install ivadomed
```

### Install from source

Bleeding-edge developments builds are available on the project's master branch on Github. Installation procedure is the following:

```bash
git clone https://github.com/neuropoly/ivadomed.git
cd ivadomed
pip install -e .
```

## Contributors

This project results from a collaboration between the [NeuroPoly Lab](https://www.neuro.polymtl.ca/)
and the [Mila](https://mila.quebec/en/). The main funding source is [IVADO](https://ivado.ca/en/).

[List of contributors](https://github.com/neuropoly/ivadomed/graphs/contributors)

## Consult our Wiki(https://github.com/ivadomed/ivadomed/wiki) here for more help


[![General Tests](https://github.com/spcl/dace/actions/workflows/general-ci.yml/badge.svg)](https://github.com/spcl/dace/actions/workflows/general-ci.yml)
[![GPU Tests](https://github.com/spcl/dace/actions/workflows/gpu-ci.yml/badge.svg)](https://github.com/spcl/dace/actions/workflows/gpu-ci.yml)
[![FPGA Tests](https://github.com/spcl/dace/actions/workflows/fpga-ci.yml/badge.svg)](https://github.com/spcl/dace/actions/workflows/fpga-ci.yml)
[![Documentation Status](https://readthedocs.org/projects/spcldace/badge/?version=latest)](https://spcldace.readthedocs.io/en/latest/?badge=latest)
[![PyPI version](https://badge.fury.io/py/dace.svg)](https://badge.fury.io/py/dace)
[![codecov](https://codecov.io/gh/spcl/dace/branch/master/graph/badge.svg)](https://codecov.io/gh/spcl/dace)

![D](dace.svg)aCe - Data-Centric Parallel Programming
=====================================================

_Decoupling domain science from performance optimization._

DaCe is a [fast](https://nbviewer.org/github/spcl/dace/blob/master/tutorials/benchmarking.ipynb) parallel programming
framework that takes code in Python/NumPy and other programming languages, and maps it to high-performance
**CPU, GPU, and FPGA** programs, which can be optimized to achieve state-of-the-art. Internally, DaCe
uses the Stateful DataFlow multiGraph (SDFG) *data-centric intermediate
representation*: A transformable, interactive representation of code based on
data movement.
Since the input code and the SDFG are separate, it is possible to optimize a
program without changing its source, so that it stays readable. On the other
hand, transformations are customizable and user-extensible, so they can be written
once and reused in many applications.
With data-centric parallel programming, we enable **direct knowledge transfer**
of performance optimization, regardless of the application or the target processor.

DaCe generates high-performance programs for:
 * Multi-core CPUs (tested on Intel, IBM POWER9, and ARM with SVE)
 * NVIDIA GPUs and AMD GPUs (with HIP)
 * Xilinx and Intel FPGAs

DaCe can be written inline in Python and transformed in the command-line/Jupyter
Notebooks or SDFGs can be interactively modified using our [Visual Studio Code extension](https://marketplace.visualstudio.com/items?itemName=phschaad.sdfv).

## [For more information, see the documentation](https://spcldace.readthedocs.io/en/latest/)

Quick Start
-----------

Install DaCe with pip: `pip install dace`

Having issues? See our full [Installation and Troubleshooting Guide](https://spcldace.readthedocs.io/en/latest/setup/installation.html).

Using DaCe in Python is as simple as adding a `@dace` decorator:
```python
import dace
import numpy as np

@dace
def myprogram(a):
    for i in range(a.shape[0]):
        a[i] += i
    return np.sum(a)
```

Calling `myprogram` with any NumPy array or GPU array (e.g., PyTorch, Numba, CuPy) will
generate data-centric code, compile, and run it. From here on out, you can
_optimize_ (interactively or automatically), _instrument_, and _distribute_
your code. The code creates a shared library (DLL/SO file) that can readily
be used in any C ABI compatible language (C/C++, FORTRAN, etc.).

For more information on how to use DaCe, see the [samples](samples) or tutorials below:

* [Getting Started](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/getting_started.ipynb)
* [Benchmarks, Instrumentation, and Performance Comparison with Other Python Compilers](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/benchmarking.ipynb)
* [Explicit Dataflow in Python](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/explicit.ipynb)
* [NumPy API Reference](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/numpy_frontend.ipynb)
* [SDFG API](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/sdfg_api.ipynb)
* [Using and Creating Transformations](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/transformations.ipynb)
* [Extending the Code Generator](https://nbviewer.jupyter.org/github/spcl/dace/blob/master/tutorials/codegen.ipynb)

Publication
-----------

The paper for the SDFG IR can be found [here](http://www.arxiv.org/abs/1902.10345).
Other DaCe-related publications are available on our [website](http://spcl.inf.ethz.ch/dace).

If you use DaCe, cite us:
```bibtex
@inproceedings{dace,
  author    = {Ben-Nun, Tal and de~Fine~Licht, Johannes and Ziogas, Alexandros Nikolaos and Schneider, Timo and Hoefler, Torsten},
  title     = {Stateful Dataflow Multigraphs: A Data-Centric Model for Performance Portability on Heterogeneous Architectures},
  year      = {2019},
  booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
  series = {SC '19}
}
```

Contributing
------------
DaCe is an open-source project. We are happy to accept Pull Requests with your contributions! Please follow the [contribution guidelines](CONTRIBUTING.md) before submitting a pull request.

License
-------
DaCe is published under the New BSD license, see [LICENSE](LICENSE).


# OCS-CI

OCS CI is a framework to test Red Hat OCS features and functionality using AWS
and other supported platforms. The framework is based on pytest.

## Automatically generated documentation

The primary documentation is available at
[ocs-ci.readthedocs.io](https://ocs-ci.readthedocs.io/en/latest/).

Source files of the documentation are maintained as a
[Sphinx](https://www.sphinx-doc.org/en/master/) project in [`docs`
directory](docs/).

## License

This project is open sourced under MIT License.


Lithops is a Python multi-cloud distributed computing framework. It allows you to run unmodified local python code at massive scale in the main
serverless computing platforms. Lithops delivers the user’s code into the cloud without requiring knowledge of how it is deployed and run. Moreover, its multicloud-agnostic architecture ensures portability across cloud providers.

Lithops is specially suited for highly-parallel programs with little or no need for communication between processes, but it also supports parallel applications that need to share state among processes. Examples of applications that run with Lithops include Monte Carlo simulations, deep learning and machine learning processes, metabolomics computations, and geospatial analytics, to name a few.

## Installation

1. Install Lithops from the PyPi repository:

    ```bash
    pip install lithops
    ```

2. Execute a *Hello World* function:

   ```bash
   lithops hello
   ```

## Configuration
Lithops provides an extensible backend architecture (compute, storage) that is designed to work with different Cloud providers and on-premise backends. In this sense, you can code in python and run it unmodified in IBM Cloud, AWS, Azure, Google Cloud, Aliyun and Kubernetes or OpenShift.

[Follow these instructions to configure your compute and storage backends](config/)

## High-level API

Lithops is shipped with 2 different high-level Compute APIs, and 2 high-level Storage APIs

Futures API

Multiprocessing API

```python
from lithops import FunctionExecutor

def hello(name):
    return f'Hello {name}!'

with FunctionExecutor() as fexec:
    f = fexec.call_async(hello, 'World')
    print(f.result())
```

```python
from lithops.multiprocessing import Pool

def double(i):
    return i * 2

with Pool() as pool:
    result = pool.map(double, [1, 2, 3, 4])
    print(result)
```

Storage API

Storage OS API

```python
from lithops import Storage

if __name__ == "__main__":
    st = Storage()
    st.put_object(bucket='mybucket',
                  key='test.txt',
                  body='Hello World')

    print(st.get_object(bucket='lithops',
                        key='test.txt'))
```

```python
from lithops.storage.cloud_proxy import os

if __name__ == "__main__":
    filepath = 'bar/foo.txt'
    with os.open(filepath, 'w') as f:
        f.write('Hello world!')

    dirname = os.path.dirname(filepath)
    print(os.listdir(dirname))
    os.remove(filepath)
```

You can find more usage examples in the [examples](/examples) folder.

## Execution Modes

Lithops is shipped with 3 different modes of execution. The execution mode allows you to decide where and how the functions are executed.

* [Localhost Mode](docs/source/execution_modes.rst#localhost-mode)

  This mode allows you to execute functions on the local machine using processes, providing a convenient way to leverage Lithops' distributed computing capabilities without relying on cloud resources. This mode is particularly useful for development, testing, and debugging purposes. This is the default mode of execution if no configuration is provided.

* [Serverless Mode](docs/source/execution_modes.rst#serverless-mode)

  This mode allows you to execute functions on popular serverless compute services, leveraging the scalability, isolation, and automatic resource provisioning provided by these platforms. With serverless mode, you can easily parallelize task execution, harness the elastic nature of serverless environments, and simplify the development and deployment of scalable data processing workloads and parallel applications.

* [Standalone Mode](docs/source/execution_modes.rst#standalone-mode)

  This mode provides the capability to execute functions on one or multiple virtual machines (VMs) simultaneously, in a serverless-like fashion, without requiring manual provisioning as everything is automatically created. This mode can be used in a private cluster or in the cloud, where functions within each VM are executed using parallel processes.

## Documentation

For documentation on using Lithops, see [latest release documentation](https://lithops-cloud.github.io/docs/) or [current github docs](docs/user_guide.md).

If you are interested in contributing, see [CONTRIBUTING.md](./CONTRIBUTING.md).

## Additional resources

### Blogs and Talks
* [Simplify the developer experience with OpenShift for Big Data processing by using Lithops framework](https://medium.com/@gvernik/simplify-the-developer-experience-with-openshift-for-big-data-processing-by-using-lithops-framework-d62a795b5e1c)
* [Speed-up your Python applications using Lithops and Serverless Cloud resources](https://itnext.io/speed-up-your-python-applications-using-lithops-and-serverless-cloud-resources-a64beb008bb5)
* [Serverless Without Constraints](https://www.ibm.com/cloud/blog/serverless-without-constraints)
* [Lithops, a Multi-cloud Serverless Programming Framework](https://itnext.io/lithops-a-multi-cloud-serverless-programming-framework-fd97f0d5e9e4)
* [CNCF Webinar - Toward Hybrid Cloud Serverless Transparency with Lithops Framework](https://www.youtube.com/watch?v=-uS-wi8CxBo)
* [Using Serverless to Run Your Python Code on 1000 Cores by Changing Two Lines of Code](https://www.ibm.com/cloud/blog/using-serverless-to-run-your-python-code-on-1000-cores-by-changing-two-lines-of-code)
* [Decoding dark molecular matter in spatial metabolomics with IBM Cloud Functions](https://www.ibm.com/cloud/blog/decoding-dark-molecular-matter-in-spatial-metabolomics-with-ibm-cloud-functions)
* [Your easy move to serverless computing and radically simplified data processing](https://www.slideshare.net/gvernik/your-easy-move-to-serverless-computing-and-radically-simplified-data-processing-238929020) Strata Data Conference, NY 2019. See video of Lithops usage [here](https://www.youtube.com/watch?v=EYa95KyYEtg&list=PLpR7f3Www9KCjYisaG7AMaR0C2GqLUh2G&index=3&t=0s) and the example of Monte Carlo [here](https://www.youtube.com/watch?v=vF5HI2q5VKw&list=PLpR7f3Www9KCjYisaG7AMaR0C2GqLUh2G&index=2&t=0s)
* [Speed up data pre-processing with Lithops in deep learning](https://developer.ibm.com/patterns/speed-up-data-pre-processing-with-pywren-in-deep-learning/)
* [Predicting the future with Monte Carlo simulations over IBM Cloud Functions](https://www.ibm.com/cloud/blog/monte-carlo-simulations-with-ibm-cloud-functions)
* [Process large data sets at massive scale with Lithops over IBM Cloud Functions](https://www.ibm.com/cloud/blog/process-large-data-sets-massive-scale-pywren-ibm-cloud-functions)
* [Industrial project in Technion on Lithops](http://www.cs.technion.ac.il/~cs234313/projects_sites/W19/04/site/)

### Papers

* [Outsourcing Data Processing Jobs with Lithops](https://ieeexplore.ieee.org/document/9619947) - IEEE Transactions on Cloud Computing 2022
* [Towards Multicloud Access Transparency in Serverless Computing](https://www.computer.org/csdl/magazine/so/5555/01/09218932/1nMMkpZ8Ko8) - IEEE Software 2021
* [Primula: a Practical Shuffle/Sort Operator for Serverless Computing](https://dl.acm.org/doi/10.1145/3429357.3430522) - ACM/IFIP International Middleware Conference 2020. [See presentation here](https://www.youtube.com/watch?v=v698iu5YfWM)
* [Bringing scaling transparency to Proteomics applications with serverless computing](https://dl.acm.org/doi/abs/10.1145/3429880.3430101) - 6th International Workshop on Serverless Computing (WoSC6) 2020. [See presentation here](https://www.serverlesscomputing.org/wosc6/#p10)
* [Serverless data analytics in the IBM Cloud](https://dl.acm.org/citation.cfm?id=3284029) - ACM/IFIP International Middleware Conference 2018

# Acknowledgements
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184 (CloudButton).


===============
VulnerableCode
===============

|Build Status| |Code License| |Data License| |Python 3.8+| |stability-wip| |Gitter chat|

.. |Build Status| image:: https://github.com/nexB/vulnerablecode/actions/workflows/main.yml/badge.svg?branch=main
   :target: https://github.com/nexB/vulnerablecode/actions?query=workflow%3ACI
.. |Code License| image:: https://img.shields.io/badge/Code%20License-Apache--2.0-green.svg
   :target: https://opensource.org/licenses/Apache-2.0
.. |Data License| image:: https://img.shields.io/badge/Data%20License-CC--BY--SA--4.0-green.svg
   :target: https://creativecommons.org/licenses/by-sa/4.0/legalcode
.. |Python 3.8+| image:: https://img.shields.io/badge/python-3.8+-green.svg
   :target: https://www.python.org/downloads/release/python-380/
.. |stability-wip| image:: https://img.shields.io/badge/stability-work_in_progress-lightgrey.svg
.. |Gitter chat| image:: https://badges.gitter.im/gitterHQ/gitter.png
   :target: https://gitter.im/aboutcode-org/vulnerablecode

VulnerableCode is a free and open database of open source software package
vulnerabilities **because open source software vulnerabilities data and tools
should be free and open source themselves**:

we are trying to change this and evolve the status quo in a few other areas!

- Vulnerability databases have been **traditionally proprietary** even though they
  are mostly about free and open source software.

- Vulnerability databases also often contain a lot of lesser value data which
  means a lot of false positive signals that require extensive expert reviews.

- Vulnerability databases are also mostly about vulnerabilities first and software
  package second, making it difficult to find if and when a vulnerability applies
  to a piece of code. VulnerableCode focus is on software package first where
  a Package URL is a key and natural identifier for packages; this is making it
  easier to find a package and whether it is vulnerable.

Package URL themselves were designed first in ScanCode and VulnerableCode
and are now a de-facto standard for vulnerability management and package references.

See https://github.com/package-url/purl-spec

The VulnerableCode project is a FOSS community resource to help improve the
security of the open source software ecosystem and its users at large.

VulnerableCode consists of a database and the tools to collect, refine and keep
the database current.

.. warning::
    VulnerableCode is under active development and is not yet fully
    usable.

Read more about VulnerableCode https://vulnerablecode.readthedocs.org/

VulnerableCode is financially supported by NLnet, nexB, Google (through the
GSoC) and the active contributions of several volunteers.

VulnerableCode tech stack is Python, Django, PostgreSQL, nginx and Docker and
several libraries.

Getting started
---------------

Run with Docker
^^^^^^^^^^^^^^^^

First install docker and docker-compose, then run::

    git clone https://github.com/nexB/vulnerablecode.git && cd vulnerablecode
    make envfile
    docker-compose build
    docker-compose up -d
    docker-compose run vulnerablecode ./manage.py import --list

Then run an importer for nginx advisories (which is small)::

    docker-compose exec vulnerablecode ./manage.py import vulnerabilities.importers.nginx.NginxImporter
    docker-compose exec vulnerablecode ./manage.py improve --all

At this point, the VulnerableCode app and API should be up and running with
some data at http://localhost

Populate VulnerableCode database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

VulnerableCode data collection works in two steps: importing data from multiple
sources and then refining and improving how package and software vulnerabilities
are related.

To run all importers and improvers use this::

   ./manage.py import --all
   ./manage.py improve --all

Local development installation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On a Debian system, use this::

    sudo apt-get install  python3-venv python3-dev postgresql libpq-dev build-essential
    git clone https://github.com/nexB/vulnerablecode.git && cd vulnerablecode
    make dev envfile postgres
    make test
    source venv/bin/activate
    ./manage.py import vulnerabilities.importers.nginx.NginxImporter
    ./manage.py improve --all
    make run

At this point, the VulnerableCode app and API is up at http://127.0.0.1:8001/

Interface
^^^^^^^^^^

VulnerableCode comes with a minimal web UI:

.. image:: vulnerablecode-ui.png

And a JSON API and its minimal web documentation:

.. image:: vulnerablecode-json-api.png
.. image:: vulnerablecode-api-doc.png

License
^^^^^^^^^^

Copyright (c) nexB Inc. and others. All rights reserved.

VulnerableCode is a trademark of nexB Inc.

SPDX-License-Identifier: Apache-2.0 AND CC-BY-SA-4.0

VulnerableCode software is licensed under the Apache License version 2.0.

VulnerableCode data is licensed collectively under CC-BY-SA-4.0.

See https://www.apache.org/licenses/LICENSE-2.0 for the license text.

See https://creativecommons.org/licenses/by-sa/4.0/legalcode for the license text.

See https://github.com/nexB/vulnerablecode for support or download.

See https://aboutcode.org for more information about nexB OSS projects.

Acknowledgements
^^^^^^^^^^^^^^^^

This project was funded through the NGI0 PET Fund, a fund established by
NLnet with financial support from the European Commission's Next Generation
Internet programme, under the aegis of DG Communications Networks, Content
and Technology under grant agreement No 825310.

https://nlnet.nl/project/VulnerableCode/

This project was funded through the NGI0 Discovery Fund, a fund established
by NLnet with financial support from the European Commission's Next Generation
Internet programme, under the aegis of DG Communications Networks, Content
and Technology under grant agreement No 825322.

https://nlnet.nl/project/vulnerabilitydatabase/


# Scalyr Agent (scalyr-agent-2)

[![Lint](https://github.com/scalyr/scalyr-agent-2/actions/workflows/lint.yml/badge.svg)](https://github.com/scalyr/scalyr-agent-2/actions/workflows/lint.yml) [![Unittests](https://github.com/scalyr/scalyr-agent-2/actions/workflows/unittests.yml/badge.svg)](https://github.com/scalyr/scalyr-agent-2/actions/workflows/unittests.yml) [![Docker Images Build](https://github.com/scalyr/scalyr-agent-2/actions/workflows/agent-build.yml/badge.svg)](https://github.com/scalyr/scalyr-agent-2/actions/workflows/agent-build.yml) [![Kubernetes End to End Tests](https://github.com/scalyr/scalyr-agent-2/actions/workflows/end_to_end_tests.yml/badge.svg)](https://github.com/scalyr/scalyr-agent-2/actions/workflows/end_to_end_tests.yml) [![codecov](https://codecov.io/gh/scalyr/scalyr-agent-2/branch/master/graph/badge.svg)](https://codecov.io/gh/scalyr/scalyr-agent-2)

This repository holds the source code for the Scalyr Agent, a daemon that collects logs and metrics from
customer machines and transmits them to Scalyr.

For more information on the Scalyr Agent, please visit https://www.scalyr.com/help/scalyr-agent.

To learn more about Scalyr, visit https://www.scalyr.com.

## Features

The Scalyr Agent is designed to be lightweight, easy to install, and safe to run on production systems.
Key features:

  * Pure Python implementation, supporting Python versions 2.6, 2.7 and >= 3.5
  * Lightweight (typically 15 MB RAM, 2% CPU or less)
  * Easy-to-use troubleshooting features
  * Modular configuration files
  * Extensibility using monitor plugins

## Release Notes and Changelog

For release notes please see [RELEASE_NOTES.md](RELEASE_NOTES.md) document and for changelog,
see [CHANGELOG.md](CHANGELOG.md) document.

## Developing

From this repository, you can create your own RPM and Debian packages containing customized versions of
the Scalyr Agent. For instance, you can bundle additional monitoring plugins to collect specialized data
from your servers.

We also welcome submissions from the community.

For information on the agent architecture and code abstractions, please refer to the
[architecture.md](docs/architecture.md) document.

Some of the tests and checks depend on on fixture file from git submodule so you should make sure you
also check out all the submodules:

```bash
git submodule update --recursive --init
```

### Local Development Environment, Tests and Lint Checks

This repository utilizes ``tox`` Python project for running various lint checks and tests in
isolated virtual environment.

Underneath, we use ``py.test`` test runner for running the tests.

For it to work, you need to have ``tox`` Python package installed on the system or inside the virtual
environment which you use for the development.

```bash
pip install tox
```

In addition to that, you also need to have Python version available which is used for a particular
tox target. By default Python 2.7 is used for the unit tests and Python 3.6 for all the lint checks.

To run all the checks you can simply run tox command:

```bash
tox
```

In addition to that, you can also run a specific target or a set of targets using ``-e`` flag. For example:

```bash
# run all the lint targets
tox -elint

# run flake8 and mypy tox target
tox -eflake8,mypy

# run py2.7-unit-tests tox target
tox -epy2.7-unit-tests

# run coverage tox target
tox -ecoverage
```

To run a sub-set of tests or a single test from a test file, you can directly invoke ``pytest``
from a specific tox virtual environment as shown below:

```bash
.tox/py2.7-unit-tests/bin/py.test -vv --durations=5 "<test file>::<test class name>" -k "<test method name>"
```

For example:

```bash
# This will run all the tests in tests/unit/url_monitor_test.py file
.tox/py2.7-unit-tests/bin/py.test -vv --durations=5 "tests/unit/url_monitor_test.py"

# This will run all the test methods on the ``UrlMonitorTestRequest`` class in the
# tests/unit/url_monitor_test.py file
.tox/py2.7-unit-tests/bin/py.test -vv --durations=5 "tests/unit/url_monitor_test.py::UrlMonitorTestRequest"

# This will run ``UrlMonitorTestRequest.test_get_request_no_headers`` test method from the
# tests/unit/url_monitor_test.py file
.tox/py2.7-unit-tests/bin/py.test -vv --durations=5 "tests/unit/url_monitor_test.py::UrlMonitorTestRequest" -k test_get_request_no_headers
```

### Continuous Integration

We run all the tox checks described above (+ more) continuously as part of our Github Actions based
build system.

Each push to a branch / pull request will trigger a build and a subset of the GHA jobs.

Additional jobs will run once the PR has been merged into master. The reason we do that is to
speed the PR builds and increase the developer feedback loop (some of the tests and checks we
run are slow so running them on every push to a branch would be slow and wasteful).

Before merging a pull request you need to ensure that all the checks have passed, pull
request has been approved and it's in sync / up to date with latest master.

When all the checks have passed, you should see something like this:

After the PR has been merged, you should wait for the ``benchmarks`` workflow to complete and then
check our [CodeSpeed instance](https://scalyr-agent-codespeed.herokuapp.com/) to ensure there are
no regressions in terms of the resource utilization (memory and CPU usage) and things such as
increased number of error or warning log lines.

Here is an example which demonstrates a regression in number of error level lines being printed to
the agent log which likely indicates a bug / regression in the code -
https://github.com/scalyr/scalyr-agent-2/pull/513#issuecomment-617228472.

### Monitor Plugins

Monitor plugins are one of the key features for Scalyr Agent 2.  These plugins can be used to augment the
functionality of Scalyr Agent 2 beyond just copying logs and metrics.  For example, there are monitor plugins
that will fetch page content at a given URL and then log portions of the returned content.  Another plugin allows
you to execute a shell command periodically and then log the output.  Essentially, plugs can be used to implement
any periodic monitoring task you have.

We encourage users to create their own plugins to cover features they desire.  In the near future, we will be
publishing documentation that describes how to implement your own monitor.  And, if you feel your monitor would
be useful to other Scalyr customers, we encourage you to submit it to monitor collection in the `monitors/contrib`
directory.

To learn how to develop plugins, please see the
[instructions for creating a monitor plugin](docs/CREATING_MONITORS.md).

### Building Packages

NOTE: Docker image build instructions now in the [BUILD.md](BUILD.md)

You can use the `build_packages.py` script to build your own RPM or Debian packages.  This is often desirable
if your company has its own yum or apt repositories, or you have modified the Scalyr Agent 2 code to suit
some particular need.  (Of course, if you are finding you need to modify the Scalyr Agent 2 code, we encourage you
to submit your changes back to the main repository).

You must have the following installed on your machine to create packages:

  * fpm, see https://github.com/jordansissel/fpm/
  * rpmbuild (for building RPMs)
  * gnutar / gtar (for building Debian packages)

We strongly suggest you use the same platform that you intend to install the agent on to build the packages.
This is because tools like `rpmbuild` and `gtar` are more available on the platforms that use those respective
packaging systems.

To build the RPM package, execute the following command in the root directory of this repository

    python build_package.py rpm

To build the Debian package, execute the following command in the root directory of this repository

    python build_package.py deb

### Pre-Commit Hooks

This project uses the [Black](http://black.readthedocs.io) code autoformatting tool with default
settings.

[Pre-commit](https://pre-commit.com) is used to automatically run checks including Black formatting
prior to a git commit.

To use pre-commit:

- Use one of the [Installation Methods](https://pre-commit.com/#install) from the documentation.
- Install the hooks with `pre-commit install`.
- To manually execute the pre-commit hooks (including black), run `pre-commit run --all-files` (
  run it on all the files) or ``pre-commit run`` (run it only on staged files).

All the pre-commit targets rely on binaries from tox virtual environment so you need to make sure
tox virtual environment for ``lint`` target exists:

```bash
tox -elint --notest --recreate
```

#### Pre-commit Configuration

- `.pre-commit-config.yaml` configures the scripts run by pre-commit

To update the Pre-commit hooks , run `pre-commit autoupdate`. This will update
`.pre-commit-config.yaml` and will need to be committed to the repository.

## Contributing

In the future, we will be pushing guidelines on how to contribute to this repository.  For now, please just
feel free to submit pull requests to the `master` branch and we will work with you.

## Copyright, License, and Contributors Agreement

Copyright 2014-2021 Scalyr, Inc. Copyright 2021 SentinelOne, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in
compliance with the License. You may obtain a copy of the License in the [LICENSE](LICENSE.txt) file, or at:

[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)

By contributing you agree that these contributions are your own (or approved by your employer) and you
grant a full, complete, irrevocable copyright license to all users and developers of the project,
present and future, pursuant to the license of the project.


Open-PDKs (open_pdks)
-----------------------------------------

Summary:

Automatic setup of PDKs for open-source tools from foundry sources.

Builds out and populates a new set of directories and subdirectories in
the efabless format, with the efabless name for the PDK at the top,
followed by categories "libs.ref" (IP) and "libs.tech" (EDA tool setup),
each with subcategories corresponding to layout, abstract views,
netlists, etc. for the IP;  and magic, netgen, qflow, etc., for the
EDA tool setup.

The populated PDK directories can contain either copies of files from
the foundry sources, links to the foundry sources, or links back to
another PDK.

Generates magic layout views for all vendor IP for which either a GDS
view or a LEF view exists.  Annotates the views as needed to handle
ports, bounding boxes, etc.

-----------------------------------------

This distribution contains sources for building out the SkyWater SKY130
130nm process.  Sources for the foundry process data must be obtained
separately.  Read the README file in subdirectory sky130/ for instructions
on obtaining and building the SkyWater sky130A PDK.

-----------------------------------------

License:

Open_PDKs is open-source software distributed under the Apache-2.0 license.
See file LICENSE for the complete license text.

-----------------------------------------

Instructions:

    git clone https://github.com/RTimothyEdwards/open_pdks.git
    cd open_pdks
    ./configure [options]
    make
    [sudo] make install

where the configure [options] are specific to each foundry PDK supported
by open_pdks and can be queried using

    ./configure --help

At a minimum you will want to pass a configure option to declare the PDK
that you want to build.  It is possible to build multiple PDKs at once,
but not recommended due to the large memory and disk space overhead
required by each one.

    ./configure --enable-[PDK_name]-pdk

The open_pdks version of the PDK can be built locally within open_pdks without
the need to declare an install target or run "make install".  However, it is
highly recommended to declare a target location and install there, using

    ./configure --prefix=[path] --enable-[PDK_name]-pdk

followed by "make" and "make install".  The default install location is
/usr/local/share/pdk ([path] above is /usr/local).  The install location
should be a read-only filesystem area for regular users, since the PDK
contents should not be altered.

Also see the website at http://opencircuitdesign.com/open_pdks/.  The "Install"
page has full instructions for configuring and installing open_pdks.


![CompilerGym](https://github.com/facebookresearch/CompilerGym/raw/development/docs/source/_static/img/logo-padded.png)

  Reinforcement learning environments for compiler optimization tasks.

    Check
    the website
    for more information.

## Introduction

CompilerGym is a library of easy to use and performant reinforcement learning
environments for compiler tasks. It allows ML researchers to interact with
important compiler optimization problems in a language and vocabulary with which
they are comfortable, and provides a toolkit for systems developers to expose
new compiler tasks for ML research. We aim to act as a catalyst for making
compilers faster using ML. Key features include:

* **Ease of use:** built on the the popular [Gym](https://gym.openai.com/)
  interface - use Python to write your agent. With CompilerGym, building ML
  models for compiler research problems is as easy as building ML models to play
  video games.

* **Batteries included:** includes everything required to get started. Wraps
  real world programs and compilers to provide millions of instances for
  training. Provides multiple kinds of pre-computed program representations: you
  can focus on end-to-end deep learning or features + boosted trees, all the way
  up to graph models. Appropriate reward functions and loss functions for
  optimization targets are provided out of the box.

* **Reproducible:** provides validation for correctness of results, common
  baselines, and [leaderboards](#leaderboards) for you to submit your results.

For a glimpse of what's to come, check out [our
roadmap](https://github.com/facebookresearch/CompilerGym/projects/1).

## News

- November 2022: CompilerGym v0.2.5 adds support for Python 3.10 and drops
  support for 3.7. See [release
  notes](https://github.com/facebookresearch/CompilerGym/releases/tag/v0.2.5)
  for full details.
- April 2022: ⭐ CompilerGym wins the **distinguished paper award** at
  [CGO'22](https://conf.researchr.org/home/cgo-2022)! You can read our work
  [here](https://arxiv.org/pdf/2109.08267.pdf).
- April 2022: 📖 Our **tutorial** at
  [CGO'22](https://conf.researchr.org/home/cgo-2022) was well attended. If you
  missed the event, you can work through the materials
  [here](https://chriscummins.cc/2022/cgo22-tutorial/).
- September 2021: 📄 CompilerGym was featured on the **Meta AI research blog**.
  You can read the post
  [here](https://ai.facebook.com/blog/compilergym-making-compiler-optimizations-accessible-to-all/).

## Installation

Install the latest CompilerGym release using:

    pip install -U compiler_gym

See
[INSTALL.md](https://github.com/facebookresearch/CompilerGym/blob/development/INSTALL.md)
for further details.

## Usage

Starting with CompilerGym is simple. If you not already familiar with the gym
interface, refer to the [getting started
guide](http://facebookresearch.github.io/CompilerGym/getting_started.html) for
an overview of the key concepts.

In Python, import `compiler_gym` to use the environments:

```py
>>> import compiler_gym                      # imports the CompilerGym environments
>>> env = compiler_gym.make(                 # creates a new environment (same as gym.make)
...     "llvm-v0",                           # selects the compiler to use
...     benchmark="cbench-v1/qsort",         # selects the program to compile
...     observation_space="Autophase",       # selects the observation space
...     reward_space="IrInstructionCountOz", # selects the optimization target
... )
>>> env.reset()                              # starts a new compilation session
>>> env.render()                             # prints the IR of the program
>>> env.step(env.action_space.sample())      # applies a random optimization, updates state/reward/actions
>>> env.close()                              # closes the environment, freeing resources
```

See the [examples](/examples) directory for agent implementations, environment
extensions, and more. See the [documentation
website](http://facebookresearch.github.io/CompilerGym/) for the API reference.

## Leaderboards

These leaderboards track the performance of user-submitted algorithms for
CompilerGym tasks. To submit a result please see
[this document](https://github.com/facebookresearch/CompilerGym/blob/development/CONTRIBUTING.md#leaderboard-submissions).

### LLVM Instruction Count

LLVM is a popular open source compiler used widely in industry and research. The
`llvm-ic-v0` environment exposes LLVM's optimizing passes as a set of actions
that can be applied to a particular program. The goal of the agent is to select
the sequence of optimizations that lead to the greatest reduction in instruction
count in the program being compiled. Reward is the reduction in instruction
count achieved scaled to the reduction achieved by LLVM's builtin `-Oz`
pipeline.

This leaderboard tracks the results achieved by algorithms on the `llvm-ic-v0`
environment on the 23 benchmarks in the `cbench-v1` dataset.

| Author                                           | Algorithm | Links | Date | Walltime (mean) | Codesize Reduction (geomean) |
|--------------------------------------------------| --- | --- | --- |-----------------|------------------------------|
 | Robin Schmöcker, Yannik Mahlau, Nicolas Fröhlich | PPO + Guided Search | [write-up](leaderboard/llvm_instcount/ppo/README.md), [results](leaderboard/llvm_instcount/ppo/results.csv) | 2022-02 | 69.821s  | **1.070×**                   |
| Facebook                                         | Random search (t=10800) | [write-up](leaderboard/llvm_instcount/random_search/README.md), [results](leaderboard/llvm_instcount/random_search/results_p125_t10800.csv) | 2021-03 | 10,512.356s     | 1.062×                       |
| Facebook                                         | Random search (t=3600) | [write-up](leaderboard/llvm_instcount/random_search/README.md), [results](leaderboard/llvm_instcount/random_search/results_p125_t3600.csv) | 2021-03 | 3,630.821s      | 1.061×                       |
| Facebook                                         | Greedy search | [write-up](leaderboard/llvm_instcount/e_greedy/README.md), [results](leaderboard/llvm_instcount/e_greedy/results_e0.csv) | 2021-03 | 169.237s        | 1.055×                       |
| Anthony. W. Jung                                 | GATv2 + DD-PPO | [write-up](leaderboard/llvm_instcount/gatv2_ddppo/README.md), [results](leaderboard/llvm_instcount/gatv2_ddppo/result.csv) | 2022-06 | 258.149s        | 1.047×                       |
| Facebook                                         | Random search (t=60) | [write-up](leaderboard/llvm_instcount/random_search/README.md), [results](leaderboard/llvm_instcount/random_search/results_p125_t60.csv) | 2021-03 | 91.215s         | 1.045×                       |
| Facebook                                         | e-Greedy search (e=0.1) | [write-up](leaderboard/llvm_instcount/e_greedy/README.md), [results](leaderboard/llvm_instcount/e_greedy/results_e10.csv) | 2021-03 | 351.611s        | 1.041×                       |
| Jiadong Guo                                      | Tabular Q (N=5000, H=10) | [write-up](leaderboard/llvm_instcount/tabular_q/README.md), [results](leaderboard/llvm_instcount/tabular_q/results-H10-N5000.csv) | 2021-04 | 2534.305        | 1.036×                       |
| Facebook                                         | Random search (t=10) | [write-up](leaderboard/llvm_instcount/random_search/README.md), [results](leaderboard/llvm_instcount/random_search/results_p125_t10.csv) | 2021-03 | 42.939s         | 1.031×                       |
| Patrick Hesse                                    | DQN (N=4000, H=10) | [write-up](leaderboard/llvm_instcount/dqn/README.md), [results](leaderboard/llvm_instcount/dqn/results-instcountnorm-H10-N4000.csv) | 2021-06 | 91.018s         | 1.029×                       |
| Jiadong Guo                                      | Tabular Q (N=2000, H=5) | [write-up](leaderboard/llvm_instcount/tabular_q/README.md), [results](leaderboard/llvm_instcount/tabular_q/results-H5-N2000.csv) | 2021-04 | 694.105         | 0.988×                       |

## Contributing

We welcome contributions to CompilerGym. If you are interested in contributing please see
[this document](https://github.com/facebookresearch/CompilerGym/blob/development/CONTRIBUTING.md).

## Citation

If you use CompilerGym in any of your work, please cite [our
paper](https://arxiv.org/pdf/2109.08267.pdf):

```
@inproceedings{CompilerGym,
      title={{CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research}},
      author={Chris Cummins and Bram Wasti and Jiadong Guo and Brandon Cui and Jason Ansel and Sahir Gomez and Somya Jain and Jia Liu and Olivier Teytaud and Benoit Steiner and Yuandong Tian and Hugh Leather},
      booktitle={CGO},
      year={2022},
}
```


[![Codecov](https://codecov.io/gh/mne-tools/mne-bids/branch/main/graph/badge.svg)](https://codecov.io/gh/mne-tools/mne-bids)
[![GitHub Actions](https://github.com/mne-tools/mne-bids/workflows/build/badge.svg)](https://github.com/mne-tools/mne-bids/actions)
[![CircleCI](https://circleci.com/gh/mne-tools/mne-bids.svg?style=shield)](https://circleci.com/gh/mne-tools/mne-bids)
[![PyPI Download count](https://pepy.tech/badge/mne-bids)](https://pepy.tech/project/mne-bids)
[![Latest PyPI release](https://img.shields.io/pypi/v/mne-bids.svg)](https://pypi.org/project/mne-bids/)
[![Latest conda-forge release](https://img.shields.io/conda/vn/conda-forge/mne-bids.svg)](https://anaconda.org/conda-forge/mne-bids/)
[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108)

MNE-BIDS
========

MNE-BIDS is a Python package that allows you to read and write
[BIDS](https://bids.neuroimaging.io/)-compatible datasets with the help of
[MNE-Python](https://mne.tools/stable/index.html).

![Schematic: From raw data to BIDS using MNE-BIDS](https://mne.tools/mne-bids/assets/MNE-BIDS.png)

Why?
----

MNE-BIDS links BIDS and MNE-Python with the goal to make your analyses faster to code, more robust, and facilitate data and code sharing with co-workers and collaborators.

How?
----

The documentation can be found under the following links:

- for the [stable release](https://mne.tools/mne-bids/)
- for the [latest (development) version](https://mne.tools/mne-bids/dev/index.html)

Getting Help
------------
[<img alt="MNE Forum" src="https://user-images.githubusercontent.com/1681963/52239617-e2683480-289c-11e9-922b-5da55472e5b4.png" height=60/>](https://mne.discourse.group)

For any usage questions, please post to the
[MNE Forum](https://mne.discourse.group). Be sure to add the `mne-bids` tag to
your question.

Citing
------

[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108)

If you use MNE-BIDS in your work, please cite our
[publication in JOSS](https://doi.org/10.21105/joss.01896):

> Appelhoff, S., Sanderson, M., Brooks, T., Vliet, M., Quentin, R., Holdgraf, C.,
Chaumon, M., Mikulan, E., Tavabi, K., Höchenberger, R., Welke, D., Brunner, C.,
Rockhill, A., Larson, E., Gramfort, A., & Jas, M. (2019): **MNE-BIDS: Organizing
electrophysiological data into the BIDS format and facilitating their analysis.**
*Journal of Open Source Software,* 4:1896. DOI: [10.21105/joss.01896](https://doi.org/10.21105/joss.01896)

Please also cite one of the following papers to credit BIDS, depending on which data type you used:

- [MEG-BIDS](https://doi.org/10.1038/sdata.2018.110)
- [EEG-BIDS](https://doi.org/10.1038/s41597-019-0104-8)
- [iEEG-BIDS](https://doi.org/10.1038/s41597-019-0105-7)
- [NIRS-BIDS](https://doi.org/10.31219/osf.io/7nmcp)


# OSBuild

Build-Pipelines for Operating System Artifacts

OSBuild is a pipeline-based build system for operating system artifacts. It
defines a universal pipeline description and a build system to execute them,
producing artifacts like operating system images, working towards an image
build pipeline that is more comprehensible, reproducible, and extendable.

See the `osbuild(1)` man-page for details on how to run osbuild, the definition
of the pipeline description, and more.

## Project

 * **Website**: https://www.osbuild.org
 * **Bug Tracker**: https://github.com/osbuild/osbuild/issues
 * **Matrix**: #image-builder on [fedoraproject.org](https://matrix.to/#/#image-builder:fedoraproject.org)
 * **Mailing List**: image-builder@redhat.com
 * **Changelog**: https://github.com/osbuild/osbuild/releases

### Principles

1. [OSBuild stages](./stages) are never broken, only deprecated. The same manifest should always produce the same output.
2. [OSBuild stages](./stages) should be explicit whenever possible instead of e.g. relying on the state of the tree.
3. Pipelines are independent, so the tree is expected to be empty at the beginning of each.
4. Manifests are expected to be machine-generated, so OSBuild has no convenience functions to support manually created manifests.
5. The build environment is confined against accidental misuse, but this should not be considered a security boundary.
6. OSBuild may only use Python language features supported by the oldest target distribution.

### Contributing

Please refer to the [developer guide](https://osbuild.org/docs/developer-guide/index) to learn about our workflow, code style and more.

## Requirements

The requirements for this project are:

 * `bubblewrap >= 0.4.0`
 * `python >= 3.6`

Additionally, the built-in stages require:

 * `bash >= 5.0`
 * `coreutils >= 8.31`
 * `curl >= 7.68`
 * `qemu-img >= 4.2.0`
 * `rpm >= 4.15`
 * `tar >= 1.32`
 * `util-linux >= 235`
 * `skopeo`

At build-time, the following software is required:

 * `python-docutils >= 0.13`
 * `pkg-config >= 0.29`

Testing requires additional software:

 * `pytest`

## Build

Osbuild is a python script so it is not compiled.
To verify changes made to the code use included makefile rules:

 * `make lint` to run linter on top of the code
 * `make test-all` to run base set of tests
 * `sudo make test-run` to run extended set of tests (takes long time)

## Installation

Installing `osbuild` requires to not only install the `osbuild` module, but also
additional artifacts such as tools (i.e: `osbuild-mpp`) sources, stages, schemas
and SELinux policies.

For this reason, doing an installation from source is not trivial and the easier
way to install it is to create the set of RPMs that contain all these components.

This can be done with the `rpm` make target, i.e:

```sh
make rpm
```

A set of RPMs will be created in the `./rpmbuild/RPMS/noarch/` directory and can
be installed in the system using the distribution package manager, i.e:

```sh
sudo dnf install ./rpmbuild/RPMS/noarch/*.rpm
```

## Repository

 - **web**:   https://github.com/osbuild/osbuild
 - **https**: `https://github.com/osbuild/osbuild.git`
 - **ssh**:   `git@github.com:osbuild/osbuild.git`

## License

 - **Apache-2.0**
 - See LICENSE file for details.


[![PyPI version](https://badge.fury.io/py/covsirphy.svg)](https://badge.fury.io/py/covsirphy)
[![Downloads](https://static.pepy.tech/badge/covsirphy)](https://pepy.tech/project/covsirphy)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/covsirphy)](https://badge.fury.io/py/covsirphy)
[![GitHub license](https://img.shields.io/github/license/lisphilar/covid19-sir)](https://github.com/lisphilar/covid19-sir/blob/master/LICENSE)
[![Quality Check](https://github.com/lisphilar/covid19-sir/actions/workflows/test.yml/badge.svg)](https://github.com/lisphilar/covid19-sir/actions/workflows/test.yml)
[![Test Coverage](https://codecov.io/gh/lisphilar/covid19-sir/branch/master/graph/badge.svg?token=9Z8Z1UHY3I)](https://codecov.io/gh/lisphilar/covid19-sir)

# CovsirPhy introduction

[Documentation](https://lisphilar.github.io/covid19-sir/index.html)
| [Installation](https://lisphilar.github.io/covid19-sir/markdown/INSTALLATION.html)
| [Tutorial](<https://lisphilar.github.io/covid19-sir/01_data_preparation.html>)
| [API reference](https://lisphilar.github.io/covid19-sir/covsirphy.html)
| [GitHub](https://github.com/lisphilar/covid19-sir)
| [Qiita (Japanese)](https://qiita.com/tags/covsirphy)

CovsirPhy is a Python library for infectious disease (COVID-19: Coronavirus disease 2019, Monkeypox 2022) data analysis with phase-dependent SIR-derived ODE models. We can download datasets and analyze them easily. Scenario analysis with CovsirPhy enables us to make data-informed decisions.

## Inspiration

* Monitor the spread of COVID-19/Monkeypox with SIR-derived ODE models
* Predict the number of cases in each country/province
* Find the relationship of reproductive number and measures taken by each country

If you have ideas or need new functionalities, please join this project.
Any suggestions with [Github Issues](https://github.com/lisphilar/covid19-sir/issues/new/choose) and [Twitter: @lisphilar](https://twitter.com/lisphilar) are always welcomed. Questions are also great. Please refer to [Guideline of contribution](https://lisphilar.github.io/covid19-sir/CONTRIBUTING.html).

## Installation

The latest stable version of CovsirPhy is available at [PyPI (The Python Package Index): covsirphy](https://pypi.org/project/covsirphy/) and supports Python 3.8 or newer versions. Details are explained in [Documentation: Installation](https://lisphilar.github.io/covid19-sir/markdown/INSTALLATION.html).

```Bash
pip install --upgrade covsirphy
```

## Demo

Quickest tour of CovsirPhy is here. The following codes analyze the records in Japan.

```Python
import covsirphy as cs
# Data preparation,time-series segmentation, parameter estimation with SIR-F model
snr = cs.ODEScenario.auto_build(geo="Japan", model=cs.SIRFModel)
# Check actual records
snr.simulate(name=None);
# Show the result of time-series segmentation
snr.to_dynamics(name="Baseline").detect();
# Perform simulation with estimated ODE parameter values
snr.simulate(name="Baseline");
# Predict ODE parameter values (30 days from the last date of actual records)
snr.build_with_template(name="Predicted", template="Baseline");
snr.predict(days=30, name="Predicted");
# Perform simulation with estimated and predicted ODE parameter values
snr.simulate(name="Predicted");
# Add a future phase to the baseline (ODE parameters will not be changed)
snr.append();
# Show created phases and ODE parameter values
snr.summary()
# Compare reproduction number of scenarios (predicted/baseline)
snr.compare_param("Rt");
# Compare simulated number of cases
snr.compare_cases("Confirmed");
# Describe representative values
snr.describe()
```

Output of `snr.simulate(name="Predicted");`

## Tutorial

Tutorials of functionalities are included in the [CovsirPhy documentation](https://lisphilar.github.io/covid19-sir/index.html).

* [Data preparation](https://lisphilar.github.io/covid19-sir/01_data_preparation.html)
* [Data Engineering](https://lisphilar.github.io/covid19-sir/02_data_engineering.html)
* [SIR-derived ODE models](https://lisphilar.github.io/covid19-sir/03_ode.html)
* [Phase-dependent SIR models](https://lisphilar.github.io/covid19-sir/04_phase_dependent.html)
* [Scenario analysis](https://lisphilar.github.io/covid19-sir/05_scenario_analysis.html)
* [ODE parameter prediction](https://lisphilar.github.io/covid19-sir/06_prediction.html)

## Release notes

Release notes are [here](https://github.com/lisphilar/covid19-sir/releases). Titles & links of issues are listed with acknowledgement.

We can see the release plan for the next stable version in [milestone page of the GitHub repository](https://github.com/lisphilar/covid19-sir/milestones). If you find a highly urgent matter, please let us know via [issue page](https://github.com/lisphilar/covid19-sir/issues).

## Developers

CovsirPhy library is developed by a community of volunteers. Please see the full list [here](https://github.com/lisphilar/covid19-sir/graphs/contributors).

This project started in Kaggle platform. Hirokazu Takaya ([@lisphilar](<https://www.kaggle.com/lisphilar>)) published [Kaggle Notebook: COVID-19 data with SIR model](https://www.kaggle.com/lisphilar/covid-19-data-with-sir-model) on 12Feb2020 and developed it, discussing with Kaggle community. On 07May2020, "covid19-sir" repository was created. On 10May2020, `covsirphy` version 1.0.0 was published in GitHub. The first release in PyPI (version 2.3.0) was on 28Jun2020. Many APIs were reviewed via 2.x series and version 3.0.0 was released on 12May2023.

## Support

Please support this project as a developer (or a backer).
[![Become a backer](https://opencollective.com/covsirphy/tiers/backer.svg?avatarHeight=36&width=600)](https://opencollective.com/covsirphy)

## License: Apache License 2.0

Please refer to [LICENSE](https://github.com/lisphilar/covid19-sir/blob/master/LICENSE) file.

## Citation

Please cite this library as follows with version number (`import covsirphy as cs; cs.__version__`).

**Hirokazu Takaya and CovsirPhy Development Team (2020-2024), CovsirPhy version [version number]: Python library for COVID-19 analysis with phase-dependent SIR-derived ODE models, [https://github.com/lisphilar/covid19-sir](https://github.com/lisphilar/covid19-sir)**

This is the output of `covsirphy.__citation__`.

```Python
import covsirphy as cs
cs.__citation__
```

**We have no original papers the author and contributors wrote, but note that some scientific approaches, including SIR-F model, S-R change point analysis, phase-dependent approach to SIR-derived models, were developed in this project.**


## KC3改 Translations ![npm test](https://github.com/KC3Kai/kc3-translations/workflows/npm/badge.svg)

![KC3改 Logo](http://puu.sh/h4Gbb.png)

If interested, you can translate KC3改 to your own language.

## Procedure
#### With GitHub (fork)
* Fork `KC3Kai/kc3-translations` repository
* Clone your repo to your local computer
* Make sure `KC3Kai/kc3-translations` is also added as secondary remote
* Go to the `data` directory where you can see language folders, copy one into a new language
* Go inside your new language folder, and edit the `json` files
* Commit your new language files, and push to your own GitHub repo
* Create a pull request to `KC3Kai/kc3-translations`

**Promotion to direct write-access**

You will first need your own translation fork (as indicated above), and has created a pull-request for the initial translation files for your language. If the language already has a directory, and you want to help existing translators, you will still need a fork, and a pull-request just to see a sample of your work before we can add you on the direct-write access team.

Once these requirements are fulfilled, you may request for promotion:
* Create a new issue on the [main `KC3Kai` repo](https://github.com/KC3Kai/KC3Kai/issues)
* Add `others` and `translation` as label
* Subject/Title: **Request translation repo access**

Then wait until reply, and an invitation to the KC3Kai organization, under the *Translation* team.

#### Without GitHub (manual)
* Download the ZIP file from the [project home](https://github.com/KC3Kai/kc3-translations)
* Extract the ZIP on your local computer
* Find the directory `\src\data\translations`
* Inside, you can see an `en` folder, copy it into a new language
* Go inside your new language folder, and edit the `json` files
* [Create an issue on the main repo](https://github.com/KC3Kai/KC3Kai/issues/new), add a label "translation"
   * You can name the subject as: "*\[Translation: (lang_code)\] (what_was_translated)*"
   * For example: "*[Translation: JP] Settings Page*"
* Paste the translation files / links to its pastebin or puush or any preferred text transmission tool.

## Notes
The only required file is `terms.json`. If there are missing files from your translation directory, it will extend the English translations by default.
If there are missing keys in your translated JSON file, it will also extend the English ones by default.

By enabling the setting `Dev-Only Strategy Pages`, you can check the missing keys in your selected language via Strategy Room Dev-Only Translations page.

#### About `terms_extend.json`
These are terms only used by Strategy Room pages. As you know, it's incomplete for now, and welcomed as always that needs help to extract hard-coded texts in main repo (html/js files) into this JSON file.
About the definition of term name, to avoid conflicts and add readabilty, current basic naming scheme is: `PageName` + `RawTextInShort`, without any delimeter in upper camel-case.
Even some texts are looked similar or nearly the same, but they may be translated in different way in other languages, so basically do not merge or share them in one name term across different pages. Already existing common terms in `terms.json` can be re-used, not necessary to add new term in that case.

#### About `quests.json`
There are some keys used as meta attributes of quests, such as `code`, `unlock` and `tracking`.
They are not needed to be translated, and can be just omitted in the `quests.json` of your own language, as long as you do not intend to use different values from English's.

By the way, the values of `code` in English file are from the quest identifier system defined by [wikiwiki.jp/kancolle](https://wikiwiki.jp/kancolle/%E4%BB%BB%E5%8B%99), which also used by other English wikis.

#### About `en/ctype.json`
These Japanese ship class names are basically extracted from in-game ship library banner image. They are supposed to be automatically translated using the same mechanism of ship name translation. So just define your translations in your `ships.json` and `ship_affix.json`, and keep the `ctype.json` being empty (only containing `[]`) in your directory.

#### About `servers.json`
Similar with the `quests.json`, there are keys used as meta data which are not needed to be translated.
The only key required to be translated is `name`, omitting other keys in your language is fine.

To update the IP address of servers, suggested to execute the script in `tools` folder, instead of manual modification.

#### About `quotes.json`
It's a bit complicated about our subtitle mechanism, you might need to learn more from our source code, before another detailed document is ready.

#### Special case for JP
Since game data is already in Japanese, `ships.json` and `items.json` do not need to be translated, but will still need to have the files exist, just containing an empty JSON object `{}`. For more information, please read [this](data/jp/README.md).

#### Kanji or Unicode character in your JSON
Generally JSON files are recommended to be saved in Unicode `UTF-8` charset.

For Kanji or any non-ascii Unicode character, such as quest descriptions like

`Have Tenryuu (\u5929\u9f8d) and Tatsuta (\u9f8d\u7530) in your main fleet.`

It is your choice if you want to still include Kanji representations like `(\u5929\u9f8d)` in your own language.

But do remember to use escaped chars like `\uHHHH` instead of literal Kanji or non-ascii chars, if you are not sure how to save your JSON file in `UTF-8` charset.

## Notes & Rules for Specific Languages

For several languages, there are extra notes and rules:

* [Simplified Chinese (scn)](data/scn/README.md)
* [Traditional Chinese (tcn)](data/tcn/readme.md)
* [Japanese (jp)](data/jp/README.md)

Make sure to read them before contribution.


# eBook Reader Dictionaries

[![Update dictionaries](https://github.com/BoboTiG/ebook-reader-dict/workflows/Update%20dictionaries/badge.svg)](https://github.com/BoboTiG/ebook-reader-dict/actions?query=workflow%3A%22Update+dictionaries%22) [![Update local-specific data](https://github.com/BoboTiG/ebook-reader-dict/actions/workflows/auto-update-data.yml/badge.svg)](https://github.com/BoboTiG/ebook-reader-dict/actions/workflows/auto-update-data.yml) [![Word of the day](https://github.com/BoboTiG/ebook-reader-dict/actions/workflows/daily.yml/badge.svg)](https://github.com/BoboTiG/ebook-reader-dict/actions/workflows/daily.yml)

- [Catalan](docs/ca/README.md)
- [Danish](docs/da/README.md)
- [English](docs/en/README.md)
- [French](docs/fr/README.md) ([news](https://www.mobileread.com/forums/showthread.php?t=330223&page=2))
- [German](docs/de/README.md)
- [Greek](docs/el/README.md)
- [Italian](docs/it/README.md)
- [Norway](docs/el/README.md)
- [Portuguese](docs/pt/README.md)
- [Romanian](docs/ro/README.md)
- [Russian](docs/ru/README.md)
- [Spanish](docs/es/README.md)
- [Swedish](docs/sv/README.md)

## Users

- [all] [AUR packages](https://aur.archlinux.org/packages?O=0&SeB=nd&K=stardict-bobotig&outdated=&SB=p&SO=d&PP=50&submit=Go) by [Raffaele Mancuso @raffaem](https://github.com/raffaem)
- [FR] [Dico alto](https://dicoalto.com/) by [Snawei](https://fr.wiktionary.org/wiki/Utilisateur:Snawei)

## Development

Setup a virtual environment:

```console
$ python3.12 -m venv venv

# For Linux and Mac users
$ . venv/bin/activate

# For Windows users
$ . venv/Scripts/activate
```

Install, or update, dependencies:

```console
$ python -m pip install -U pip
$ python -m pip install -r requirements-tests.txt
```

Run tests:

```console
$ python -m pytest --doctest-modules wikidict tests
```

Run linters, and quality checkers, before submitting a pull-request:

```console
$ ./check.sh
```

## Contributors ✨

[![All Contributors](https://img.shields.io/badge/all_contributors-15-orange.svg?style=flat-square)](#contributors-)

Thanks go to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

Mickaël Schoentgen
🐛 💻 📖 📆

Nicolas Froment
🐛 💻 📖 📆

Attilio
💻

chopinesque
💻

Saeed Rasooli
🚇

Matthias C. Hormann
💻

tjader
💻

Victor
💻

John Koll
🌍

Marta Malberti
🌍

Arsenii Chaplinskii
🌍

Ander Romero
🌍

Dongchen Yue | 岳东辰
🌍

Johan Larsson
💻

kyxap
📖

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!