# Substra-Backend [![Python](https://github.com/Substra/substra-backend/actions/workflows/python.yml/badge.svg)](https://github.com/Substra/substra-backend/actions/workflows/python.yml) [![Helm](https://github.com/Substra/substra-backend/actions/workflows/helm.yml/badge.svg)](https://github.com/Substra/substra-backend/actions/workflows/helm.yml) [![Docker build](https://github.com/Substra/substra-backend/actions/workflows/docker-build.yaml/badge.svg)](https://github.com/Substra/substra-backend/actions/workflows/docker-build.yaml) [![Doc](https://img.shields.io/badge/read-docs-purple?logo=mdbook)]("https://docs.substra.org/") [![Slack](https://img.shields.io/badge/chat-on%20slack-blue?logo=slack)]("https://join.slack.com/t/substra-workspace/shared_invite/zt-1fqnk0nw6-xoPwuLJ8dAPXThfyldX8yA")

<picture>
  <object-position: center>
  <source media="(prefers-color-scheme: dark)" srcset="Substra-logo-white.svg">
  <source media="(prefers-color-scheme: light)" srcset="Substra-logo-colour.svg">

</picture>

Substra is an open source federated learning (FL) software. This specific repository is the backend of the Substra software.

# Setup

## Running a development instance

This section details how you can get started on running a local substra backend. For this you will need a running Kubernetes cluster and the [orchestrator](https://github.com/Substra/orchestrator) deployed in this cluster.

If you want to run the substra-backend with Skaffold you will need to add the twuni and bitnami helm repos:
```sh
helm repo add twuni https://helm.twun.io
helm repo add bitnami https://charts.bitnami.com/bitnami
```

To launch the substra backend:
```sh
skaffold dev
```
or
```sh
skaffold run
```

This will spawn several pods in two different namespaces to simulate several organizations: 'org-1' and 'org-2'.
Each organization will have:
- Postgres database
- Redis message broker
- Docker registry
- Kaniko cache warmer
- [Celery beat](./charts/substra-backend/templates/deployment-scheduler.yaml)
- 2 celery workers (a [worker](./charts/substra-backend/templates/statefulset-worker.yaml) executing tasks and a [scheduler](./charts/substra-backend/templates/deployment-scheduler-worker.yaml) restarting hanging tasks and performing cleanup tasks)
- a [_Job_](./charts/substra-backend/templates/job-migrations.yaml) to run migrations and add accounts.
- the [API backend](./charts/substra-backend/templates/deployment-server.yaml)
- the [events backend](./charts/substra-backend/templates/deployment-events.yaml) converting events from the orchestrator into celery tasks

### VSCode

For Visual Studio resources for substra-backend, see [this page](./docs/vscode.md)

### Use dev profile

Use `skaffold [run|dev] -p dev` in order to take advantage of:
- Pre-installed dev tools: debugger, profiler
- Test modules: fixtures, assets factory, utils
- Run application container as root: install packages, edit files, restart services

### Use debugger

Add a breakpoint, then attach to the `backend-server` container.

```sh
kubectl attach -it -n org-1 $(kubectl get pods -o=name -n org-1 -l app.kubernetes.io/name=substra-backend-server)
```

Use `Ctrl+p` then `Ctrl+q` to detach without killing the container.

See: https://docs.python.org/fr/3/library/pdb.html

### Running on multiple kubernetes nodes

The backend can be deployed on a kubernete cluster running more than one kubernetes node.

To run the worker and the server on two different nodes:
Label the nodes:
```sh
kubectl label nodes <node_name> server=true
kubectl label nodes <node_name> worker=true
```

Deploy the backend with:
    - `skaffold run -p add-worker-server-node-selectors` to run 1 server pod and 1 worker pod on 2 separate kubernetes nodes. The nodes need to be labelled (see `add-worker-server-node-selectors.yaml`)
    - `skaffold run -p spread-workers` to spread the workers across the `x` different nodes.

You can draw from this documentation to set up the config according to your needs: number of kubernetes nodes, number of worker replicas, way to spread workers across the kubernetes nodes.

### Run with servermedia profile

To enable storing datasamples on the servermedia PVC instead of minio, use `skaffold run -p servermedias`. The PVs need to be created prior to requesting them inside the PVC:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: backend-org-1-substra-backend-servermedias
  labels:
    app.kubernetes.io/instance: < Release.Name > # here it will be backend-org-1
spec:
  storageClassName: manual
  persistentVolumeReclaimPolicy: Delete # You can also use "Retain" or "Recycle"
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/org-1/servermedias"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: backend-org-2-substra-backend-servermedias
  labels:
    app.kubernetes.io/instance: < Release.Name > # here it will be backend-org-2
spec:
  storageClassName: manual
  persistentVolumeReclaimPolicy: Delete # You can also use "Retain" or "Recycle"
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/org-2/servermedias"
```

## Django management

To get access to the [Django management tool](https://docs.djangoproject.com/en/2.2/ref/django-admin), spawn a shell in the running container:
```sh
kubectl -n org-1 exec -i -t $(kubectl -n org-1 get pods -l=app.kubernetes.io/name=substra-backend-server -o name) -c substra-backend -- /bin/bash
```

This will also gives you access to the [celery CLI](https://docs.celeryproject.org/en/stable/reference/cli.html). You can issue `celery inspect active_queues` to examine consumed queues.

## Running substrapp in a venv
If your useCase only involves the django app (not the eventapp, nor the compute engine for instance), you may easily run it and serve the API without a k8s cluster.
See [this page](./docs/localdev.md)

## Execute unit tests

Unit tests require to have a running postgres instance.

If it is not already running, you could start a new DB, using the command below.
This command will only work if you have a local docker daemon running.
It is recommended to use the same version as the one defined in the charts.

```sh
make db
```

Alternatively, you could also set the environment variables defined in `backend.settings.test` to provide any DB info.

```sh
export BACKEND_DB_NAME=<db_name> \
    BACKEND_DB_USER=<db_user> \
    BACKEND_DB_PWD=<db_password> \
    BACKEND_DB_HOST=<db_host> \
    BACKEND_DB_PORT=<db_port>
```

Make sure you have the requirements installed:
```sh
pip install -r backend/dev-requirements.txt
```

Then launch unit tests:
```sh
make test
```

A coverage report can be obtained with:
```sh
make coverage
```

Should you prefer an HTML report, you can use `coverage html` from the `backend` directory.

## Updating gRPC definitions

Assuming you have cloned the [orchestrator](https://github.com/owkin/orchestrator) in `<orchestrator_root>`:
```sh
cd backend
ORCHESTRATOR_ROOT=<orchestrator_root> make orchestrator-grpc
```

Note that this requires grpcio-tools, which should be available if you installed requirements (see above).

## Accessing the app

On deployment, several user accounts are created (for [./values/backend-org-1.yaml](org1) and [./values/backend-org-2.yaml](org2)).

The sample credentials for org1 are:
- user: `org-1`
- pass: `p@sswr0d44`

Provided you have correctly setup your [network configuration](https://doc.substra.ai/setup/local_install_skaffold.html#network), you can use them to access the exposed API at http://substra-backend.org-1.com/

## Compatibility

Make sure you deploy this backend with a compatible ecosystem.
Always refer to the [compatibility table](https://docs.substra.org/en/stable/additional/release.html).

The recommended way to run a specific version (X.Y.Z) of substra-backend is to execute:

```bash
SUBSTRA_BACKEND_VERSION=X.Y.Z
git checkout $SUBSTRA_BACKEND_VERSION
skaffold deploy --images substrafoundation/substra-backend:$SUBSTRA_BACKEND_VERSION
```

## Code formatting

You can opt into auto-formatting of code on pre-commit using [Black](https://github.com/psf/black).

This relies on hooks managed by [pre-commit](https://pre-commit.com/), which you can set up as follows.

Install [pre-commit](https://pre-commit.com/), then run:

```sh
pre-commit install
```

## How to generate the changelog

The changelog is managed with [towncrier](https://towncrier.readthedocs.io/en/stable/index.html).
To add a new entry in the changelog, add a file in the `changes` folder. The file name should have the following structure:
`<unique_id>.<change_type>`.
The `unique_id` is a unique identifier, we currently use the PR number.
The `change_type` can be of the following types: `added`, `changed`, `removed`, `fixed`.

To generate the changelog (for example during a release), use the following command (you must have the dev dependencies installed):

```
towncrier build --version=<x.y.z>
```

You can use the `--draft` option to see what would be generated without actually writing to the changelog (and without removing the fragments).

## Companion repositories

The Substra platform is built from several components (see the [architecture](https://doc.substra.ai/architecture.html) documentation for a comprehensive overview):

- [orchestrator](https://github.com/Substra/orchestrator) contains the orchestration logic of a federated learning deployment
- [substra-frontend](https://github.com/Substra/substra-frontend) is the frontend consuming the API exposed by the backend
- [substra-tests](https://github.com/Substra/substra-tests) is the Substra end to end test suite

## Running the backend on arm64 architecture (apple chip)

Tested with:

* python 3.9.4
* pip 22.0.4

Currently, only the dev mode is supported with this architecture.

1. uwsgi

When using pyenv-virtualenv, the python library might not be linked correctly into the python directory. Adding a manual link `sudo ln -s` to the expected folder will solve the issue:
```
sudo ln -s /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9.4/lib/python3.9/config-3.9-darwin <INSERT YOUR NOT-FOUND-PATH HERE WITHOUT libpython3.9.a FILENAME>
```

[Link to the Gihtub issue giving the hack](https://github.com/unbit/uwsgi/issues/2270#issuecomment-844702000)

2. Deploy with `skaffold run -p dev,arm64`

3. If you're using `conda` to manage your Python environments, you need to install `grpcio` from `conda`:
```
pip uninstall grpcio
conda install -c conda-forge grpcio
```

## License

This project is developed under the Apache License, Version 2.0 (Apache-2.0), located in the [LICENSE](./LICENSE) file.


# Qiskit IBM Quantum Provider (_NOW DEPRECATED_)

[![License](https://img.shields.io/github/license/Qiskit/qiskit-ibm-provider.svg?style=popout-square)](https://opensource.org/licenses/Apache-2.0)
[![CI](https://github.com/Qiskit/qiskit-ibm-provider/actions/workflows/ci.yml/badge.svg)](https://github.com/Qiskit/qiskit-ibm-provider/actions/workflows/ci.yml)
[![](https://img.shields.io/github/release/Qiskit/qiskit-ibm-provider.svg?style=popout-square)](https://github.com/Qiskit/qiskit-ibm-provider/releases)
[![](https://img.shields.io/pypi/dm/qiskit-ibm-provider.svg?style=popout-square)](https://pypi.org/project/qiskit-ibm-provider/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Coverage Status](https://coveralls.io/repos/github/Qiskit/qiskit-ibm-provider/badge.svg?branch=main)](https://coveralls.io/github/Qiskit/qiskit-ibm-provider?branch=main)

---

**_PLEASE NOTE:_** _As of version 0.11.0, released in April 2024, `qiskit-ibm-provider` has been deprecated
with its support ending and eventual archival being no sooner than 3 months from that date. The
function provided by `qiskit-ibm-provider` has been moved to [qiskit-ibm-runtime](https://github.com/Qiskit/qiskit-ibm-runtime).
Please see the [Runtime migration Guide] for more details. We encourage you
to migrate over at your earliest convenience._

---

**Qiskit** is an open-source SDK for working with quantum computers at the level of circuits, algorithms, and application modules.

This project contains a provider that allows accessing the **[IBM Quantum]**
systems and simulators.

## Migrating from qiskit-ibmq-provider

If you are familiar with the `qiskit-ibmq-provider` repository, check out the [migration guide].

## Installation

You can install the provider using pip:

```bash
pip install qiskit-ibm-provider
```

## Provider Setup

1. Create an IBM Quantum account or log in to your existing account by visiting the [IBM Quantum login page].

1. Copy (and/or optionally regenerate) your API token from your
   [IBM Quantum account page].

1. Take your token from step 2, here called `MY_API_TOKEN`, and save it by calling `IBMProvider.save_account()`:

   ```python
   from qiskit_ibm_provider import IBMProvider
   IBMProvider.save_account(token='MY_API_TOKEN')
   ```

   The command above stores your credentials locally in a configuration file called `qiskit-ibm.json`. By default, this file is located in `$HOME/.qiskit`, where `$HOME` is your home directory.
   Once saved you can then instantiate the provider like below and access the backends:

   ```python
   from qiskit_ibm_provider import IBMProvider
   provider = IBMProvider()

   # display current supported backends
   print(provider.backends())

   # get IBM's simulator backend
   simulator_backend = provider.get_backend('ibmq_qasm_simulator')
   ```

### Load Account from Environment Variables
Alternatively, the IBM Provider can discover credentials from environment variables:
```bash
export QISKIT_IBM_TOKEN='MY_API_TOKEN'
```

Then instantiate the provider without any arguments and access the backends:
```python
from qiskit_ibm_provider import IBMProvider
provider = IBMProvider()
```

### Enable Account for Current Session
As another alternative, you can also enable an account just for the current session by instantiating the provider with the token.

```python
from qiskit_ibm_provider import IBMProvider
provider = IBMProvider(token='MY_API_TOKEN')
```

## Next Steps

Now you're set up and ready to check out some of the tutorials.
- [IBM Quantum Learning]
- [Qiskit]

## Contribution Guidelines

If you'd like to contribute to qiskit-ibm-provider, please take a look at our
[contribution guidelines]. This project adheres to Qiskit's [code of conduct].
By participating, you are expect to uphold to this code.

We use [GitHub issues] for tracking requests and bugs. Please use our [slack]
for discussion and simple questions. To join our Slack community, use the
invite link [here](https://docs.quantum.ibm.com/support#qiskit).

## Authors and Citation

The Qiskit IBM Quantum Provider is the work of [many people] who contribute to the
project at different levels. If you use Qiskit, please cite as per the included
[BibTeX file].

## License

[Apache License 2.0].

[IBM Quantum]: https://www.ibm.com/quantum-computing/
[IBM Quantum login page]:  https://quantum-computing.ibm.com/login
[IBM Quantum account page]: https://quantum-computing.ibm.com/account
[contribution guidelines]: https://github.com/Qiskit/qiskit-ibm-provider/blob/main/CONTRIBUTING.md
[code of conduct]: https://github.com/Qiskit/qiskit-ibm-provider/blob/main/CODE_OF_CONDUCT.md
[GitHub issues]: https://github.com/Qiskit/qiskit-ibm-provider/issues
[slack]: https://qiskit.slack.com
[many people]: https://github.com/Qiskit/qiskit-ibm-provider/graphs/contributors
[IBM Quantum Learning]: https://learning.quantum.ibm.com/catalog/tutorials
[Qiskit]: https://github.com/Qiskit/qiskit-tutorial
[BibTeX file]: https://github.com/Qiskit/qiskit/blob/master/Qiskit.bib
[Apache License 2.0]: https://github.com/Qiskit/qiskit-ibm-provider/blob/main/LICENSE.txt
[migration guide]: https://github.com/Qiskit/qiskit-ibm-provider/blob/6be5f3297ede75bb062b20601058b55a397668e3/docs/tutorials/Migration_Guide_from_qiskit-ibmq-provider.ipynb
[Runtime migration guide]: https://docs.quantum.ibm.com/api/migration-guides/qiskit-runtime


  A Document AI Package

**deep**doctection is a Python library that orchestrates document extraction and document layout analysis tasks using deep learning models. It does
not implement models but enables you to build pipelines using highly acknowledged libraries for object detection, OCR
and selected NLP tasks and provides an integrated framework for fine-tuning, evaluating and running models. For more
 specific text processing tasks use one of the many other great NLP libraries.

**deep**doctection focuses on applications and is made for those who want to solve real world problems related to
document extraction from PDFs or scans in various image formats.

Check the demo of a document layout analysis pipeline with OCR on
:hugs: [**Hugging Face spaces**](https://huggingface.co/spaces/deepdoctection/deepdoctection).

# Overview

**deep**doctection provides model wrappers of supported libraries for various tasks to be integrated into
pipelines. Its core function does not depend on any specific deep learning library. Selected models for the following
 tasks are currently supported:

 - Document layout analysis including table recognition in Tensorflow with [**Tensorpack**](https://github.com/tensorpack),
   or PyTorch with [**Detectron2**](https://github.com/facebookresearch/detectron2/tree/main/detectron2),
 - OCR with support of [**Tesseract**](https://github.com/tesseract-ocr/tesseract), [**DocTr**](https://github.com/mindee/doctr)
   (Tensorflow and PyTorch implementations available) and a wrapper to an API for a commercial solution,
 - Text mining for native PDFs with  [**pdfplumber**](https://github.com/jsvine/pdfplumber),
 - Language detection with [**fastText**](https://github.com/facebookresearch/fastText),
 - Deskewing and rotating images with [**jdeskew**](https://github.com/phamquiluan/jdeskew).
 - Document and token classification with all LayoutLM models provided by the
   [**Transformer library**](https://github.com/huggingface/transformers).
   (Yes, you can use any LayoutLM-model with any of the provided OCR-or pdfplumber tools straight away!).
 - Table detection and table structure recognition with
   [**table-transformer**](https://github.com/microsoft/table-transformer).
 - There is a small dataset for token classification [available](https://huggingface.co/datasets/deepdoctection/FRFPE)
   and a lot of new [tutorials](https://github.com/deepdoctection/notebooks/blob/main/Layoutlm_v2_on_custom_token_classification.ipynb)
   to show, how to train and evaluate this dataset using LayoutLMv1, LayoutLMv2, LayoutXLM and LayoutLMv3.
 - Comprehensive configuration of **analyzer** like choosing different models, output parsing, OCR selection.
   Check this [notebook](https://github.com/deepdoctection/notebooks/blob/main/Analyzer_Configuration.ipynb) or the
   [docs](https://deepdoctection.readthedocs.io/en/latest/tutorials/analyzer_configuration_notebook/) for more infos.
 - Document layout analysis and table recognition now runs with
   [**Torchscript**](https://pytorch.org/docs/stable/jit.html) (CPU) as well and [**Detectron2**](https://github.com/facebookresearch/detectron2/tree/main/detectron2) is not required
   anymore for basic inference.
 - [**new**] More angle predictors for determining the rotation of a document based on [**Tesseract**](https://github.com/tesseract-ocr/tesseract) and [**DocTr**](https://github.com/mindee/doctr)
   (not contained in the built-in Analyzer).
 - [**new**] Token classification with [**LiLT**](https://github.com/jpWang/LiLT) via
   [**transformers**](https://github.com/huggingface/transformers).
   We have added a model wrapper for token classification with LiLT and added a some LiLT models to the model catalog
   that seem to look promising, especially if you want to train a model on non-english data. The training script for
   LayoutLM can be used for LiLT as well and we will be providing a notebook on how to train a model on a custom dataset soon.

**deep**doctection provides on top of that methods for pre-processing inputs to models like cropping or resizing and to
post-process results, like validating duplicate outputs, relating words to detected layout segments or ordering words
into contiguous text. You will get an output in JSON format that you can customize even further by yourself.

Have a look at the [**introduction notebook**](https://github.com/deepdoctection/notebooks/blob/main/Get_Started.ipynb) in the
[notebook repo](https://github.com/deepdoctection/notebooks) for an easy start.

Check the [**release notes**](https://github.com/deepdoctection/deepdoctection/releases) for recent updates.

## Models

**deep**doctection or its support libraries provide pre-trained models that are in most of the cases available at the
[**Hugging Face Model Hub**](https://huggingface.co/deepdoctection) or that will be automatically downloaded once
requested. For instance, you can find pre-trained object detection models from the Tensorpack or Detectron2 framework
 for coarse layout analysis, table cell detection and table recognition.

## Datasets and training scripts

Training is a substantial part to get pipelines ready on some specific domain, let it be document layout analysis,
document classification or NER. **deep**doctection provides training scripts for models that are based on trainers
developed from the library that hosts the model code. Moreover, **deep**doctection hosts code to some well established
datasets like **Publaynet** that makes it easy to experiment. It also contains mappings from widely used data
formats like COCO and it has a dataset framework (akin to [**datasets**](https://github.com/huggingface/datasets) so that
 setting up training on a custom dataset becomes very easy. [**This notebook**](https://github.com/deepdoctection/notebooks/blob/main/Datasets_and_Eval.ipynb)
shows you how to do this.

## Evaluation

**deep**doctection comes equipped with a framework that allows you to evaluate predictions of a single or multiple
models in a pipeline against some ground truth. Check again [**here**](https://github.com/deepdoctection/notebooks/blob/main/Datasets_and_Eval.ipynb) how it is
done.

## Inference

Having set up a pipeline it takes you a few lines of code to instantiate the pipeline and after a for loop all pages will
be processed through the pipeline.

```python
import deepdoctection as dd
from IPython.core.display import HTML
from matplotlib import pyplot as plt

analyzer = dd.get_dd_analyzer()  # instantiate the built-in analyzer similar to the Hugging Face space demo

df = analyzer.analyze(path = "/path/to/your/doc.pdf")  # setting up pipeline
df.reset_state()                 # Trigger some initialization

doc = iter(df)
page = next(doc)

image = page.viz()
plt.figure(figsize = (25,17))
plt.axis('off')
plt.imshow(image)
```

![text](https://github.com/deepdoctection/deepdoctection/raw/master/docs/tutorials/_imgs/dd_rm_sample.png)

```
HTML(page.tables[0].html)
```

![table](https://github.com/deepdoctection/deepdoctection/raw/master/docs/tutorials/_imgs/dd_rm_table.png)

```
print(page.text)
```

![table](https://github.com/deepdoctection/deepdoctection/raw/master/docs/tutorials/_imgs/dd_rm_text.png)

## Documentation

There is an extensive [**documentation**](https://deepdoctection.readthedocs.io/en/latest/index.html#) available
containing tutorials, design concepts and the API. We want to present things as comprehensively and understandably
as possible. However, we are aware that there are still many areas where significant improvements can be made in terms
of clarity, grammar and correctness. We look forward to every hint and comment that increases the quality of the
documentation.

## Requirements

![requirements](https://github.com/deepdoctection/deepdoctection/raw/master/docs/tutorials/_imgs/requirements_deepdoctection.png)

Everything in the overview listed below the **deep**doctection layer are necessary requirements and have to be installed
separately.

- Linux or macOS. (Windows is not supported but there is a [Dockerfile](./docker/pytorch-cpu-jupyter/Dockerfile) available)
- Python >= 3.9
- 1.13 <= PyTorch  **or** 2.11 <= Tensorflow < 2.16. (For lower Tensorflow versions the code will only run on a GPU).
In general, if you want to train or fine-tune models, a GPU is required.
- **deep**doctection uses Python wrappers for [Poppler](https://poppler.freedesktop.org/) to convert PDF documents into
images.
- With respect to the Deep Learning framework, you must decide between [Tensorflow](https://www.tensorflow.org/install?hl=en)
  and [PyTorch](https://pytorch.org/get-started/locally/).
- [Tesseract](https://github.com/tesseract-ocr/tesseract) OCR engine will be used through a Python wrapper. The core
  engine has to be installed separately.

The following overview shows the availability of the models in conjunction with the DL framework.

| Task                                          | PyTorch | Torchscript    |  Tensorflow  |
|-----------------------------------------------|:-------:|----------------|:------------:|
| Layout detection via Detectron2/Tensorpack    |    ✅    | ✅ (CPU only)   | ✅ (GPU only) |
| Table recognition via Detectron2/Tensorpack   |    ✅    | ✅ (CPU only)   | ✅ (GPU only) |
| Table transformer via Transformers            |    ✅    | ❌              |      ❌       |
| DocTr                                         |    ✅    | ❌              |      ✅       |
| LayoutLM (v1, v2, v3, XLM) via Transformers   |    ✅    | ❌              | ❌            |

## Installation

We recommend using a virtual environment. You can install the package via pip or from source.

### Install with pip from PyPi

#### Minimal installation

If you want to get started with a minimal setting (e.g. running the **deep**doctection analyzer with
default configuration or trying the 'Get started notebook'), install **deep**doctection with

```
pip install deepdoctection
```

If you want to use the Tensorflow framework, please install Tensorpack separately. Detectron2 will not be installed
and layout models/ table recognition models will run with Torchscript on a CPU.

#### Full installation

The following installation will give you ALL models available within the Deep Learning framework as well as all models
that are independent of Tensorflow/PyTorch. Please note, that the dependencies are very complex. We try hard to keep
the requirements up to date though.

For **Tensorflow**, run

```
pip install deepdoctection[tf]
```

For **PyTorch**,

first install **Detectron2** separately as it is not distributed via PyPi. Check the instruction
[here](https://detectron2.readthedocs.io/en/latest/tutorials/install.html). Then run

```
pip install deepdoctection[pt]
```

This will install **deep**doctection with all dependencies listed above the **deep**doctection layer. Use this setting,
if you want to get started or want to explore all features.

If you want to have more control with your installation and are looking for fewer dependencies then
install **deep**doctection with the basic setup only.

```
pip install deepdoctection
```

This will ignore all model libraries (layers above the **deep**doctection layer in the diagram) and you
will be responsible to install them by yourself. Note, that you will not be able to run any pipeline with this setup.

For further information, please consult the [**full installation instructions**](https://deepdoctection.readthedocs.io/en/latest/install/).

### Installation from source

Download the repository or clone via

```
git clone https://github.com/deepdoctection/deepdoctection.git
```

To get started with **Tensorflow**, run:

```
cd deepdoctection
pip install ".[tf]"
```

Installing the full **PyTorch** setup from source will also install **Detectron2** for you:

```
cd deepdoctection
pip install ".[source-pt]"
```

### Running a Docker container from Docker hub

Starting from release `v.0.27.0`, pre-existing Docker images can be downloaded from the
[Docker hub](https://hub.docker.com/r/deepdoctection/deepdoctection).

```
docker pull deepdoctection/deepdoctection:<release_tag>
```

To start the container, you can use the Docker compose file `./docker/pytorch-gpu/docker-compose.yaml`.
In the `.env` file provided, specify the host directory where **deep**doctection's cache should be stored.
This directory will be mounted. Additionally, specify a working directory to mount files to be processed into the
container.

```
docker compose up -d
```

will start the container.

## Credits

We thank all libraries that provide high quality code and pre-trained models. Without, it would have been impossible
to develop this framework.

## Problems

We try hard to eliminate bugs. We also know that the code is not free of issues. We welcome all issues relevant to this
repo and try to address them as quickly as possible. Bug fixes or enhancements will be deployed in a new release every 4
to 6 weeks.

## If you like **deep**doctection ...

 ...you can easily support the project by making it more visible. Leaving a star or a recommendation will help.

## License

Distributed under the Apache 2.0 License. Check [LICENSE](https://github.com/deepdoctection/deepdoctection/blob/master/LICENSE)
for additional information.


![Codabench logo](src/static/img/codabench_black.png) [![Circle CI](https://circleci.com/gh/codalab/codabench.svg?style=shield)](https://app.circleci.com/pipelines/github/codalab/codabench)

## What is Codabench?

Codabench is an open-source web-based platform that enables researchers, developers, and data scientists to collaborate, with the goal of advancing research fields where machine learning and advanced computation is used. Codabench helps to solve many common problems in the arena of data-oriented research through its online community where people can share worksheets and participate in competitions and benchmarks. It can be seen as a version 2 of [CodaLab Competitions](https://github.com/codalab/codalab-competitions).

To see Codabench in action, visit [codabench.org](https://www.codabench.org/).

## Documentation

- [Codabench Wiki](https://github.com/codalab/codabench/wiki)

## Quick installation (for Linux)

_To participate, or even organize your own benchmarks or competitions, **you don't need to install anything**, you just need to sign in an instance of the platform (e.g. [this one](https://www.codabench.org/)).
If you wish to configure your own instance of Codabench platform, here are the instructions:_

```
$ cp .env_sample .env
$ docker-compose up -d
$ docker-compose exec django ./manage.py migrate
$ docker-compose exec django ./manage.py generate_data
$ docker-compose exec django ./manage.py collectstatic --noinput
```

You can now login as username "admin" with password "admin" at http://localhost/

If you ever need to reset the database, use the script `./reset_db.sh`

For more information about installation, checkout [Codabench Basic Installation Guide](https://github.com/codalab/codabench/wiki/Codabench-Installation) and [How to Deploy Server](https://github.com/codalab/codabench/wiki/How-to-deploy-Codabench-on-your-server).

## License

Copyright (c) 2020-2022, Université Paris-Saclay.
This software is released under the Apache License 2.0 (the "License"); you may not use the software except in compliance with the License.

The text of the Apache License 2.0 can be found online at:
http://www.opensource.org/licenses/apache2.0.php

## Cite Codabench in your research

```
@article{codabench,
    title = {Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform},
    author = {Zhen Xu and Sergio Escalera and Adrien Pavão and Magali Richard and
              Wei-Wei Tu and Quanming Yao and Huan Zhao and Isabelle Guyon},
    journal = {Patterns},
    volume = {3},
    number = {7},
    pages = {100543},
    year = {2022},
    issn = {2666-3899},
    doi = {https://doi.org/10.1016/j.patter.2022.100543},
    url = {https://www.sciencedirect.com/science/article/pii/S2666389922001465}
}
```


hazen

Quality assurance framework for Magnetic Resonance Imaging

Explore the docs »

View repo
·
Report Bug
·
Request Feature

Please STAR this repo to receive updates about new versions of hazen!

---

## Overview

hazen is a command line tool (Python package) for performing automated analysis of magnetic resonance imaging (MRI) quality assurance
(QA) data. hazen consists of multiple [Tasks](hazenlib/tasks) which perform quantitative processing and analysis of
MRI phantom data. Currently, hazen supports the [ACR Large MRI Phantom](https://www.acraccreditation.org/-/media/acraccreditation/documents/mri/largephantomguidance.pdf)
and the MagNET Test Objects collection of phantoms.

The hazen Tasks provide the following measurements within these phantoms:
- Signal-to-noise ratio (SNR)
- Spatial resolution
- Slice position
- Slice width
- Uniformity
- Ghosting
- MR relaxometry

Each Task outputs numerical results to the user's terminal. Below is an output from the `hazen snr` Task performed on
some example MRI data:

```shell
hazen snr tests/data/snr/Siemens
{
  'snr_smoothing_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 173.97,
  'snr_smoothing_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_3_1': 177.91,
  'snr_smoothing_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 1698.21,
  'snr_smoothing_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_3_1': 1736.66,
  'snr_subtraction_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 220.73,
  'snr_subtraction_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 2154.69
}
```

The optional `--report` flag allows the user to generate diagrams that visualise the image processing performed by each hazen Task:

| `hazen snr tests/data/snr/Siemens --report` | `hazen acr_ghosting tests/data/acr/Siemens --report` |
|---------------------------------------------|------------------------------------------------------|
|  |  |

---

## Installation and usage

There are two main options for running hazen.
1. Install using Python and run directly via command line interface (CLI)
2. Run using the latest Docker container build

### 1) Python install and run (CLI)

hazen can be installed with Python 3.9, 3.10 or 3.11 versions via pip (Python 3.12 is not currently supported).

It is strongly recommended to use a virtual environment, that can be made using the following commands.

```bash
python3 -m venv hazen-venv
source hazen-venv/bin/activate
pip install hazen
# Check that hazen was correctly installed and see which version it is:
hazen --version
```

#### Updating hazen
If you already have an old version of hazen installed, upgrade to the latest version with:

```shell
source hazen-venv/bin/activate
pip install --upgrade pip
pip install --upgrade hazen
```

#### Running hazen via CLI
The CLI version of hazen is designed to be pointed at single folders containing DICOM file(s). Example datasets are
provided in the `tests/data/` directory. If you are using the Docker version of hazen (installation described below),
replace `hazen` with `hazen-app` in the following commands.

```bash
# To see the full list of available Tasks and optional arguments, enter:
hazen -h

# To perform the SNR Task on example data:
hazen snr tests/data/snr/Philips

# The `--report` option generates additional visualisation about the image processing measurement methods and is available
# for all Tasks. Example usage for the SNR Task, which returns images showing the regions used for SNR calculation.
hazen snr tests/data/snr/Philips --report

# a directory path can be provided to save the report images to:
hazen snr tests/data/snr/Philips --report ./report_images
```

### 2) Docker

The Docker version of hazen has been made available as it is easy to get up-and-running and is linked to the most recent
stable release. Refer to the [Docker installation instructions](https://docs.docker.com/engine/install) to install
Docker on your host computer.

The containerised version of hazen can be obtained from DockerHub (see commands below). For ease of use, it is
recommended to copy the `hazen-app` script to a location accessible on the PATH such as `/usr/local/bin`. This will
allow you to run hazen from any directory on your computer. Then, to use Docker hazen, simply run the `hazen-app` script
appended with the function you want to use (e.g.: `snr`).

In Terminal:

```shell
# Ensure Docker installed and running, then pull the latest hazen Docker container
docker pull gsttmriphysics/hazen:latest

# Command line output will look something like:
latest: Pulling from gsttmriphysics/hazen
Digest: sha256:18603e40b45f3af4bf45f07559a08a7833af92a6efe21cb7306f758e8eeab24a
Status: Image is up to date for gsttmriphysics/hazen:latest
docker.io/gsttmriphysics/hazen:latest

# Copy the 'hazen-app' executable file into your local bin folder
cd hazen
cp hazen-app /usr/local/bin

# Run hazen via Docker with the normal CLI inputs
hazen-app snr tests/data/snr/Siemens/

# Example command line output for the SNR Task:
{
  'snr_smoothing_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 173.97,
  'snr_smoothing_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_3_1': 177.91,
  'snr_smoothing_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 1698.21,
  'snr_smoothing_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_3_1': 1736.66,
  'snr_subtraction_measured_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 220.73,
  'snr_subtraction_normalised_SNR_seFoV250_2meas_slice5mm_tra_repeat_PSN_noDC_2_1': 2154.69
}
```

### Web Interface

Development of a [web interface for hazen](https://github.com/GSTT-CSC/hazen-web-app) is in progress.

---

## Contributing to hazen

### Users
Please [raise an Issue](https://github.com/GSTT-CSC/hazen/issues) for any of the following reasons:
- Problems installing or running hazen
- Suggestions for improvements
- Requests for new features

We have used hazen with MRI data from a handful of different MRI scanners, including multiple different vendors. If
hazen does not perform with your MRI data, or the results are unexpected, please raise an Issue.

### Developers
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for developer guidelines.

---


![](https://github.com/blaylockbk/Herbie/blob/main/images/logo_new/Herbie-logo.png?raw=True)

# Herbie: Retrieve NWP Model Data 🏁

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/herbie-data)
](https://pypi.org/project/herbie-data/) [![PyPI](https://img.shields.io/pypi/v/herbie-data)](https://pypi.python.org/pypi/herbie-data/)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/herbie-data)
](https://anaconda.org/conda-forge/herbie-data)
[![DOI](https://zenodo.org/badge/275214142.svg)](https://zenodo.org/badge/latestdoi/275214142)

[![Tests (Conda)](https://github.com/blaylockbk/Herbie/actions/workflows/tests-conda.yml/badge.svg)](https://github.com/blaylockbk/Herbie/actions/workflows/tests-conda.yml)
[![Tests (Python)](https://github.com/blaylockbk/Herbie/actions/workflows/tests-python.yml/badge.svg)](https://github.com/blaylockbk/Herbie/actions/workflows/tests-python.yml)
[![Documentation Status](https://readthedocs.org/projects/herbie/badge/?version=latest)](https://herbie.readthedocs.io/?badge=latest)
[![Conda Recipe](https://img.shields.io/badge/recipe-herbie--data-green.svg)](https://anaconda.org/conda-forge/herbie-data)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

![Conda Downloads](https://img.shields.io/conda/d/conda-forge/herbie-data)
![PyPI - Downloads](https://img.shields.io/pypi/dm/herbie-data?label=pypi|downloads)
![PyPI - License](https://img.shields.io/pypi/l/herbie-data)

## [📘 Documentation](https://herbie.readthedocs.io/) | [💬 Discussions](https://github.com/blaylockbk/Herbie/discussions) | [❔ Ask For Help](https://github.com/blaylockbk/Herbie/discussions/categories/q-a)

**Herbie** is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. NWP data is distributed in GRIB2 format which Herbie reads using xarray+cfgrib. Herbie also provides some extra features to help visualize and extract data.

Herbie helps you discover, download, and read data from:

- [High Resolution Rapid Refresh (HRRR)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hrrr.html) | [HRRR-Alaska](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hrrrak.html)
- [Rapid Refresh (RAP)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/rap.html)
- [Global Forecast System (GFS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/gfs.html)
- [Global Ensemble Forecast System (GEFS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/gefs.html)
- [ECMWF Open Data Forecasts](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/ecmwf.html) (IFS and AIFS)
- [Navy Global Environmental Model (NAVGEM)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/navgem.html)
- [North American Mesoscale Model (NAM)](https://github.com/blaylockbk/Herbie/blob/main/docs/user_guide/_model_notebooks/nam.ipynb)
- [National Blend of Models (NBM)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/nbm.html)
- [Rapid Refresh Forecast System (RRFS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/rrfs.html) _prototype_
- [Real-Time/Un-Restricted Mesoscale Analysis (RTMA/URMA)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/rtma.html)
- [Hurricane Analysis And Forecast System (HAFS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hafs.html)
- [High Resolution Deterministic Prediction System (HRDPS)](https://herbie.readthedocs.io/en/latest/user_guide/_model_notebooks/hrdps.html)

Much of this data is made available through the [NOAA Open Data Dissemination](https://www.noaa.gov/information-technology/open-data-dissemination) (NODD) program (formerly the Big Data Program) which has made weather data more accessible than ever before.

## Installation

The easiest way to instal Herbie and its dependencies is with [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) from conda-forge.

```bash
conda install -c conda-forge herbie-data
```

You may also create the provided Conda environment, **[`environment.yml`](https://github.com/blaylockbk/Herbie/blob/main/environment.yml)**.

```bash
# Download environment file
wget https://github.com/blaylockbk/Herbie/raw/main/environment.yml

# Modify that file if you wish.

# Create the environment
conda env create -f environment.yml

# Activate the environment
conda activate herbie
```

Alternatively, Herbie is published on PyPI and you can install it with pip, _but_ it requires some dependencies that you will have to install yourself:

- Python 3.9+
- [cURL](https://anaconda.org/conda-forge/curl)
- [eccodes](https://anaconda.org/conda-forge/eccodes), a requirement for [cfgrib](https://github.com/ecmwf/cfgrib).
- _Optional:_ [wgrib2](https://anaconda.org/conda-forge/wgrib2)

When those are installed within your environment, _then_ you can install Herbie with pip.

```bash
# Latest published version
pip install herbie-data

# ~~ or ~~

# Most recent changes
pip install git+https://github.com/blaylockbk/Herbie.git

# Dependecies for extra features
pip install herbie-data[extra]
```

## Capabilities

- Search for model output from different data sources.
- Download full GRIB2 files.
- Download subset GRIB2 files (by grib field).
- Read data with xarray.
- Read index file with Pandas.
- Extra features (herbie xarray accessors)
  - Extract data at a point
  - Get Cartopy coordinate references system
  - Plot data with Cartopy (very early development).

```mermaid
  graph TD;
      d1[(HRRR)] -..-> H
      d2[(RAP)] -.-> H
      d3[(GFS)] -..-> H
      d33[(GEFS)] -.-> H
      d4[(IFS)] -..-> H
      d44[(AIFS)] -..-> H
      d5[(NBM)] -.-> H
      d6[(RRFS)] -..-> H
      d7[(RTMA)] -.-> H
      d8[(URMA)] -..-> H
      H((Herbie))
      H --- .inventory
      H --- .download
      H --- .xarray

      style H fill:#d8c89d,stroke:#0c3576,stroke-width:4px,color:#000000
```

```python
from herbie import Herbie

# Herbie object for the HRRR model 6-hr surface forecast product
H = Herbie(
  '2021-01-01 12:00',
  model='hrrr',
  product='sfc',
  fxx=6
)

# Look at file contents
H.inventory()

# Download the full GRIB2 file
H.download()

# Download a subset, like all fields at 500 mb
H.download(":500 mb")

# Read subset with xarray, like 2-m temperature.
H.xarray("TMP:2 m")
```

## Data Sources

Herbie downloads model data from the following sources, but can be extended to include others:

- [NOMADS](https://nomads.ncep.noaa.gov/)
- [NOAA Open Data Dissemination Program (NODD)](https://www.noaa.gov/information-technology/open-data-dissemination) partners (i.e., AWS, Google, Azure).
- [ECMWF Open Data Forecasts](https://www.ecmwf.int/en/forecasts/datasets/open-data)
- University of Utah CHPC Pando archive
- Local file system

## Community

Having trouble using Herbie or have a question? [❔ GitHub Discussions/Ask For Help](https://github.com/blaylockbk/Herbie/discussions/categories/q-a)

Just want to talk about Herbie or have an idea? [💬 GitHub Discussions](https://github.com/blaylockbk/Herbie/discussions/categories)

See something that might be wrong? [🚑 GitHub Issues](https://github.com/blaylockbk/Herbie/issues)

Want to contribute? Great! I'd love your help.

- "Watch" this repo's discussions and issues.
- Participate in [💬 GitHub Discussions](https://github.com/blaylockbk/Herbie/discussions/categories) and answer questions.
- Share how you use Herbie in [🙌 GitHub Discussions/Show and tell](https://github.com/blaylockbk/Herbie/discussions/categories/show-and-tell)
- Open an issue or file a pull request to make Herbie even better!
- Contribute to documentation.
- Test latest releases and report issues.
- Read the [👨🏻💻 Disclaimer & Contributing Guide](https://herbie.readthedocs.io/en/stable/user_guide/housekeeping/disclaimer.html)

## How to Cite and Acknowledge

If Herbie played an important role in your work, please [tell me about it](https://github.com/blaylockbk/Herbie/discussions/categories/show-and-tell)! Also, consider including a citation or acknowledgement in your article or product.

**_Suggested Citation_**

> Blaylock, B. K. (YEAR). Herbie: Retrieve Numerical Weather Prediction Model Data (Version 20xx.x.x) [Computer software]. https://doi.org/10.5281/zenodo.4567540

**_Suggested Acknowledgment_**

> A portion of this work used code generously provided by Brian Blaylock's Herbie python package (Version 20xx.x.x) (https://doi.org/10.5281/zenodo.4567540)

## History

During my PhD at the University of Utah, I created, at the time, the [only publicly-accessible archive of HRRR data](http://hrrr.chpc.utah.edu/). Over 1,000 research scientists and professionals used that archive.

 Blaylock B., J. Horel and S. Liston, 2017: Cloud Archiving and Data Mining of High Resolution Rapid Refresh Model Output. Computers and Geosciences. 109, 43-50. https://doi.org/10.1016/j.cageo.2017.08.005.

Herbie was then developed to access HRRR data from that archive and was first used on the Open Science Grid.

 Blaylock, B. K., J. D. Horel, and C. Galli, 2018: High-Resolution Rapid Refresh Model Data Analytics Derived on the Open Science Grid to Assist Wildland Fire Weather Assessment. J. Atmos. Oceanic Technol., 35, 2213–2227, https://doi.org/10.1175/JTECH-D-18-0073.1.

In the later half of 2020, the HRRR dataset from 2014 to present was made available through the [NODD Open Data Dissemination Program](https://www.noaa.gov/information-technology/open-data-dissemination) (formerly NOAA's Big Data Program). The latest version of Herbie organizes and expands my original download scripts into a more coherent package with the extended ability to download data for other models from many different archive sources, and it will continues to evolve.

I originally released this package under the name “HRRR-B” because it only worked with the HRRR dataset; the “B” was for Brian. Since then, I have added the ability to download many more models including RAP, GFS, ECMWF, GEFS, and RRFS with the potential to add more models in the future. Thus, this package was renamed **_Herbie_**, named after one of my favorite childhood movies.

The University of Utah MesoWest group now manages a [HRRR archive in Zarr format](http://hrrr.chpc.utah.edu/). Maybe someday, Herbie will be able to take advantage of that archive.

**Thanks for using Herbie, and happy racing!**

🏁 Brian

- [🌐 Personal Webpage](http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/home.html)

P.S. If you like Herbie, check out my other repos:

- [🌎 GOES-2-go](https://github.com/blaylockbk/goes2go): A python package to download GOES-East/West data and make RGB composites.
- [🌡 SynopticPy](https://github.com/blaylockbk/SynopticPy): A python package to download mesonet data from the Synoptic API.
- [🔨 Carpenter Workshop](https://github.com/blaylockbk/Carpenter_Workshop): A python package with various tools I made that are useful (like easy funxtions to build Cartopy maps).
- [💬 Bubble Print](https://github.com/blaylockbk/BubblePrint): A silly little python package that gives your print statement's personality.
- [📜 MET Syntax](https://github.com/blaylockbk/vscode-met-syntax): An extension for Visual Studio Code that gives syntax highlighting for Model Evaluation Tools (MET) configuration files.

> **Note**: Alternative Download Tools
> As an alternative to Herbie, you can use [rclone](https://rclone.org/) to download files from AWS or GCP. I love rclone. Here is a short [rclone tutorial](https://github.com/blaylockbk/pyBKB_v3/blob/master/rclone_howto.md)

| [Visualize Structure](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=blaylockbk%2FHerbie) | [Star History](https://star-history.com/#blaylockbk/Herbie&Date) | [PyPI Download Statistics](https://pypistats.org/packages/herbie-data)


     Explore the docs »

---

Connexion is a modern Python web framework that makes spec-first and api-first development easy.
You describe your API in an [OpenAPI][OpenAPI] (or [Swagger][Swagger]) specification with as much
detail as you want and Connexion will guarantee that it works as you specified.

It works either standalone, or in combination with any ASGI or WSGI-compatible framework!

   📢 Connexion 3 was recently released! Read about the changes here »

## ✨ Features

Connexion provides the following functionality **based on your specification**:

- 🚏 **Automatic route registration**, no ``@route`` decorators needed
- 🔒 **Authentication**, split from your application logic
- 🔎 **Request and response validation** of headers, parameters, and body
- 📬 **Parameter parsing and injection**, no request object needed
- 📨 **Response serialization**, you can return regular Python objects
- 📺 **A Swagger UI console** with live documentation and ‘try it out’ feature
- 🧩 **Pluggability**, in all dimensions

Connexion also **helps you write your OpenAPI specification** and develop against it by providing a command line interface which lets you test and mock your specification.

```shell
   connexion run openapi.yaml
```

(back to top)

## 🫶 Sponsors

Sponsors help us dedicate time to maintain Connexion. Want to help?

Explore the options »

(back to top)

## 🪤 Why Connexion

With Connexion, you write the spec first. Connexion then calls your Python
code, handling the mapping from the specification to the code. This
incentivizes you to write the specification so that all of your
developers can understand what your API does, even before you write a
single line of code.

If multiple teams depend on your APIs, you can use Connexion to easily
send them the documentation of your API. This guarantees that your API will
follow the specification that you wrote. This is a different process from
the one offered by most frameworks, which generate a specification
*after* you've written the code.
Some disadvantages of generating specifications based on code is that
they often end up lacking details or mix your documentation with the implementation
logic of your application.

(back to top)

## ⚒ How to Use

### Installation

You can install connexion using pip:

```shell
    $ pip install connexion
```

Connexion provides 'extras' with optional dependencies to unlock additional features:

- `swagger-ui`: Enables a Swagger UI console for your application.
- `uvicorn`: Enables to run the your application using `app.run()` for
  development instead of using an external ASGI server.
- `flask`: Enables the `FlaskApp` to build applications compatible with the Flask
  ecosystem.

You can install them as follows:

```shell
    $ pip install connexion[swagger-ui]
    $ pip install connexion[swagger-ui,uvicorn]
```

(back to top)

### Creating your application

Connexion can be used either as a standalone application or as a middleware wrapping an existing
ASGI (or WSGI) application written using a different framework. The standalone application can be
built using either the `AsyncApp` or `FlaskApp`.

- The `AsyncApp` is a lightweight application with native asynchronous support. Use it if you
  are starting a new project and have no specific reason to use one of the other options.

  ```Python
      from connexion import AsyncApp

      app = AsyncApp(__name__)
  ```

- The `FlaskApp` leverages the `Flask` framework, which is useful if you're migrating from
  connexion 2.X or you want to leverage the `Flask` ecosystem.

  ```python
      from connexion import FlaskApp

      app = FlaskApp(__name__)
  ```

- The `ConnexionMiddleware` can be wrapped around any existing ASGI or WSGI application.
  Use it if you already have an application written in a different framework and want to add
  functionality provided by connexion

  ```python
      from asgi_framework import App
      from connexion import ConnexionMiddleware

      app = App(__name__)
      app = ConnexionMiddleware(app)
  ```

(back to top)

### Registering an API

While you can register individual routes on your application, Connexion really shines when you
register an API defined by an OpenAPI (or Swagger) specification.
The operation described in your specification is automatically linked to your Python view function via the ``operationId``

**run.py**

```python
   def post_greeting(name: str, greeting: str):  # Paramaeters are automatically unpacked
       return f"{greeting} {name}", 200          # Responses are automatically serialized

   app.add_api("openapi.yaml")
```

**openapi.yaml**

```yaml
   ...
   paths:
     /greeting/{name}:
       post:
         operationId: run.post_greeting
         responses:
           200:
             content:
               text/plain:
                 schema:
                   type: string
         parameters:
           - name: name
             in: path
             required: true
             schema:
               type: string
           - name: greeting
             in: query
             required: true
             schema:
               type: string
```

(back to top)

### Running your application

If you installed connexion using `connexion[uvicorn]`, you can run it using the
`run` method. This is only recommended for development:

```python
    app.run()
```

In production, run your application using an ASGI server such as `uvicorn`. If you defined your
`app` in a python module called `run.py`, you can run it as follows:

```shell
    $ uvicorn run:app
```

Or with gunicorn:

```shell
    $ gunicorn -k uvicorn.workers.UvicornWorker run:app
```

----

Now you're able to run and use Connexion!

See the [examples][examples] folder for more examples.

(back to top)

## 📜 Changes

A full changelog is maintained on the [GitHub releases page][Releases].

(back to top)

## 🤲 Contributing

We welcome your ideas, issues, and pull requests. Just follow the
usual/standard GitHub practices.

For easy development, install connexion using poetry with all extras, and
install the pre-commit hooks to automatically run black formatting and static analysis checks.

```shell
    pip install poetry
    poetry install --all-extras
    pre-commit install
```

You can find out more about how Connexion works and where to apply your changes by having a look
at our [architecture][Architecture].

Unless you explicitly state otherwise in advance, any non trivial
contribution intentionally submitted for inclusion in this project by you
to the steward of this repository shall be under the
terms and conditions of Apache License 2.0 written below, without any
additional copyright information, terms or conditions.

(back to top)

## 🙏 Thanks

We'd like to thank all of Connexion's contributors for working on this
project, Swagger/OpenAPI for their support, and Zalando for originally developing and releasing Connexion.

## 📚 Recommended Resources

About the advantages of working spec-first:

* [Blog Atlassian][Blog Atlassian]
* [API guidelines Zalando][API guidelines Zalando]
* [Blog ML6][Blog ML6]
* [Blog Zalando][Blog Zalando]

Tools to help you work spec-first:

* [Online swagger editor][Online swagger editor]
* [VS Code plugin][VS Code plugin]
* [Pycharm plugin][Pycharm plugin]

[OpenAPI]: https://openapis.org/
[Swagger]: http://swagger.io/open-source-integrations/
[Blog atlassian]: https://www.atlassian.com/blog/technology/spec-first-api-development
[Blog ML6]: https://blog.ml6.eu/why-we-decided-to-help-maintain-connexion-c9f449877083
[Blog Zalando]: https://engineering.zalando.com/posts/2016/12/crafting-effective-microservices-in-python.html
[API guidelines Zalando]: https://opensource.zalando.com/restful-api-guidelines/#api-first
[Online swagger editor]: https://editor.swagger.io/
[VS Code plugin]: https://marketplace.visualstudio.com/items?itemName=42Crunch.vscode-openapi
[Pycharm plugin]: https://plugins.jetbrains.com/plugin/14837-openapi-swagger-editor
[examples]: https://github.com/spec-first/connexion/blob/main/examples
[Releases]: https://github.com/spec-first/connexion/releases
[Architecture]: https://github.com/spec-first/connexion/blob/main/docs/images/architecture.png


# QMCkl: Quantum Monte Carlo Kernel Library

<img src="https://trex-coe.eu/sites/default/files/styles/responsive_no_crop/public/2022-01/QMCkl%20code.png?itok=UvOUClA5" width=200>

![Build Status](https://github.com/TREX-CoE/qmckl/actions/workflows/test-build.yml/badge.svg?branch=master)
The domain of quantum chemistry needs a library in which the main
kernels of Quantum Monte Carlo (QMC) methods are implemented. In the
library proposed in this project, we expose the main algorithms in a
simple language and provide a standard API and tests to enable the
development of high-performance QMCkl implementations taking
advantage of modern hardware.

See the [source code](https://github.com/TREX-CoE/qmckl/blob/master/org/qmckl.org)
to read the documentation.

To clone the repository, use:
```
git clone https://github.com/TREX-CoE/qmckl.git
```

# Installation

The simplest way to obtain the source files of QMCkl is to download a source
distribution. This particular repository is for maintainers, who write the kernels
in org-mode files and produce the source code and the documentation from these files.

## For maintainers

```
./autogen.sh
./configure --prefix=$PWD/_install

make
make check
```

A good practice is to make out-of-source builds, because you can easily find
out what files have been produced by the build system, and you can also work
with differently configured versions of the library at the same time.

For example, you can create a debug build compiled with gcc, and a fast build
compiled with Intel compilers:

```
./autogen.sh

mkdir -p _build_gcc_debug/_install
cd _build_gcc_debug
../configure --enable-debug --prefix=$PWD/_install
make -j
make -j check

cd ..

mkdir -p _build_intel_fast/_install
cd _build_intel_fast
../configure --prefix=$PWD/_install --enable-hpc --with-icc --with-ifort
make -j
make -j check
```

## For users

Obtain a source distribution.

To build the documentation version:

```
./configure
```

To build an optimized version with Intel compilers:
```
./configure \
   --with-icc \
   --with-ifort \
   --enable-hpc
```

To build an optimized version with GCC:
```
./configure \
  CC=gcc \
  CFLAGS="-g -O2 -march=native  -flto -fno-trapping-math -fno-math-errno -ftree-vectorize" \
  FC=gfortran \
  FCFLAGS="-g -O2 -march=native  -flto -ftree-vectorize" \
  --enable-hpc
```

Then, compile with:
```
make -j
make -j check
sudo make install
sudo make installcheck
```

## Python API

- [SWIG](https://www.swig.org) (>= 4.0) is required to build the Python API for maintainers

In order to install the `qmckl` Python package, first install the shared C library
`libqmckl` following the installation guide above and then run the following command:
```
make python-install
```

To test the installation, run
```
make python-test
```

Minimal example demonstrating the use of the `qmckl` Python API can be found in the
[test_api.py](https://github.com/TREX-CoE/qmckl/blob/master/python/test/test_api.py) file.

We highly recommend to use
[virtual environments](https://docs.python.org/3/tutorial/venv.html)
to avoid compatibility issues and to improve reproducibility.

## Installation procedure for Guix users

QMCkl can be installed with the [GNU Guix](https://guix.gnu.org) functional package manager.
The [qmckl.scm](https://github.com/TREX-CoE/qmckl/blob/master/tools/qmckl.scm)
Schema file contains the manifest specification for the `qmckl` installations.
It can be installed within the selected `$GUIX_PROFILE` as follows:

```
guix package \
	--profile=$GUIX_PROFILE 		\
	--load-path=<path_to_trexio_scm> 	\
	--cores=<n_cores>			\
	--install-from-file=qmckl.scm
```

where `<path_to_trexio_scm>` should point to a folder, which contains the TREXIO manifest file
[trexio.scm](https://github.com/TREX-CoE/trexio/blob/master/tools/trexio.scm)
(e.g. `~/trexio/tools/` if TREXIO repository was cloned under $HOME).

Installation procedures for both development version (`qmckl-dev`)
and stable releases (`qmckl-hpc`) are provided.
One can switch between them using the return value (last line)
in the `qmckl.scm` file.

## Linking to your program

The `make install` command takes care of installing the QMCkl shared library on the user machine.
Once installed, add `-lqmckl` to the list of compiler options.

In some cases (e.g. when using custom `prefix` during configuration), the QMCkl library might end up installed in a directory, which is absent in the default `$LIBRARY_PATH`.
In order to link the program against QMCkl, the search paths can be modified as follows:

`export LIBRARY_PATH=$LIBRARY_PATH:<path_to_qmckl>/lib`

(same holds for `$LD_LIBRARY_PATH`). The `<path_to_qmckl>` has to be replaced with the prefix used during the installation.

If your project relies on the CMake build system, feel free to use the
[FindQMCKL.cmake](https://github.com/TREX-CoE/qmckl/blob/master/cmake/FindQMCKL.cmake)
module to find and link the QMCkl library automatically.

## Verificarlo CI

Since Verificarlo should not be a dependency of QMCkl, all Verificarlo
functions are called only when the support is explicitely enabled (and ignored
by the preprocessor otherwise). To enable vfc_ci support, the library should be
configured with the following command :

```
./configure \
  CC="verificarlo-f" \
  FC="verificarlo-f" \
  --prefix=$PWD/_install \
  --enable-vfc_ci \
  --host=x86_64 \
```

where CC and FC are set to verificarlo-f, and support is explicitely enabled
with the --enable-vfc_ci flag. Configuring the library with the "standard"
command will cause all calls to Verificarlo related functions to be ignored,
and the library will be built as usual.

------------------------------

![European flag](https://trex-coe.eu/sites/default/files/inline-images/euflag.jpg)
[TREX: Targeting Real Chemical Accuracy at the Exascale](https://trex-coe.eu) project has received funding from the European Union’s Horizon 2020 - Research and Innovation program - under grant agreement no. 952165. The content of this document does not represent the opinion of the European Union, and the European Union is not responsible for any use that might be made of such content.


# norwegianblue

[![PyPI version](https://img.shields.io/pypi/v/norwegianblue.svg?logo=pypi&logoColor=FFE873)](https://pypi.org/project/norwegianblue/)
[![Supported Python versions](https://img.shields.io/pypi/pyversions/norwegianblue.svg?logo=python&logoColor=FFE873)](https://pypi.org/project/norwegianblue/)
[![PyPI downloads](https://img.shields.io/pypi/dm/norwegianblue.svg)](https://pypistats.org/packages/norwegianblue)
[![Test](https://github.com/hugovk/norwegianblue/actions/workflows/test.yml/badge.svg)](https://github.com/hugovk/norwegianblue/actions)
[![codecov](https://codecov.io/gh/hugovk/norwegianblue/branch/main/graph/badge.svg)](https://codecov.io/gh/hugovk/norwegianblue)
[![Licence](https://img.shields.io/github/license/hugovk/norwegianblue.svg)](LICENSE.txt)
[![Code style: Black](https://img.shields.io/badge/code%20style-Black-000000.svg)](https://github.com/psf/black)

Python interface to [endoflife.date](https://endoflife.date/docs/api/) to show
end-of-life dates for a number of products.

## Installation

### From PyPI

```bash
python3 -m pip install --upgrade norwegianblue
```

### With [pipx][pipx]

```bash
pipx install norwegianblue
```

[pipx]: https://github.com/pypa/pipx

### From source

```bash
git clone https://github.com/hugovk/norwegianblue
cd norwegianblue
python3 -m pip install .
```

## Example command-line use

Run `norwegianblue` or `eol`, they do the same thing.

Top-level help:

```console
$ eol --help
usage: eol [-h] [-f {html,json,md,markdown,pretty,rst,csv,tsv,yaml}] [-c {yes,no,auto}] [--clear-cache] [--show-title {yes,no,auto}] [-v] [-V] [-w]
           [--pretty | --md | --rst | --json | --csv | --tsv | --html | --yaml]
           [product ...]

CLI to show end-of-life dates for a number of products, from https://endoflife.date

For example:

* `eol python` to see Python EOLs
* `eol ubuntu` to see Ubuntu EOLs
* `eol centos fedora` to see CentOS and Fedora EOLs
* `eol all` or `eol` to list all available products

Something missing? Please contribute! https://endoflife.date/contribute

positional arguments:
  product               product to check, or 'all' to list all available (default: 'all')

options:
  -h, --help            show this help message and exit
  -f {html,json,md,markdown,pretty,rst,csv,tsv,yaml}, --format {html,json,md,markdown,pretty,rst,csv,tsv,yaml}
                        deprecated: use direct options instead: --html, --json, --md, --pretty, --rst, --csv, --tsv or
                        --yaml
  -c {yes,no,auto}, --color {yes,no,auto}
                        colour the output (default: auto)
  --clear-cache         clear cache before running
  --show-title {yes,no,auto}
                        show or hide product title, 'auto' to show title only for multiple products (default: auto)
  -v, --verbose         print extra messages to stderr
  -V, --version         show program's version number and exit
  -w, --web             open product page in web browser

formatters:
  --pretty              output in pretty (default)
  --md                  output in Markdown
  --rst                 output in reStructuredText
  --json                output in JSON
  --csv                 output in CSV
  --tsv                 output in TSV
  --html                output in HTML
  --yaml                output in YAML
```

List all available products with end-of-life dates:

```console
$ # eol all
$ # or:
```

```console
$ eol
akeneo-pim
alibaba-dragonwell
almalinux
alpine
amazon-cdk
...
```

Show end-of-life dates:

```console
$ norwegianblue python
┌───────┬────────────┬─────────┬────────────────┬────────────┬────────────┐
│ cycle │  release   │ latest  │ latest release │  support   │    eol     │
├───────┼────────────┼─────────┼────────────────┼────────────┼────────────┤
│ 3.12  │ 2023-10-02 │ 3.12.0  │   2023-10-02   │ 2025-04-02 │ 2028-10-02 │
│ 3.11  │ 2022-10-24 │ 3.11.6  │   2023-10-02   │ 2024-04-01 │ 2027-10-24 │
│ 3.10  │ 2021-10-04 │ 3.10.13 │   2023-08-24   │ 2023-04-05 │ 2026-10-04 │
│ 3.9   │ 2020-10-05 │ 3.9.18  │   2023-08-24   │ 2022-05-17 │ 2025-10-05 │
│ 3.8   │ 2019-10-14 │ 3.8.18  │   2023-08-24   │ 2021-05-03 │ 2024-10-14 │
│ 3.7   │ 2018-06-26 │ 3.7.17  │   2023-06-05   │ 2020-06-27 │ 2023-06-27 │
│ 3.6   │ 2016-12-22 │ 3.6.15  │   2021-09-03   │ 2018-12-24 │ 2021-12-23 │
│ 3.5   │ 2015-09-12 │ 3.5.10  │   2020-09-05   │   False    │ 2020-09-13 │
│ 3.4   │ 2014-03-15 │ 3.4.10  │   2019-03-18   │   False    │ 2019-03-18 │
│ 3.3   │ 2012-09-29 │ 3.3.7   │   2017-09-19   │   False    │ 2017-09-29 │
│ 2.7   │ 2010-07-03 │ 2.7.18  │   2020-04-19   │   False    │ 2020-01-01 │
│ 2.6   │ 2008-10-01 │ 2.6.9   │   2013-10-29   │   False    │ 2013-10-29 │
└───────┴────────────┴─────────┴────────────────┴────────────┴────────────┘
```

You can format in Markdown, ready for pasting in GitHub issues and PRs:

| cycle |  release   | latest  | latest release |  support   |    eol     |
| :---- | :--------: | :------ | :------------: | :--------: | :--------: |
| 3.12  | 2023-10-02 | 3.12.0  |   2023-10-02   | 2025-04-02 | 2028-10-02 |
| 3.11  | 2022-10-24 | 3.11.6  |   2023-10-02   | 2024-04-01 | 2027-10-24 |
| 3.10  | 2021-10-04 | 3.10.13 |   2023-08-24   | 2023-04-05 | 2026-10-04 |
| 3.9   | 2020-10-05 | 3.9.18  |   2023-08-24   | 2022-05-17 | 2025-10-05 |
| 3.8   | 2019-10-14 | 3.8.18  |   2023-08-24   | 2021-05-03 | 2024-10-14 |
| 3.7   | 2018-06-26 | 3.7.17  |   2023-06-05   | 2020-06-27 | 2023-06-27 |
| 3.6   | 2016-12-22 | 3.6.15  |   2021-09-03   | 2018-12-24 | 2021-12-23 |
| 3.5   | 2015-09-12 | 3.5.10  |   2020-09-05   |   False    | 2020-09-13 |
| 3.4   | 2014-03-15 | 3.4.10  |   2019-03-18   |   False    | 2019-03-18 |
| 3.3   | 2012-09-29 | 3.3.7   |   2017-09-19   |   False    | 2017-09-29 |
| 2.7   | 2010-07-03 | 2.7.18  |   2020-04-19   |   False    | 2020-01-01 |
| 2.6   | 2008-10-01 | 2.6.9   |   2013-10-29   |   False    | 2013-10-29 |

With options:

```console
$ eol nodejs --rst
.. table::

    ========  ============  ==========  ================  ============  ============  ======================================================================================
     cycle      release       latest     latest release     support         eol                                                link
    ========  ============  ==========  ================  ============  ============  ======================================================================================
     21        2023-10-17    21.1.0      2023-10-24        2024-04-01    2024-06-01
     20 LTS    2023-04-18    20.9.0      2023-10-24        2024-10-22    2026-04-30
     19        2022-10-18    19.9.0      2023-04-10        2023-04-01    2023-06-01
     18 LTS    2022-04-19    18.18.2     2023-10-13        2023-10-18    2025-04-30
     17        2021-10-19    17.9.1      2022-06-01        2022-04-01    2022-06-01
     16 LTS    2021-04-20    16.20.2     2023-08-09        2022-10-18    2023-09-11
     15        2020-10-20    15.14.0     2021-04-06        2021-04-01    2021-06-01
     14 LTS    2020-04-21    14.21.3     2023-02-16        2021-10-19    2023-04-30
     13        2019-10-22    13.14.0     2020-04-29        2020-04-01    2020-06-01
     12 LTS    2019-04-23    12.22.12    2022-04-05        2020-10-20    2022-04-30
     11        2018-10-23    11.15.0     2019-04-30        2019-04-01    2019-06-30
     10 LTS    2018-04-24    10.24.1     2021-04-06        2020-05-19    2021-04-30
     9         2017-10-31    9.11.2      2018-06-12        2018-06-30    2018-06-30
     8 LTS     2017-05-30    8.17.0      2019-12-17        2019-01-01    2019-12-31
     7         2016-10-25    7.10.1      2017-07-11        2017-06-30    2017-06-30
     6 LTS     2016-04-26    6.17.1      2019-04-03        2018-04-30    2019-04-30
     5         2015-10-30    5.12.0      2016-06-23        2016-06-30    2016-06-30
     4 LTS     2015-09-09    4.9.1       2018-03-29        2017-04-01    2018-04-30
     3         2015-08-04    3.3.1       2015-09-15        False         True          https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_IOJS.md#__LATEST__
     2         2015-05-04    2.5.0       2015-07-28        False         True          https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_IOJS.md#__LATEST__
     1         2015-01-20    1.8.4       2015-07-09        False         True          https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_IOJS.md#__LATEST__
    ========  ============  ==========  ================  ============  ============  ======================================================================================
```

## Example programmatic use

Return values are from the JSON responses documented in the API:
https://endoflife.date/docs/api/

```python
import norwegianblue

# Call the API
print(norwegianblue.norwegianblue())
print(norwegianblue.norwegianblue(product="ubuntu"))
print(norwegianblue.norwegianblue(format="json"))
```

## Why "Norwegian Blue"?

[The Norwegian Blue has reached end-of-life.](https://youtu.be/vnciwwsvNcc)


[![Actions Status][actions-badge]][actions-link]
[![PyPI version](https://badge.fury.io/py/modeci-mdf.svg)](https://badge.fury.io/py/modeci-mdf)
[![Documentation Status][rtd-badge]][rtd-link]
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# ModECI Model Description Format (MDF)
[**Click here for the full MDF documentation**](https://mdf.readthedocs.io)

**Note: MDF is still in development! See the [open issues related to the specification](https://github.com/ModECI/MDF/issues?q=is%3Aissue+is%3Aopen+label%3Aspecification) or go [here](http://modeci.org/#contactPage) to get in contact regarding MDF.**
*The MDF format was first proposed following a meeting organised at Princeton in July 2019 by Russ Poldrack of the Center for Reproducible Neuroscience (CRN) at Stanford and the [Brain Imaging Data Standard (BIDS)](https://bids.neuroimaging.io/) initiative. For more on the previous work in this area, see [here](https://github.com/OpenSourceBrain/PsyNeuLinkShowcase/tree/master/BIDS-MDF).*

## Overview

MDF is an open source, community-supported standard and associated library of tools for expressing computational models in a form that allows them to be exchanged between diverse programming languages and execution environments. The overarching aim is to provide a common format for models across [computational neuroscience](https://en.wikipedia.org/wiki/Computational_neuroscience), [cognitive science](https://cogsci.jhu.edu/about/#:~:text=What%20Is%20Cognitive%20Science%3F,are%20realized%20in%20the%20brain.) and [machine learning.](https://www.ibm.com/topics/machine-learning)

It consists of a specification for expressing models in serialized formats (currently [JSON](https://developer.mozilla.org/en-US/docs/Glossary/JSON), [YAML](https://circleci.com/blog/what-is-yaml-a-beginner-s-guide/) and [BSON](https://en.wikipedia.org/wiki/BSON) representations are supported, though others such as [HDF5](https://www.geeksforgeeks.org/hdf5-files-in-python/) are planned) and a set of Python tools for implementing a model described using MDF. The serialized formats can be used when importing a model into a supported target environment to execute it; and, conversely, when exporting a model built in a supported environment so that it can be re-used in other environments.

The MDF Python API can be used to create or load an MDF model for inspection and validation. It also includes a basic [execution engine](https://mdf.readthedocs.io/en/latest/api/_autosummary/modeci_mdf.execution_engine.html#module-modeci_mdf.execution_engine) for simulating models in the format. However, this is not intended to provide a efficient, general-purpose simulation environment, nor is MDF intended as a programming language. Rather, the primary purpose of the Python API is to facilitate and validate the exchange of models between existing environments that serve different communities. Accordingly, these Python tools include bi-directional support for importing to and exporting from widely-used programming environments in a range of disciplines, and for easily extending these to other environments.

## Development

The implementation and dissemination of the MDF language and associated tools is being carried out by the [Model Exchange and Convergence Initiative (ModECI)](http://modeci.org/), which has been supported by the [NSF Convergence Accelerator Program](https://www.nsf.gov/od/oia/convergence-accelerator/) (Track D: AI-Driven Innovation via Data and Model Sharing), as a publicly accessible [open-source project](https://github.com/ModECI/MDF). The initial design has been informed by a [series of workshops](http://modeci.org/#communityPage) involving developers of key software environments and other stakeholders in machine learning, cognitive science and neuroscience. Future workshops will address broadening of support to other domains in basic and applied science and technology development (e.g., population biology, medical informatics, structural and environmental monitoring, and complex systems control). Environments for which support is currently being developed include [PyTorch](https://pytorch.org), [ONNX](http://onnx.ai), [WebGME](https://webgme.org), [NeuroML](https://neuroml.org), [PsyNeuLink](http://www.psyneuln.deptcpanel.princeton.edu), and [ACT-R](http://act-r.psy.cmu.edu).

Fig 1: Some of the current and planned formats which MDF will interact with. Click on the image for more information.

Successful interfacing of MDF to existing disciplinary standards (such as [ONNX](http://onnx.ai) in machine learning, and [NeuroML](https://neuroml.org) in neuroscience) as well as general-purpose simulation environments (such as [WebGME](https://webgme.org)) will permit bridging between these environments, and translation to the broader set of environments supported by those standards (such as [Tensorflow](https://www.tensorflow.org) & [Keras](https://keras.io) in the case of ONNX, and [The Virtual Brain](https://www.thevirtualbrain.org) and [SONATA](https://github.com/AllenInstitute/sonata) in the case of NeuroML). Initial investigations have also taken place, in collaboration with projects in the NSF Accelerator Track C (Quantum Technology), to use MDF for facilitating the implementation of computational models on [quantum hardware](https://github.com/ModECI/MDF/blob/readme_update/examples/Quantum).

### The core elements of the MDF standard

**[Models](https://mdf.readthedocs.io/en/latest/api/Specification.html#model)** The highest level construct in MDF is a model that consists of one or more **graphs** and model attributes. The former describe the operational features of the model (its structure and execution), while the latter provide additional information (metadata) useful for executing, evaluating, testing or visualizing it.

**[Graphs](https://mdf.readthedocs.io/en/latest/api/Specification.html#graph)** A graph specifies the structure and process flow of a **model**. The most fundamental element of a graph is a **node**, which specifies some unit of computation in terms of its **parameters** and **functions**. Nodes are connected to other nodes via directed **edges**, which, in the absence of additional **conditions**, define the computational flow of the model.

**[Nodes](https://mdf.readthedocs.io/en/latest/api/Specification.html#node)** These define the core elements of computation in a **graph**, that receive and transmit information via their **input and output ports**. In general, ports represent points of contact between a **node** and the **edges**that connect it to other nodes.

**[Output Ports](https://mdf.readthedocs.io/en/latest/api/Specification.html#outputport)** An output port is the starting point of the data transmission process. After processing the information in a **node**, an output port is used to begin the transmission of information to the next **node** through **edges**.

**[Edges](https://mdf.readthedocs.io/en/latest/api/Specification.html#edge)** These transmit information from the **output port** of one **node** to the **input port** of another, collectively defining a **graph’s** topography. Edges may contain weights that can operate on the information they carry.

**[Input Ports](https://mdf.readthedocs.io/en/latest/api/Specification.html#inputport)** An input port is the endpoint of the data transmission process. It receives the information transmitted through an **edge** and inputs it to the next **node** for further processing.

**[Conditions](https://mdf.readthedocs.io/en/latest/api/Specification.html#condition)** These are a core and distinctive element of the MDF specification, that complement other computational graph-based formats by providing a high-level set of descriptors for specifying conditional execution of **nodes**. This allows models with relatively complex execution requirements (e.g., containing cycles, branches, and/or temporal dependencies) to be expressed as **graphs** in a sufficiently abstract form that facilities exchange among high-level modeling environments without requiring that they be “lowered” to and then recovered from more elaborated procedural descriptions.

**[Parameters](https://mdf.readthedocs.io/en/latest/api/Specification.html#parameter)** Attributes that determine the configuration and operation of **nodes** and **edges**, can be defined in the MDF using parameters. In the case of parameters specifying large data structures (e.g., weight-matrices), arrays in widely used formats (e.g. numpy arrays, TensorFlow tensors) can be used, and serialisation in portable binary formats (e.g. BSON) is supported. Parameters can either be fixed values, which don't change when the **node** is executed, or can change over time (stateful parameters).

**[Functions](https://mdf.readthedocs.io/en/latest/api/Specification.html#function)** A single value which is evaluated as a function of values on **input ports** and other functions and **parameters**. A key distinction with **parameters** is that a function is always stateless.

**[Model metadata](https://mdf.readthedocs.io/en/latest/api/Specification.html#model)** There is the ability to add “metadata” to the **model**, **graph**, **nodes** and many of their sub elements which provide additional information about that element. While the metadata should not be essential to the mathematical description of the behavior/structure of the element, it could be useful for human interpretability of its function/purpose, or used when it is mapped to a specific application for simulation/visualization. Metadata can be added to the top level model to specify contact information, citations, acknowledgements, pointers to sample data and benchmark results, and environments in which the specified model was originally implemented and any that have been validated to support its execution.

Fig 2: A simple graph with 3 nodes and 2 edges expressed in MDF.

Fig 3: This graph illustrates the ability to specify behavior that extends beyond the directed flow through the graph. Here, Node 1 generates a random number and transmits that number to Node 2. Node 2 will only run if the number it receives from Node 1 is greater than 10.

## Installation

### Requirements
Requires Python >= 3.7

### Quick start
```
pip install modeci-mdf
```
For more detailed installation instructions see [here](https://mdf.readthedocs.io/en/latest/api/Installation.html).

For guidelines on contributing to the development of MDF, see [here](https://github.com/ModECI/MDF/blob/main/CONTRIBUTING.md).

## Examples

To get started, follow the simple example in a Jupyter notebook [here](https://github.com/ModECI/MDF/blob/main/examples/SimpleExample.ipynb)

Multiple examples of serialized MDF files, the Python scripts used to generate them, as well as mappings to target environments can be found [here](https://github.com/ModECI/MDF/blob/main/examples/README.md).

[actions-badge]:            https://github.com/ModECI/MDF/actions/workflows/ci.yml/badge.svg
[actions-link]:             https://github.com/ModECI/MDF/actions
[rtd-badge]:                https://readthedocs.org/projects/mdf/badge/?version=latest
[rtd-link]:                 https://mdf.readthedocs.io/en/latest/?badge=latest


![CI Status](https://github.com/JaGeo/LobsterPy/actions/workflows/python-package.yml/badge.svg) [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/JaGeo/LobsterPy/main.svg)](https://results.pre-commit.ci/latest/github/JaGeo/LobsterPy/main) [![codecov](https://codecov.io/gh/JaGeo/LobsterPy/graph/badge.svg?token=MC5BRXVEGW)](https://codecov.io/gh/JaGeo/LobsterPy) [![build-docs](https://github.com/JaGeo/LobsterPy/actions/workflows/docs.yml/badge.svg)](https://jageo.github.io/LobsterPy/) [![PyPI version](https://badge.fury.io/py/lobsterpy.svg)](https://badge.fury.io/py/lobsterpy) [![PyPI downloads](https://img.shields.io/pypi/dm/lobsterpy?style=flat&color=blue&label=pypi%20downloads)](https://pypi.org/project/lobsterpy) [![Downloads](https://pepy.tech/badge/lobsterpy)](https://pepy.tech/project/lobsterpy) ![supported python versions](https://img.shields.io/pypi/pyversions/lobsterpy) [![DOI](https://zenodo.org/badge/343384088.svg)](https://zenodo.org/badge/latestdoi/343384088) [![status](https://joss.theoj.org/papers/4e8524125e36486c65a4b435bbfe2df2/status.svg)](https://joss.theoj.org/papers/4e8524125e36486c65a4b435bbfe2df2)
# Getting started

LobsterPy is a package that enables automatic analysis of LOBSTER outputs to get summarized bonding information and relevant bond plots. Additionally, one can also generate features for machine learning studies from LOBSTER outputs. One can download LOBSTER from [http://www.cohp.de](http://www.cohp.de).

        Important

Recently released [LOBSTER 5.0](https://schmeling.ac.rwth-aachen.de/cohp/index.php?menuID=6) now generates `POSCAR.lobster` for any kind of LOBSTER calculation by default (This file has same format as the POSCAR from VASP). Thus, LobsterPy in principle, now supports usage with **all** DFT codes supported by LOBSTER and is **no** longer limited to `VASP`. Almost all of the core functionalities of LobsterPy could be used. The user must use `POSCAR.lobster` for `path_to_poscar` and `-fstruct` argument in python and cli interface, respectively.

The only functionality limited to VASP is DOS comparisons and basis set analysis in the `calc_quality_summary` method of the `Analysis` class, as it relies on VASP output files, namely `vasprun.xml` and `POTCAR`.

Please note that LobsterPy relies on the LOBSTER computation output files. Thus, it will be only able to analyze data that has been computed in the LOBSTER run.

![LobsterPyAnimation](https://github.com/JaGeo/LobsterPy/assets/22094846/8f06b84c-db6d-414c-8590-aa04c957c728)

## Installation

### Python version
Before the installation, please make sure that you are using one of the supported Python versions (see [pyproject.toml](https://github.com/JaGeo/LobsterPy/blob/main/pyproject.toml)).

### Standard installation
Install using ``pip install lobsterpy``

### Installation with featurizer
Install using ``pip install lobsterpy[featurizer]``

## Basic usage

* **Automatic analysis and plotting of COHPs / COBIS / COOPs:**

You can use ``lobsterpy description`` for an automated analysis of COHPs for relevant cation-anion bonds or ``lobsterpy automatic-plot`` to plot the results automatically.
It will evaluate all COHPs with ICOHP values down to 10% of the strongest ICOHP.
You can enforce an analysis of all bonds by using ``lobsterpy automatic-plot --allbonds``.
You can also switch the automatic analysis to use the ICOBIs or ICOOPs. You need to add `--cobis` or `--coops` along with the mentioned commands
for e.g.like  ``lobsterpy description --cobis``

An interactive plotter is available via ``lobsterpy automatic-plot-ia``.

Currently, the computed Mulliken charges will be used to determine cations and anions. If no ``CHARGE.lobster`` is available, the algorithm will fall back to the BondValence analysis from pymatgen.

*Please be aware that LobsterPy can only analyze bonds that have been included in the initial Lobster computation. Thus, please use the cohpgenerator within Lobster (i.e., put `cohpGenerator from 0.1 to 5.0` in the *lobsterin*).*

It is also possible to start this automatic analysis from a Python script. See "examples" for scripts.

* **Plotting DOS from LOBSTER computations:**

  To plot densities of states obtained from LOBSTER use ``lobsterpy plot-dos``.

* **Generic COHP/ COOP / COBI plotter**:

  We included options to plot COHPs/COBIs/COOPs from the command line.
``lobsterpy plot 1 2`` will plot COHPs of the first and second bond from ``COHPCAR.lobster``. It is possible to sum or integrate the COHPs as well (``--summed``, ``--integrated``). You can switch to COBIs or COOPs by using ``--cobis`` or ``--coops``, respectively.

* **Other command line tools**:

    ``lobsterpy create-inputs`` will create standard inputs based on existing POSCAR, POTCAR, and INCAR files. It will allow testing for different basis sets that are available in Lobster. This feature is currently only available for PBE_54 POTCARs, as only the pbeVASPfit2015 basis in LOBSTER that has been fitted to PBE POTCARs includes additional orbitals relevant to solid-state materials. Please
                        check out our publication [https://doi.org/10.1002/cplu.202200123](https://doi.org/10.1002/cplu.202200123) and LOBSTER program manual for more information

* **Further help?**

    You can get further information by using ``lobsterpy --help`` and also by typing ``lobsterpy description --help``,
``lobsterpy automatic-plot --help``, ``lobsterpy plot --help``.

## Comprehensive documentation
* Checkout the [documentation and tutorials](https://jageo.github.io/LobsterPy/) for more details.

## Contributing
A short guide to contributing to LobsterPy can be found [here](https://jageo.github.io/LobsterPy/dev/contributing.html).
Additional information for developers can be found [here](https://jageo.github.io/LobsterPy/dev/dev_installation.html).

## How to cite?
Please cite our paper: A. A. Naik, K. Ueltzen, C. Ertural, A. J. Jackson, J. George, *Journal of Open Source Software* **2024**, *9*, 6286. [https://joss.theoj.org/papers/10.21105/joss.06286](https://joss.theoj.org/papers/10.21105/joss.06286).
Please cite [pymatgen](https://github.com/materialsproject/pymatgen), [Lobster](https://schmeling.ac.rwth-aachen.de/cohp/index.php?menuID=1), and [ChemEnv](https://doi.org/10.1107/S2052520620007994) correctly as well.

You can find more information on the methodology of the automatic analysis in J. George, G. Petretto, A. Naik, M. Esters, A. J. Jackson, R. Nelson, R. Dronskowski, G.-M. Rignanese, G. Hautier, *ChemPlusChem* **2022**, *87*, e202200123. [https://doi.org/10.1002/cplu.202200123](https://doi.org/10.1002/cplu.202200123).

## LobsterPy is now a part of an atomate2 workflow
![LobsterWorkflow](https://github.com/JaGeo/LobsterPy/assets/22094846/337615ac-542e-446c-bc63-fb5946b16544)

We have now also included the automatic analysis into a fully automatic workflow using VASP and Lobster in [atomate2](https://github.com/materialsproject/atomate2). More documentation and information will follow soon.

## Acknowledgements
The development of the program has been supported by a computing time grant. We gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS Supercomputer SuperMUC-NG at Leibniz Supercomputing Centre (www.lrz.de) (project pn73da).


# Scirpy: single-cell immune receptor analysis in Python

[![Tests][badge-tests]][link-tests]
[![Documentation][badge-docs]][link-docs]
[![PyPI][badge-pypi]][link-pypi]
[![bioconda][badge-bioconda]][link-bioconda]
[![airr][badge-airr]][link-airr]
[![Powered by NumFOCUS][badge-numfocus]][link-numfocus]

Scirpy is a package to analyse T cell receptor (TCR) or B cell receptor (BCR)
repertoires from single-cell RNA sequencing (scRNA-seq) data in Python.
It seamlessly integrates with [scanpy][] and [mudata][] and provides various modules for data import, analysis and visualization.

[//]: # "numfocus-fiscal-sponsor-attribution"

Scirpy is part of the scverse project ([website](https://scverse.org), [governance](https://scverse.org/about/roles)) and is fiscally sponsored by [NumFOCUS](https://numfocus.org/).
Please consider making a tax-deductible [donation](https://numfocus.org/donate-to-scverse) to help the project pay for developer time, professional services, travel, workshops, and a variety of other needs.

<a href="https://numfocus.org/project/scverse">
  <img
    src="https://raw.githubusercontent.com/numfocus/templates/master/images/numfocus-logo.png"
    width="200"
  >
</a>

[badge-tests]: https://img.shields.io/github/actions/workflow/status/scverse/scirpy/test.yaml?branch=main
[link-tests]: https://github.com/scverse/scirpy/actions/workflows/test.yml
[badge-docs]: https://img.shields.io/readthedocs/scirpy
[badge-pypi]: https://img.shields.io/pypi/v/scirpy?logo=PyPI
[link-pypi]: https://pypi.org/project/scirpy/
[link-bioconda]: http://bioconda.github.io/recipes/scirpy/README.html
[badge-bioconda]: https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat
[badge-airr]: https://img.shields.io/static/v1?label=AIRR-C%20sw-tools%20v1&message=compliant&color=008AFF&labelColor=000000&style=flat
[link-airr]: https://docs.airr-community.org/en/stable/swtools/airr_swtools_standard.html
[scverse]: https://scverse.org
[scanpy]: https://scanpy.readthedocs.io/
[mudata]: https://github.com/scverse/mudata
[badge-numfocus]: https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A
[link-numfocus]: http://numfocus.org

## Getting started

Please refer to the [documentation][link-docs]. In particular, the

-   [Tutorial][link-tutorial], and the
-   [API documentation][link-api].

## Installation

You need to have Python 3.9 or newer installed on your system. If you don't have
Python installed, we recommend installing [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge).

There are several alternative options to install scirpy:

1. Install the latest release of `scirpy` from [PyPI](https://pypi.org/project/scirpy/):

    ```bash
    pip install scirpy
    ```

2. Get it from [Bioconda][link-bioconda]:

    First **setup conda channels [as described here](https://bioconda.github.io/#usage)**. Then install scirpy:

    ```bash
    conda install scirpy
    ```

3. Install the latest development version:

    ```bash
    pip install git+https://github.com/scverse/scirpy.git@main
    ```

4. Run it in a container using [Docker][] or [Podman][]:

    ```bash
    docker pull quay.io/biocontainers/scirpy:<tag>
    ```

where `tag` is one of [these tags](https://quay.io/repository/biocontainers/scirpy?tab=tags).

## Release notes

See the [changelog][changelog].

## Support and Contact

We are happy to assist with problems when using scirpy.

-   If you need help with scirpy or have questions regarding single-cell immune-cell receptor analysis in general, please join us in the [scverse discourse][scverse-discourse].
-   For bug report or feature requests, please use the [issue tracker][issue-tracker].

We try to respond within two working days, however fixing bugs or implementing new features
can take substantially longer, depending on the availability of our developers.

## Citation

If you use `scirpy` in your work, please cite the `scirpy`
publication as follows:

> **Scirpy: A Scanpy extension for analyzing single-cell T-cell
> receptor sequencing data**
>
> Gregor Sturm, Tamas Szabo, Georgios Fotakis, Marlene Haider, Dietmar
> Rieder, Zlatko Trajanoski, Francesca Finotello
>
> _Bioinformatics_ 2020 Sep 15. doi:
> [10.1093/bioinformatics/btaa611](https://doi.org/10.1093/bioinformatics/btaa611).

You can cite the scverse publication as follows:

> **The scverse project provides a computational ecosystem for
> single-cell omics data analysis**
>
> Isaac Virshup, Danila Bredikhin, Lukas Heumos, Giovanni Palla, Gregor
> Sturm, Adam Gayoso, Ilia Kats, Mikaela Koutrouli, Scverse Community,
> Bonnie Berger, Dana Pe’er, Aviv Regev, Sarah A. Teichmann, Francesca
> Finotello, F. Alexander Wolf, Nir Yosef, Oliver Stegle & Fabian J.
> Theis
>
> _Nat Biotechnol._ 2023 Apr 10. doi:
> [10.1038/s41587-023-01733-8](https://doi.org/10.1038/s41587-023-01733-8).

[scverse-discourse]: https://discourse.scverse.org/
[issue-tracker]: https://github.com/scverse/scirpy/issues
[changelog]: https://scirpy.readthedocs.io/latest/changelog.html
[link-docs]: https://scirpy.readthedocs.io
[link-api]: https://scirpy.readthedocs.io/latest/api.html
[link-tutorial]: https://scirpy.scverse.org/en/latest/tutorials.html
[Docker]: https://www.docker.com/
[Podman]: https://podman.io/