# :notebook_with_decorative_cover: Table of Contents

- [About the Project](#star2-about-the-project)
  * [Functionality](#mechanical_arm-functionality)
  * [Model collection](#file_cabinet-model-collection)
- [Getting Started](#running-getting-started)
  * [Setting up your environment](#computer-setting-up-your-environment)
  * [Basic usage](#mag-basic-usage)
- [Contributing](#wave-how-to-contribute)
- [License](#warning-license)
- [Citation](#page_with_curl-citation)
- [Contributions](#gem-contributions)

## :star2: About the Project
`thingsvision` is a Python package for extracting (image) representations from many state-of-the-art computer vision models. Essentially, you provide `thingsvision` with a directory of images and specify the neural network you're interested in. Subsequently, `thingsvision` returns the representation of the selected neural network for each image, resulting in one feature map (vector or matrix, depending on the layer) per image. These features, used interchangeably with _image representations_, can then be used for further analyses.

:rotating_light: NOTE: some function calls mentioned in the original [paper](https://www.frontiersin.org/articles/10.3389/fninf.2021.679838/full) have been deprecated. To use this package successfully, exclusively follow this `README` and the [documentation](https://vicco-group.github.io/thingsvision/)! :rotating_light:

(back to top)

### :mechanical_arm: Functionality
With `thingsvision`, you can:
- extract features for any imageset from many popular networks.
- extract features for any imageset from your custom networks.
- extract features for >26,000 images from the [THINGS image database](https://osf.io/jum2f/).
- [align](https://vicco-group.github.io/thingsvision/Alignment.html) the extracted features with human object perception (e.g., using [gLocal](https://proceedings.neurips.cc/paper_files/paper/2023/hash/9febda1c8344cc5f2d51713964864e93-Abstract-Conference.html)).
- extract features from [HDF5 datasets](https://vicco-group.github.io/thingsvision/LoadingYourData.html#using-the-hdf5dataset-class) directly (e.g., [NSD stimuli](https://naturalscenesdataset.org/))
- conduct basic [Representational Similarity Analysis (RSA)](https://vicco-group.github.io/thingsvision/RSA.html#representational-similarity-analysis-rsa) after feature extraction.
- perform efficient [Centered Kernel Alignment (CKA)](https://vicco-group.github.io/thingsvision/RSA.html#centered-kernel-alignment-cka) to compare image features across model-module combinations.

(back to top)

### :file_cabinet: Model collection
Neural networks come from different sources. With `thingsvision`, you can extract image representations of all models from:
- [torchvision](https://pytorch.org/vision/0.8/models.html)
- [Keras](https://www.tensorflow.org/api_docs/python/tf/keras/applications)
- [timm](https://github.com/rwightman/pytorch-image-models)
- `ssl` (self-supervised learning models)
  - `simclr-rn50`, `mocov2-rn50`, `barlowtwins-rn50`, `pirl-rn50`
  - `jigsaw-rn50`, `rotnet-rn50`, `swav-rn50`, `vicreg-rn50`
  - `dino-rn50`, `dino-xcit-{small/medium}-{12/24}-p{8/16}`
  - `dino-vit-{tiny/small/base}-p{8/16}`
  - `dinov2-vit-{small/base/large/giant}-p14`
  - `mae-vit-{base/large}-p16`, `mae-vit-huge-p14`

- [OpenCLIP](https://github.com/mlfoundations/open_clip) models (CLIP trained on LAION-{400M/2B/5B})
- [CLIP](https://github.com/openai/CLIP) models (CLIP trained on WiT)
- a few custom models (Alexnet, VGG-16, Resnet50, and Inception_v3) trained on [Ecoset](https://www.pnas.org/doi/10.1073/pnas.2011417118) rather than ImageNet and one Alexnet model pretrained on ImageNet and fine-tuned on [SalObjSub](https://cs-people.bu.edu/jmzhang/sos.html)

- each of the many [CORnet](https://github.com/dicarlolab/CORnet) versions (recurrent vision models)
- [Harmonization](https://arxiv.org/abs/2211.04533) models (see [Harmonization repo](https://github.com/serre-lab/harmonization)). The default variant is `ViT_B16`. Other available models are `ResNet50`, `VGG16`, `EfficientNetB0`, `tiny_ConvNeXT`, `tiny_MaxViT`, and `LeViT_small`

- [DreamSim](https://dreamsim-nights.github.io/) models  (see [DreamSim repo](https://github.com/ssundaram21/dreamsim)). The default variant is `open_clip_vitb32`. Other available models are `clip_vitb32`, `dino_vitb16`, and an `ensemble`. See the [docs](https://vicco-group.github.io/thingsvision/AvailableModels.html#dreamsim) for more information
- FAIR's [Segment Anything (SAM)](https://vicco-group.github.io/thingsvision/AvailableModels.html#align-model) model
- Kakaobrain's [ALIGN](https://vicco-group.github.io/thingsvision/AvailableModels.html#align-model) implementation

(back to top)

## :running: Getting Started

### :computer: Setting up your environment
#### Working locally
First, create a new `conda environment` with Python version 3.8, 3.9, or 3.10 e.g. by using `conda`:

```bash
$ conda create -n thingsvision python=3.9
$ conda activate thingsvision
```

Then, activate the environment and simply install `thingsvision` via running the following `pip` command in your terminal.

```bash
$ pip install --upgrade thingsvision
$ pip install git+https://github.com/openai/CLIP.git
```

If you want to extract features for [harmonized models](https://vicco-group.github.io/thingsvision/AvailableModels.html#harmonization) from the [Harmonization repo](https://github.com/serre-lab/harmonization), you have to additionally run the following `pip` command in your `thingsvision` environment (FYI: as of now, this seems to be working smoothly on Ubuntu only but not on macOS),

```bash
$ pip install git+https://github.com/serre-lab/Harmonization.git
$ pip install keras-cv-attention-models>=1.3.5
```

If you want to extract features for [DreamSim](https://dreamsim-nights.github.io/) from the [DreamSim repo](https://github.com/ssundaram21/dreamsim), you have to additionally run the following `pip` command in your `thingsvision` environment,

```bash
$ pip install dreamsim==0.1.2
```

See the [docs](https://vicco-group.github.io/thingsvision/AvailableModels.html#dreamsim) for which `DreamSim` models are available in `thingsvision`.

#### Google Colab
Alternatively, you can use Google Colab to play around with `thingsvision` by uploading your image data to Google Drive (via directory mounting).
You can find the jupyter notebook using `PyTorch` [here](https://colab.research.google.com/github/ViCCo-Group/thingsvision/blob/master/notebooks/pytorch.ipynb) and the `TensorFlow` example [here](https://colab.research.google.com/github/ViCCo-Group/thingsvision/blob/master/notebooks/tensorflow.ipynb).

(back to top)

### :mag: Basic usage

#### Command Line Interface (CLI)

`thingsvision` was designed to simplify feature extraction. If you have some folder of images (e.g., `./images`) and want to extract features for each of these images without opening a Jupyter Notebook instance or writing a Python script, it's probably easiest to use our CLI. The interface includes two options,

- `thingsvision show-model`
- `thingsvision extract-features`

Example calls might look as follows:

```bash
thingsvision show-model --model-name "alexnet" --source "torchvision"
thingsvision extract-features --image-root "./data" --model-name "alexnet" --module-name "features.10" --batch-size 32 --device "cuda" --source "torchvision" --file-format "npy" --out-path "./features"
```

See `thingsvision show-model -h` and `thingsvision extract-features -h` for a list of all possible arguments. Note that the CLI provides just the basic extraction functionalities but is probably enough for most users that don't want to dive too deep into various models and modules. If you need more fine-grained control over the extraction itself, we recommend to use the python package directly and write your own Python script.

#### Python commands

To do this start by importing all the necessary components and instantiating a `thingsvision` extractor. Here we're using `CLIP` from the official clip repo as the model to extract features from and also load the model to GPU for faster inference,

```python
import torch
from thingsvision import get_extractor
from thingsvision.utils.storing import save_features
from thingsvision.utils.data import ImageDataset, DataLoader

model_name = 'clip'
source = 'custom'
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model_parameters = {
    'variant': 'ViT-L/14'
}

extractor = get_extractor(
  model_name=model_name,
  source=source,
  device=device,
  pretrained=True,
  model_parameters=model_parameters,
)
```

As a next step, create both dataset and dataloader for your images. We assume that all of your images are in a single `root` directory which can contain subfolders (e.g., for individual classes). Therefore, we leverage the `ImageDataset` class.

```python
root='path/to/your/image/directory' # (e.g., './images/)
batch_size = 32

dataset = ImageDataset(
    root=root,
    out_path='path/to/features',
    backend=extractor.get_backend(), # backend framework of model
    transforms=extractor.get_transformations(resize_dim=256, crop_dim=224) # set the input dimensionality to whichever values are required for your pretrained model
)

batches = DataLoader(
    dataset=dataset,
    batch_size=batch_size,
    backend=extractor.get_backend() # backend framework of model
)
```

Now all that is left is to extract the image features and store them on disk! Here we're extracting features from the image encoder module of CLIP (`visual`), but if you don't know which modules are available for a given model, just call `extractor.show_model()` to print all the modules.

```python
module_name = 'visual'

features = extractor.extract_features(
    batches=batches,
    module_name=module_name,
    flatten_acts=True,
    output_type="ndarray", # or "tensor" (only applicable to PyTorch models of which CLIP and DINO are ones!)
)

save_features(features, out_path='path/to/features', file_format='npy') # file_format can be set to "npy", "txt", "mat", "pt", or "hdf5"
```

#### Feature extraction with custom data pipeline

##### PyTorch

```python
module_name = 'visual'

# your custom dataset and dataloader classes come here (for example, a PyTorch data loader)
my_dataset = ...
my_dataloader = ...

with extractor.batch_extraction(module_name, output_type="tensor") as e:
  for batch in my_dataloader:
    ... # whatever preprocessing you want to add to the batch
    feature_batch = e.extract_batch(
      batch=batch,
      flatten_acts=True, # flatten 2D feature maps from an early convolutional or attention layer
      )
    ... # whatever post-processing you want to add to the extracted features
```

##### TensorFlow / Keras

```python
module_name = 'visual'

# your custom dataset and dataloader classes come here (for example, TFRecords files)
my_dataset = ...
my_dataloader = ...

for batch in my_dataloader:
  ... # whatever preprocessing you want to add to the batch
  feature_batch = extractor.extract_batch(
    batch=batch,
    module_name=module_name,
    flatten_acts=True, # flatten 2D feature maps from an early convolutional or attention layer
    )
  ... # whatever post-processing you want to add to the extracted features
```

#### Human alignment

*Human alignment*: If you want to align the extracted features with human object similarity according to the approach introduced in *[Improving neural network representations using human similiarty judgments](https://proceedings.neurips.cc/paper_files/paper/2023/hash/9febda1c8344cc5f2d51713964864e93-Abstract-Conference.html)* you can optionally `align` the extracted features using the following method:

```python
aligned_features = extractor.align(
    features=features,
    module_name=module_name,
    alignment_type="gLocal",
)
```

For more information about the available alignment types and aligned models see the [docs](https://vicco-group.github.io/thingsvision/Alignment.html).

_For more examples on the many models available in `thingsvision` and explanations of additional functionality like how to optionally turn off center cropping, how to use HDF5 datasets (e.g. NSD stimuli), how to perform RSA or CKA, or how to easily extract features for the [THINGS image database](https://osf.io/jum2f/), please refer to the [Documentation](https://vicco-group.github.io/thingsvision/)._

(back to top)

## :wave: How to contribute
If you come across problems or have suggestions please submit an issue!

(back to top)

## :warning: License
This GitHub repository is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.

(back to top)

## :page_with_curl: Citation
If you use this GitHub repository (or any modules associated with it), please cite our [paper](https://www.frontiersin.org/articles/10.3389/fninf.2021.679838/full) for the initial version of `thingsvision` as follows:

```latex
@article{Muttenthaler_2021,
	author = {Muttenthaler, Lukas and Hebart, Martin N.},
	title = {THINGSvision: A Python Toolbox for Streamlining the Extraction of Activations From Deep Neural Networks},
	journal ={Frontiers in Neuroinformatics},
	volume = {15},
	pages = {45},
	year = {2021},
	url = {https://www.frontiersin.org/article/10.3389/fninf.2021.679838},
	doi = {10.3389/fninf.2021.679838},
	issn = {1662-5196},
}
```

(back to top)

## :gem: Contributions

This is a joint open-source project between the [Max Planck Institute for Human Cognitive and Brain Sciences](https://www.cbs.mpg.de/en), Leipzig, and the [Machine Learning Group](https://web.ml.tu-berlin.de/) at Technische Universtität Berlin. Correspondence and requests for contributing should be adressed to [Lukas Muttenthaler](https://lukasmut.github.io/). Feel free to contact us if you want to become a contributor or have any suggestions/feedback. For the latter, you could also just post an issue or engange in discussions. We'll try to respond as fast as we can.

(back to top)


# Aidants Connect

[![CircleCI](https://circleci.com/gh/betagouv/Aidants_Connect/tree/main.svg?style=svg)](https://circleci.com/gh/betagouv/Aidants_Connect/tree/main)

Aidants Connect est une application web qui propose à des aidants les fonctionnalités suivantes :

- créer un mandat de connexion via [FranceConnect](https://franceconnect.gouv.fr/) avec un ou plusieurs usagers sur un périmètre et une durée définis ;
- connecter via FranceConnect un usager dans les conditions des mandats créés ;
- accéder à des ressources sur l'accompagnement des usagers ;
- accéder à un suivi de ses mandats.

Aidants Connect est construit sur les éléments suivants :

- Python 3.11
- Django 4.2
- PostgreSQL

## Installer et lancer l'application

Lancement rapide si vous avez déjà installé la base de données et les dépendances de test :

```shell
git clone git@github.com:betagouv/Aidants_Connect.git
cd Aidants_Connect
cp .env.example .env
virtualenv venv
source venv/bin/activate
pip install --user pipenv
pipenv install --dev
python manage.py collectstatic
pre-commit install
```

Ensuite, vous devriez pouvoir faire tourner les tests :

```shell
python manage.py test
```

Et lancer le serveur :

```shell
python manage.py runserver 3000
```

### Installer la base de données

Utilisez votre gestionnaire de paquets préféré pour installer la base.
L'exemple qui suit emploie le gestionnaire [Homebrew](https://brew.sh) via la commande `brew`.

Dans un terminal, installez [PostgreSQL](https://www.postgresql.org) :

```sh
brew install postgresql
```

Démarrez le service postgresql :

```sh
brew services start postgresql
```

Ceci démarre le serveur de la base de données et active sa réexécution au login.

Dans le cas où ce serait votre première utilisation de PostgreSQL, créez une base d'essai à votre nom :

```sh
createdb `whoami`
```

Puis, démarrez l'invite de commande PostgreSQL :

```sh
psql
```

Vous pouvez dès à présent visualiser :

- la liste des bases de données existantes avec cette commande PostgreSQL `\list`
- la liste des rôles existants avec `\du`

Ajoutez une base `aidants_connect` appartenant au nouvel utilisateur `aidants_connect_team` en poursuivant dans l'invite de commmande PostgreSQL :

```sql
CREATE USER aidants_connect_team;
CREATE DATABASE aidants_connect OWNER aidants_connect_team;
ALTER USER aidants_connect_team CREATEDB;
```

:tada: La base de donnée `aidants_connect` est installée. Vous pouvez la voir et quitter l'invite de commande avec :

```sql
\list
\q
```

### Installer l'application

Dans votre répertoire de travail, créez et activez un environnement virtuel :

```shell
virtualenv venv
source venv/bin/activate
```
Installez pipenv :

```shell
brew install pipenv  # Sur Mac
# Ou
pip install pipenv
```

Installez les dépendances :

```shell
pipenv install --dev
```

Si vous avez un Mac M1, ou si la commande précédente déclenche le message d'erreur `ld: library not found for -lssl`, référez-vous à la section [Troubleshooting](#troubleshooting).

### Configurer les variables d'environnement

Dupliquez le fichier `.env.example` à la racine du projet en tant que `.env` :

```shell
cp .env.example .env
```

En test en local, vous ne devriez pas avoir à modifier ce `.env`.

Créez un répertoire `staticfiles` à la racine du projet :

```shell
mkdir staticfiles
```

Appliquez les migrations de la base de données :

```shell
python manage.py migrate
```

### Peupler la base de données

Il existe plusieurs moyens de peupler la base de données.

#### Utiliser les _fixtures_

Des données de test qui créent un environnement complet :

```shell
  python manage.py loaddata admin.json
  python manage.py loaddata usager_autorisation.json
  python manage.py loaddata faq.json
```

Ce process crée automatiquement un _superuser_ `admin@email.com`. Plus d'information sur comment se connecter avec ce compte sont disponible dans la section [Se connecter à l'application](#se-connecter-à-lapplication)

#### Créer manuellement un _superuser_

Créez un profil administrateur avec une organisation par défaut :

```shell
python manage.py createsuperuser --username <insert_admin_email> --organisation-name <insert_organisation_name>
```

Une organisation avec l'email que vous avez spécifié sera automatiquement créée pour ce profil.
Si vous avez déjà créé une organisation vous pouvez passer son numéro dans la base de donnée à la création du profil
admin :

```shell
python manage.py createsuperuser --username <insert_admin_email> --organisation <insert_organisation_pk>
```

Pour pouvoir vous connecter à votre instance locale, il faut apparier à votre `superuser` un dispositif TOTP (`TOTP device`).

Pour cela, commencez par lui adjoindre un [jeton OTP](https://fr.wikipedia.org/wiki/Mot_de_passe_%C3%A0_usage_unique) [statique](https://django-otp-official.readthedocs.io/en/stable/overview.html#module-django_otp.plugins.otp_static) :

```shell
python manage.py addstatictoken <insert_admin_email> -t <insert_6_numbers>
```

Notez ce code, il vous permettra de vous connecter la première fois à l'interface d'administration.

### Lancer les tests

Si vous ne les avez pas, installez les éléments suivants :

- Navigateur Firefox en [téléchargement](https://www.mozilla.org/fr/firefox/download/thanks/)
- [Gecko driver](https://github.com/mozilla/geckodriver/releases)

Installation de Gecko Driver :

- Mac OS : `brew install geckodriver `
- Linux : vous pouvez télécharger [ici](https://github.com/mozilla/geckodriver/releases) la dernière
  version du driver et déposer le fichier `geckodriver` dans `VOTRE_VENV/bin` (ou dans `/usr/local/bin` si vous voulez
  donner un accès global au driver).

Puis lancez les commandes suivantes pour vérifier le style du code source et exécuter les tests de l'application :

```shell
flake8
black --check .
python manage.py test
```

Les tests fonctionnels (Selenium) sont lancés sur `http://localhost:3000`.
Il faut s'assurer que rien d'autre n'occupe ce port pendant leur exécution.

Par défaut, les tests Selenium sont lancés en mode _headless_, c'est-à-dire sans ouverture de fenêtre de navigateur. Pour modifier ce comportement, inversez la valeur de la variable d'environnement `HEADLESS_FUNCTIONAL_TESTS` dans votre fichier `.env`.

Astuce : la plupart des cas de tests portent une directive `@tag` pour leur associer des tags décrivant des fonctionnalités ou des caractéristiques des tests. Par exemple, `functional` pour les tests fonctionnels, `create_mandat` pour ce qui implique une création de mandat, etc.
Cela vous permet de lancer seulement certains tests, grâce à l'option `--tag`. Par exemple, pour lancer les tests portant le tag `parrot` :

```shell
python manage.py test --tag parrot
```

### Lancer l'application

Pour lancer l'application sur le port `3000` :

```shell
python manage.py runserver 3000
```

L'application sera disponible à l'URL `http://localhost:3000/`.

### Se connecter à l'application

Votre _superuser_ est créé et a un login, un mot de passe et un _static token_ c'est-à-dire un code à 6 chiffres utilisable une seule fois. Il faut maintenant obtenir le QR code qui vous permettra de vous connecter de manière pérenne.

#### Trouver la page d'admin

La page d'admin se trouve sur `/[Variable d'environnement ADMIN_URL]`. En local, `/adm`.

Attention, `/admin` n'est pas un vrai site d'admin… c'est un [pot de miel](https://pypi.org/project/django-admin-honeypot/).

#### Se connecter à l'admin

- ⏩ Si vous avez utilisé les _fixtures_ pour peupler votre base de données,
  - identifiant : `admin@email.com`;
  - mot de passe : `admin`;
  - Static OTP : `111111`
- Sinon, utilisez le login, mot de passe et static token créés dans la section [Installation sur un serveur : Créer un _superuser_](#installation-sur-un-serveur--créer-un-superuser)

#### Pérenniser son authentification à double facteur (2FA)

Une fois connecté à l'admin, cliquez sur **_TOTP devices_**

- ⏩ Si vous avez utilisé les _fixtures_ :
  - Cliquez sur le lien `qr code` à droite de l'entrée pour Admin
  - Scannez le QRcode dans une application TOTP telle que [Authy](https://authy.com/) ou [Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2)

Si vous avez créé votre propre _superuser_ :

1. Cliquez sur le bouton `Ajouter TOTP device +`
2. Choisissez votre _superuser_ grâce à l'icône "loupe" située à côté du champ _User_
3. Saisissez un nom pour votre dispositif TOTP (par exemple : _Mon téléphone_) dans le champ `Name`
4. Cliquez ensuite sur _Enregistrer et continuer les modifications_ tout en bas du formulaire
5. Une fois l'enregistrement effectué, l'écran devrait se rafraîchir et vous proposer un lien vers un [QR Code](https://fr.wikipedia.org/wiki/Code_QR)
6. Vous pouvez à présent scanner celui-ci dans une application TOTP telle que [Authy](https://authy.com/) ou [Google Authenticator](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2) pour utiliser l'authentification à double facteur dans votre environnement local.

## Contribuer à l'application

Il faut d'abord avoir correctement installé l'application.

Installez les _git hooks_ :

```
pre-commit install
```

## Travailler sur le côté client (CSS et JavaScript)

### CSP et JavaScript inline

La CSP (content security policy) de Aidants Connect fonctionne en liste blanche : elle nécessite de lister tous les scripts inline (dans des balises `<script>`).

Si vous devez modifier ou ajouter un bout de JavaScript inline, mettez à jour la constante `CSP_SCRIPT_SRC` dans le fichier `settings.py`.

Pour calculer le hash à mettre, le plus simple est d'ouvrir la page sur Chrome et de chercher dans la console le message commençant par "Refused to execute inline script".
Il mentionne le hash (commençant par `'sha256-`) à ajouter à la CSP.

### CSS et SCSS

Pour compiler les fichiers SCSS en CSS, vous devez avoir installé sass sur votre poste, la commande `sass` doit être disponible dans votre `$PATH`.

Ensuite, utilisez une des deux commandes suivantes :

```
python manage.py scss # compilation one-shot
# ou bien :
python manage.py scss --watch # compilation automatique à chaque modification de SCSS
```

## Installation sur un serveur

### Variables d'environnement

Sur un serveur, vous voudrez ajouter vos informations dans le fichier .env ou dans les variables d'environnement de votre hébergeur :

- Les champs obligatoires sont indiqués par le préfixe `<insert_`.
- Les informations de production `FC_AS_FS` et `FC_AS_FI` sont à récupérer via des [habilitations FranceConnect](https://franceconnect.gouv.fr/partenaires).
- Vous allez devoir calculer la valeur `HASH_FC_AS_FI_SECRET` à partir de la valeur de `FC_AS_FI_SECRET` : pour cela voir dans les annexes [la procédure](#calcul-de-hash_fc_as_fi_secret-à-partir-de-la-valeur-de-fc_as_fi_secret).
- Les valeurs de sécurité sont issues de [la section "sécurité" de la documentation Django](https://docs.djangoproject.com/fr/3.2/topics/security/) et de [la conférence Django and Web Security Headers](https://www.youtube.com/watch?v=gvQW1vVNohg).

### Création d'un superuser sans accès à la console Django

Il existe deux solutions.

### Solution avant installation du code

- déployer la branche de la [PR 473](https://github.com/betagouv/Aidants_Connect/pull/473) avant de déployer main (utilisé pour notre environnement sandbox).
  - avant de déployer le code, insérer les variables d'environnement `INIT_*` mentionnées dans la PR ;
  - installer l'application à partir de la branche de la PR 473 ;
  - déployer ensuite l'application à partir de la branche `main`.

### Solution après installation du code

- insérer directement les données dans la base de données après l'installation de l'application.

## Annexes

### Documentation de FranceConnect

- Fournisseur d'Identité (FI): [ici](https://partenaires.franceconnect.gouv.fr/fcp/fournisseur-identite)
- Fournisseur de Service (FS): [ici](https://partenaires.franceconnect.gouv.fr/fcp/fournisseur-service)

### Ré-initialiser la base de données

#### Avec les données de test (_fixtures_) : Utiliser le Makefile

Pour simplifier le lancement de certaines commandes, un Makefile est disponible. Exemples de commandes :

```shell
make destroy-rebuild-dev-db
```

Sur Windows, la commande `make` n'est pas disponible ;
Il faut passer chaque commande du `Makefile` (fichier présent à la racine du projet) les unes après les autres.

#### Avec des données existantes

Si vous avez des données existantes, vous pouvez d'abord les sauvegarder :

```shell
python manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json
```

Puis il vous faudra recréer la base de donnée PostgreSQL :

- Dans le shell :

  ```shell
  psql
  ```

- Puis, dans l'invite de commande `psql` :

  ```sql
  DROP DATABASE aidants_connect;
  CREATE DATABASE aidants_connect OWNER aidants_connect_team;
  ALTER USER aidants_connect_team CREATEDB;
  \q
  ```

Ensuite, de retour dans le _shell_, pour lancer les migrations :

```shell
python manage.py migrate
```

Enfin, chargez les données :

- Soit des données sauvegardées précédement :

  ```shell
  python manage.py loaddata db.json
  ```

- Soit repartir de zéro en recréant un _superuser_ (plus de détails dans la section [Installer l'application](#installer-lapplication)) :

  ```shell
  python manage.py createsuperuser
  ```

### Purger les connexions expirées

Les objets Django de type `Connection` repésentent une forme de cache pendant l'établissement de la connexion FranceConnect.
À ce titre, ils contiennent des données personnelles et doivent donc être purgés régulièrement pour respecter nos engagements en la matière.
Pour ce faire, il suffit d'exécuter ou de planifier la commande suivante :

```shell
python manage.py delete_expired_connections
```

### Calcul de `HASH_FC_AS_FI_SECRET` à partir de la valeur de `FC_AS_FI_SECRET`

Il faut utiliser `generate_sha256_hash`.

```python
from aidants_connect_web.utilities import generate_sha256_hash
generate_sha256_hash("VALUE_FC_AS_FI_SECRET".encode("utf-8"))
```

## Troubleshooting

### Erreur `ld: library not found for -lssl`

Si la commande précédente déclenche le message d'erreur suivant `ld: library not found for -lssl`, essayez :

```shell
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/
```

### Si vous avez un Mac M1

- Si, lors de l'installation de `psycopg2-binary`, vous avez le message d'erreur suivant :

```
 ld: warning: directory not found for option '-L/usr/bin/openssl/lib/'
    ld: library not found for -lssl
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'clang' failed with exit status 1
```

- Si vous avez installé `openssl` via `brew`, vous pouvez essayer la commande suivante :

```shell
env LDFLAGS='-L/opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/opt/readline/lib' pip install -r requirements.txt
```

> https://stackoverflow.com/a/42264168

- Si, lors de l'installation de `pillow` vous avez le message d'erreur suivant :

```shell
The headers or library files could not be found for zlib,
    a required dependency when compiling Pillow from source.

```

Vous pouvez essayer :

```shell
brew install libjpeg
```

> https://github.com/python-pillow/Pillow/issues/5042#issuecomment-746681171

## Les tests fonctionnels échouent de manière inexpliquée

Dans de rares cas (comportement observé à ce jour sur une seule machine de dev), les tests d'intégration échouent car _la première connexion_ à une URL via l'API Selenium plante de manière inexpliquée.

Un contournement empirique a été mis en place ; si vous rencontrez ce problème vous pouvez l'activer en passant à `True` la variable d'environnement `BYPASS_FIRST_LIVESERVER_CONNECTION` dans votre fichier `.env`.


![build](https://ci.ext.devshift.net/buildStatus/icon?job=app-sre-qontract-reconcile-gh-build-master)
![license](https://img.shields.io/github/license/app-sre/qontract-reconcile.svg?style=flat)

# qontract-reconcile

A tool to reconcile services with their desired state as defined in App-Interface.
Additional tools that use the libraries created by the reconciliations are also hosted here.

## Subcommands

### qontract-reconcile

```text
  acs-notifiers                   Manages RHACS notifier configurations
  acs-policies                    Manages RHACS security policy configurations
  acs-rbac                        Manages RHACS rbac configuration
  advanced-upgrade-scheduler      Manage Cluster Upgrade Policy schedules in
                                  OCM organizations based on OCM labels.
  aws-account-manager             Create and manage AWS accounts.
  aws-ami-cleanup                 Cleanup old and unused AMIs.
  aws-ami-share                   Share AMI and AMI tags between accounts.
  aws-cloudwatch-log-retention    Set up retention period for Cloudwatch logs.
  aws-ecr-image-pull-secrets      Generate AWS ECR image pull secrets and
                                  store them in Vault.
  aws-garbage-collector           Delete orphan AWS resources.
  aws-iam-keys                    Delete IAM access keys by access key ID.
  aws-iam-password-reset          Reset IAM user password by user reference.
  aws-saml-idp                    Manage the SAML IDP config for all AWS
                                  accounts.
  aws-saml-roles                  Manage the SAML IAM roles for all AWS
                                  accounts with SSO enabled.
  aws-support-cases-sos           Scan AWS support cases for reports of leaked
                                  keys and remove them (only submits PR)
  aws-version-sync                Sync AWS asset version numbers to App-
                                  Interface
  blackbox-exporter-endpoint-monitoring
                                  Manages Prometheus Probe resources for
                                  blackbox-exporter
  change-owners                   Detects owners for changes in app-interface
                                  PRs and allows them to self-service merge.
  cluster-auth-rhidp              Manages the OCM subscription labels for
                                  clusters with RHIDP authentication. Part of
                                  RHIDP.
  cluster-deployment-mapper       Maps ClusterDeployment resources to Cluster
                                  IDs.
  cna-resources                   Manage Cloud Resources using Cloud Native
                                  Assets (CNA).
  dashdotdb-cso                   Collects the ImageManifestVuln CRs from all
                                  the clusters and posts them to Dashdotdb.
  dashdotdb-dora                  Collects dora metrics.
  dashdotdb-dvo                   Collects the DeploymentValidations from all
                                  the clusters and posts them to Dashdotdb.
  dashdotdb-slo                   Collects the ServiceSloMetrics from all the
                                  clusters and posts them to Dashdotdb.
  database-access-manager         Manage Databases and Database Users.
  deadmanssnitch                  Automate Deadmanssnitch Creation/Deletion
  dynatrace-token-provider        Automatically provide dedicated Dynatrace
                                  tokens to management clusters
  email-sender                    Send email notifications to app-interface
                                  audience.
  external-resources              Manages External Resources
  gabi-authorized-users           Manages user access for GABI instances.
  gcr-mirror                      Mirrors external images into Google
                                  Container Registry.
  github                          Configures the teams and members in a GitHub
                                  org.
  github-owners                   Configures owners in a GitHub org.
  github-repo-invites             Accept GitHub repository invitations for
                                  known repositories.
  github-repo-permissions-validator
                                  Validates permissions in github
                                  repositories.
  github-users                    Validate compliance of GitHub user profiles.
  github-validator                Validates GitHub organization settings.
  gitlab-fork-compliance          Ensures that forks of App Interface are
                                  compliant.
  gitlab-housekeeping             Manage issues and merge requests on GitLab
                                  projects.
  gitlab-labeler                  Guesses and adds labels to merge requests
                                  according to changed paths.
  gitlab-members                  Manage GitLab group members.
  gitlab-mr-sqs-consumer          Listen to SQS and creates MRs out of the
                                  messages.
  gitlab-owners                   Manages labels on gitlab merge requests
                                  based on OWNERS files schema.
  gitlab-permissions              Manage permissions on GitLab projects.
  gitlab-projects                 Create GitLab projects.
  glitchtip                       Configure and enforce glitchtip instance
                                  configuration.
  glitchtip-project-alerts        Configure Glitchtip project alerts.
  glitchtip-project-dsn           Glitchtip project dsn as openshift secret.
  integrations-manager            Manages Qontract Reconcile integrations.
  jenkins-job-builder             Manage Jenkins jobs configurations using
                                  jenkins-jobs.
  jenkins-job-builds-cleaner      Clean up jenkins job history.
  jenkins-job-cleaner             Delete Jenkins jobs in multiple tenant
                                  instances.
  jenkins-roles                   Manage Jenkins roles association via REST
                                  API.
  jenkins-webhooks                Manage web hooks to Jenkins jobs.
  jenkins-webhooks-cleaner        Remove webhooks to previous Jenkins
                                  instances.
  jenkins-worker-fleets           Manage Jenkins worker fleets via JCasC.
  jira-permissions-validator      Validate permissions in Jira.
  jira-watcher                    Watch for changes in Jira boards and notify
                                  on Slack.
  ldap-groups                     Manages LDAP groups based on App-Interface
                                  roles.
  ldap-users                      Removes users which are not found in LDAP
                                  search.
  ocm-additional-routers          Manage additional routers in OCM.
  ocm-addons                      Manages cluster Addons in OCM.
  ocm-addons-upgrade-scheduler-org
                                  Manage Addons Upgrade Policy schedules in
                                  OCM organizations.
  ocm-addons-upgrade-tests-trigger
                                  Trigger jenkins jobs following Addon
                                  upgrades.
  ocm-aws-infrastructure-access   Grants AWS infrastructure access to members
                                  in AWS groups via OCM.
  ocm-clusters                    Manages clusters via OCM.
  ocm-external-configuration-labels
                                  Manage External Configuration labels in OCM.
  ocm-github-idp                  Manage GitHub Identity Providers in OCM.
  ocm-groups                      Manage membership in OpenShift groups via
                                  OCM.
  ocm-internal-notifications      Notifications to internal Red Hat users
                                  based on conditions in OCM.
  ocm-labels                      Manage cluster OCM labels.
  ocm-machine-pools               Manage Machine Pools in OCM.
  ocm-oidc-idp                    Manage OIDC cluster configuration in OCM
                                  organizations based on OCM labels. Part of
                                  RHIDP.
  ocm-standalone-user-management  Manages OCM cluster usergroups and
                                  notifications via OCM labels.
  ocm-update-recommended-version  Update recommended version for OCM orgs
  ocm-upgrade-scheduler-org       Manage Upgrade Policy schedules in OCM
                                  organizations.
  ocm-upgrade-scheduler-org-updater
                                  Update Upgrade Policy schedules in OCM
                                  organizations.
  openshift-cluster-bots          Manages dedicated-admin and cluster-admin
                                  creds.
  openshift-clusterrolebindings   Configures ClusterRolebindings in OpenShift
                                  clusters.
  openshift-groups                Manages OpenShift Groups.
  openshift-limitranges           Manages OpenShift LimitRange objects.
  openshift-namespace-labels      Manages labels on OpenShift namespaces.
  openshift-namespaces            Manages OpenShift Namespaces.
  openshift-network-policies      Manages OpenShift NetworkPolicies.
  openshift-prometheus-rules      Manages OpenShift Prometheus Rules.
  openshift-resourcequotas        Manages OpenShift ResourceQuota objects.
  openshift-resources             Manages OpenShift Resources.
  openshift-rolebindings          Configures Rolebindings in OpenShift
                                  clusters.
  openshift-routes                Manages OpenShift Routes.
  openshift-saas-deploy           Manage OpenShift resources defined in Saas
                                  files.
  openshift-saas-deploy-change-tester
                                  Runs openshift-saas-deploy for each saas-
                                  file that changed within a bundle.
  openshift-saas-deploy-trigger-cleaner
                                  Clean up deployment related resources.
  openshift-saas-deploy-trigger-configs
                                  Trigger deployments when configuration
                                  changes.
  openshift-saas-deploy-trigger-images
                                  Trigger deployments when images are pushed.
  openshift-saas-deploy-trigger-moving-commits
                                  Trigger deployments when a commit changed
                                  for a ref.
  openshift-saas-deploy-trigger-upstream-jobs
                                  Trigger deployments when upstream job runs.
  openshift-serviceaccount-tokens
                                  Use OpenShift ServiceAccount tokens across
                                  namespaces/clusters.
  openshift-tekton-resources      Manages custom resources for Tekton based
                                  deployments.
  openshift-upgrade-watcher       Watches for OpenShift upgrades and sends
                                  notifications.
  openshift-users                 Deletion of users from OpenShift clusters.
  openshift-vault-secrets         Manages OpenShift Secrets from Vault.
  prometheus-rules-tester         Tests prometheus rules using promtool.
  quay-membership                 Configures the teams and members in Quay.
  quay-mirror                     Mirrors external images into Quay.
  quay-mirror-org                 Mirrors entire Quay orgs.
  quay-permissions                Manage permissions for Quay Repositories.
  quay-repos                      Creates and Manages Quay Repos.
  query-validator                 Validate queries to maintain consumer schema
                                  compatibility.
  requests-sender                 Send emails to users based on requests
                                  submitted to app-interface.
  resource-scraper                Get resources from clusters and store in
                                  Vault.
  resource-template-tester        Tests templating of resources.
  rhidp-sso-client                Manage Keycloak SSO clients for OCM
                                  clusters. Part of RHIDP.
  saas-auto-promotions-manager    Manage auto-promotions defined in SaaS files
  saas-file-validator             Validates Saas files.
  sendgrid-teammates              Manages SendGrid teammates for a given
                                  account.
  service-dependencies            Validate dependencies are defined for each
                                  service.
  signalfx-prometheus-endpoint-monitoring
                                  Manages Prometheus Probe resources for
                                  signalfx exporter
  skupper-network                 Manages Skupper Networks.
  slack-usergroups                Manage Slack User Groups (channels and
                                  users).
  sql-query                       Runs SQL Queries against app-interface RDS
                                  resources.
  status-board-exporter           Export Product and Application informnation
                                  to Status Board.
  status-page-components          Manages components on statuspage.io hosted
                                  status pages.
  status-page-maintenances        Manages maintenances on statuspage.io hosted
                                  status pages.
  template-renderer               Render datafile templates in app-interface.
  template-validator              Test app-interface templates.
  terraform-aws-route53           Manage AWS Route53 resources using
                                  Terraform.
  terraform-cloudflare-dns        Manage Cloudflare DNS using Terraform.
  terraform-cloudflare-resources  Manage Cloudflare Resources using Terraform.
  terraform-cloudflare-users      Manage Cloudflare Users using Terraform.
  terraform-init                  Initialize AWS accounts for Terraform usage.
  terraform-repo                  Manages raw HCL Terraform from a separate
                                  repository.
  terraform-resources             Manage AWS Resources using Terraform.
  terraform-tgw-attachments       Manages Transit Gateway attachments.
  terraform-users                 Manage AWS users using Terraform.
  terraform-vpc-peerings          Manage VPC peerings between OSD clusters and
                                  AWS accounts or other OSD clusters.
  terraform-vpc-resources         Manage VPC creation
  unleash-feature-toggles         Manage Unleash feature toggles.
  vault-replication               Allow vault to replicate secrets to other
                                  instances.
  version-gate-approver           Approves OCM cluster upgrade version gates.
  vpc-peerings-validator          Validates that VPC peerings do not exist
                                  between public and internal clusters.
```

### tools

- `app-interface-metrics-exporter`: Exports metrics from App-Interface.
- `app-interface-reporter`: Creates service reports and submits PR to App-Interface.
- `qontract-cli`: A cli tool for qontract (currently very good at getting information).

## Usage

Use [config.toml.example](config.toml.example) as a template to create a `config.toml` file.

Run a reconcile integration like this:

```sh
qontract-reconcile --config config.toml --dry-run <subcommand>

# review output and run without `--dry-run` to perform actual changes
qontract-reconcile --config config.toml <subcommand>
```

> Note: you can use the `QONTRACT_CONFIG` environment variable instead of using `--config`.

## OpenShift usage

OpenShift templates can be found [here](/openshift/qontract-reconcile.yaml). In order to add integrations there please use the [helm](/helm/README.md) chart provided.

## Installation

This project targets Python version 3.11.x for best compatibility. Verify the Python3 version that your shell is using with `python3 --version`. You can optionally use a tool like [pyenv](https://github.com/pyenv/pyenv) or [asdf](https://asdf-vm.com/) to manage Python versions on your computer.

Create and enter the [virtualenv](https://virtualenv.pypa.io/en/latest/) environment:

```sh
python3 -m venv venv
source venv/bin/activate

# make sure you are running the latest setuptools
pip install --upgrade pip setuptools
```

Install build prerequisites for [psycopg2](https://www.psycopg.org/docs/install.html#build-prerequisites):

- A C compiler
- The Python header files. They are usually installed in a package such as `python-dev` or `python3-dev`.
- `libpq` and `pg_config`
  - The `libpq` header files. They are usually installed in a package such as `libpq-dev`.
  - The `pg_config` program: it is usually installed by the `libpq-dev` package but sometimes it is not in a `PATH` directory.
  - On macOS, can be installed via `brew install libpq`. Make sure the installation path is added to your PATH, otherwise `pg_config` will not be available.
  - On Fedora, it can be installed with `dnf install libpq-devel`

Note:
In macOS with M1/M2 `pip` will fail to install in a virtualenvironment unless LDFLAGS reference the openssl library path. It can be fixed with `export LDFLAGS="-I/opt/homebrew/opt/openssl/include -L/opt/homebrew/opt/openssl/lib"`.

Install the package:

```sh
pip install .

# or use this for development mode so rebuild/reinstall isn't necessary after
# each change that is made during development
pip install -e .

# optionally install all test/type dependencies - useful when writing tests,
# auto-completion in your IDE, etc.
pip install -r ./requirements/requirements-dev.txt
```

If the commands above don't work maybe you need to install the `python-devel` and `gcc-c++` packages.
You may also need to first [install a rust compiler](https://www.rust-lang.org/tools/install) ([Mac OS directions](https://sourabhbajaj.com/mac-setup/Rust/)) and then run `python3 -m pip install --upgrade pip setuptools_rust`.

Install runtime requirements:

Versions can be found in [qontract-reconcile-base Dockerfile](https://github.com/app-sre/container-images/blob/master/qontract-reconcile-base/Dockerfile).

- amtool
- git-secrets
- helm
- kubectl
- oc
- promtool
- skopeo
- terraform

### Requirements

Please see [setup.py](setup.py).

All requirements files are gathered in [./requirements/](./requirements).
It consists of:

- [requirements-test.txt](requirements/requirements-test.txt) for unit test and linting dependencies
- [requirements-type.txt](requirements/requirements-type.txt) for type checking dependencies
- [requirements-dev.txt](requirements/requirements-dev.txt) installs all above mentioned dependencies

### Image build

In order to speed up frequent builds and avoid issues with dependencies, docker image makes use
[`qontract-reconcile-build`](https://quay.io/repository/app-sre/qontract-reconcile-base?tag=latest&tab=tags)
image. See [`app-sre/coontainer-images`](https://github.com/app-sre/container-images) repository
if you want to make changes to the base image.

This repo [`Dockerfile`](dockerfiles/Dockerfile) must only contain instructions related to the python code build.

The [README](dockerfiles/README.md) contains more information about the Dockerfile and the build stages.

## CI Tooling

This project uses [tox](https://tox.readthedocs.io/en/latest/) for running
tests, linting/static analysis, and type checkers. Some of the more common
commands have been provided below, but see the tox docs for more complete
documentation.

Running all checks (tests, linting, and type checkers):

```
tox
```

To run the checks faster (run in parallel):

```
tox -p
```

Running specific checks (can be much faster):

```
# Only run unit tests using Python 3.6
tox -e py36

# Only run linters
tox -e lint

# Only run the type checker
tox -e type

# Look at tox.ini for usage of posargs, this allows us to override which
# options are passed to the CLI where it's being used. This can be helpful
# for type checking a specific file, or running a subset of unit tests (this
# can be even faster).
tox -e type -- reconcile/utils/slack_api.py
```

## Run reconcile loop for an integration locally in a container

This is currently only tested with the docker container engine.

For more flexible way to run in container, please see [qontract-development-cli](https://github.com/app-sre/qontract-development-cli).

### Prepare config.toml

Make sure the file `./config.dev.toml` exists and contains your current configuration.
Your `config.dev.toml` should point to the following qontract-server address:

```
[graphql]
server = "http://host.docker.internal:4000/graphql"
```

### Run qontract-server

Start the [qontract-server](https://github.com/app-sre/qontract-server) in a different window, e.g., via:

```
qontract-server$ make dev
```

### Trigger integration

```
make dev-reconcile-loop INTEGRATION_NAME=terraform-resources DRY_RUN=--dry-run INTEGRATION_EXTRA_ARGS=--light SLEEP_DURATION_SECS=100
```

## Query Classes

We use [qenerate](https://github.com/app-sre/qenerate) to generate data classes for GQL queries.
GQL definitions and generated classes can be found [here](reconcile/gql_definitions/).

### Workflow

1. Define your query or fragment in a `.gql` file somewhere in `reconcile/gql_definitions`.
2. Every gql file must hold exactly one `query` OR `fragment` definition. You must not have multiple definitions within one file.
3. Do not forget to add `# qenerate: plugin=pydantic_v1` in the beginning of the file. This tells `qenerate` which plugin is used to render the code.
4. Have an up-to-date schema available at localhost:4000
5. `make gql-introspection` gets the type definitions. They will be stored in `reconcile/gql_definitions/introspection.json`
6. `make gql-query-classes` generates the data classes for your queries and fragments

## Troubleshooting

`faulthandler` is enabled for this project and SIGUSR1 is registered to dump the traceback. To do so, you can use `kill -USR1 pid` where pid is the ID of the qontract-reconcile process.

## Code style guide

Qontract-reconcile uses [PEP8](https://peps.python.org/pep-0008/) as the code style guide.
The style is enforced via [PR checks](#ci-tooling) with the help of the following utilities:

- [Ruff - An extremely fast Python linter and code formatter, written in Rust.](https://docs.astral.sh/ruff/)
- [Mypy](https://mypy.readthedocs.io/en/stable/)

Run `make format` before you commit your changes to keep the code compliant.

## Release

Release version are calculated from git tags of the form X.Y.Z.

- If the current commit has such a tag, it will be used as is
- Otherwise the latest tag of that format is used and:
  - the patch label (Z) is incremented
  - the string `.pre<count>+<commitid>` is appended. `<count>` is the number of commits since the X.Y.Z tag. `<commitid> is... the current commitid.

After the PR is merged, a CI job will be triggered that will publish the package to pypi: <https://pypi.org/project/qontract-reconcile>.

## Licence

[Apache License Version 2.0](LICENSE).

## Authors

These tools have been written by the [Red Hat App-SRE Team](mailto:sd-app-sre@redhat.com).


[![Typing SVG](https://readme-typing-svg.demolab.com?font=Fira+Code&size=40&duration=7000&pause=1000&random=false&width=1200&height=100&lines=Simplify+Conversational+AI+Development)](https://git.io/typing-svg)

| 📘 [Tutorials](https://speechbrain.github.io/tutorial_basics.html) | 🌐 [Website](https://speechbrain.github.io/) | 📚 [Documentation](https://speechbrain.readthedocs.io/en/latest/index.html) | 🤝 [Contributing](https://speechbrain.readthedocs.io/en/latest/contributing.html) | 🤗 [HuggingFace](https://huggingface.co/speechbrain) | ▶ [YouTube](https://www.youtube.com/@SpeechBrainProject) | 🐦 [X](https://twitter.com/SpeechBrain1) |

![GitHub Repo stars](https://img.shields.io/github/stars/speechbrain/speechbrain?style=social) *Please, help our community project. Star on GitHub!*

**Exciting News (January, 2024):** Discover what is new in SpeechBrain 1.0 [here](https://colab.research.google.com/drive/1IEPfKRuvJRSjoxu22GZhb3czfVHsAy0s?usp=sharing)!
#
# 🗣💬 What SpeechBrain Offers

- SpeechBrain is an **open-source** [PyTorch](https://pytorch.org/) toolkit that accelerates **Conversational AI** development, i.e., the technology behind *speech assistants*, *chatbots*, and *large language models*.

- It is crafted for fast and easy creation of advanced technologies for **Speech** and **Text** Processing.

## 🌐  Vision
- With the rise of [deep learning](https://www.deeplearningbook.org/), once-distant domains like speech processing and NLP are now very close. A well-designed neural network and large datasets are all you need.

- We think it is now time for a **holistic toolkit** that, mimicking the human brain, jointly supports diverse technologies for complex Conversational AI systems.

- This spans *speech recognition*, *speaker recognition*, *speech enhancement*, *speech separation*, *language modeling*, *dialogue*, and beyond.

## 📚 Training Recipes
- We share over 200 competitive training [recipes](https://github.com/speechbrain/speechbrain/tree/develop/recipes) on more than 40 datasets supporting 20 speech and text processing tasks (see below).

- We support both training from scratch and fine-tuning pretrained models such as [Whisper](https://huggingface.co/openai/whisper-large), [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2), [WavLM](https://huggingface.co/docs/transformers/model_doc/wavlm), [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert), [GPT2](https://huggingface.co/gpt2), [Llama2](https://huggingface.co/docs/transformers/model_doc/llama2), and beyond. The models on [HuggingFace](https://huggingface.co/) can be easily plugged in and fine-tuned.

- For any task, you train the model using these commands:
```python
python train.py hparams/train.yaml
```

- The hyperparameters are encapsulated in a YAML file, while the training process is orchestrated through a Python script.

- We maintained a consistent code structure across different tasks.

- For better replicability, training logs and checkpoints are hosted on Dropbox.

##    Pretrained Models and Inference

- Access over 100 pretrained models hosted on [HuggingFace](https://huggingface.co/speechbrain).
- Each model comes with a user-friendly interface for seamless inference. For example, transcribing speech using a pretrained model requires just three lines of code:

```python
from speechbrain.inference import EncoderDecoderASR

asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-conformer-transformerlm-librispeech", savedir="pretrained_models/asr-transformer-transformerlm-librispeech")
asr_model.transcribe_file("speechbrain/asr-conformer-transformerlm-librispeech/example.wav")
```

##      Documentation
- We are deeply dedicated to promoting inclusivity and education.
- We have authored over 30 [tutorials](https://speechbrain.github.io/) on Google Colab that not only describe how SpeechBrain works but also help users familiarize themselves with Conversational AI.
- Every class or function has clear explanations and examples that you can run. Check out the [documentation](https://speechbrain.readthedocs.io/en/latest/index.html) for more details 📚.

## 🎯 Use Cases
- 🚀 **Research Acceleration**: Speeding up academic and industrial research. You can develop and integrate new models effortlessly, comparing their performance against our baselines.

- ⚡ **Rapid Prototyping**: Ideal for quick prototyping in time-sensitive projects.

- 🎓 **Educational Tool**: SpeechBrain's simplicity makes it a valuable educational resource. It is used by institutions like [Mila](https://mila.quebec/en/), [Concordia University](https://www.concordia.ca/), [Avignon University](https://univ-avignon.fr/en/), and many others for student training.

#
# 🚀 Quick Start

To get started with SpeechBrain, follow these simple steps:

## 🛠 Installation

### Install via PyPI

1. Install SpeechBrain using PyPI:

    ```bash
    pip install speechbrain
    ```

2. Access SpeechBrain in your Python code:

    ```python
    import speechbrain as sb
    ```

### Install from GitHub
This installation is recommended for users who wish to conduct experiments and customize the toolkit according to their needs.

1. Clone the GitHub repository and install the requirements:

    ```bash
    git clone https://github.com/speechbrain/speechbrain.git
    cd speechbrain
    pip install -r requirements.txt
    pip install --editable .
    ```

2. Access SpeechBrain in your Python code:

    ```python
    import speechbrain as sb
    ```

Any modifications made to the `speechbrain` package will be automatically reflected, thanks to the `--editable` flag.

## ✔ Test Installation

Ensure your installation is correct by running the following commands:

```bash
pytest tests
pytest --doctest-modules speechbrain
```

## 🏃♂ Running an Experiment

In SpeechBrain, you can train a model for any task using the following steps:

```python
cd recipes/<dataset>/<task>/
python experiment.py params.yaml
```

The results will be saved in the `output_folder` specified in the YAML file.

## 📘 Learning SpeechBrain

- **Website:** Explore general information on the [official website](https://speechbrain.github.io).

- **Tutorials:** Start with [basic tutorials](https://speechbrain.github.io/tutorial_basics.html) covering fundamental functionalities. Find advanced tutorials and topics in the Tutorials menu on the [SpeechBrain website](https://speechbrain.github.io).

- **Documentation:** Detailed information on the SpeechBrain API, contribution guidelines, and code is available in the [documentation](https://speechbrain.readthedocs.io/en/latest/index.html).

#
# 🔧 Supported Technologies
- SpeechBrain is a versatile framework designed for implementing a wide range of technologies within the field of Conversational AI.
- It excels not only in individual task implementations but also in combining various technologies into complex pipelines.

## 🎙 Speech/Audio Processing
| Tasks        | Datasets           | Technologies/Models  |
| ------------- |-------------| -----|
| Speech Recognition      | [AISHELL-1](https://github.com/speechbrain/speechbrain/tree/develop/recipes/AISHELL-1), [CommonVoice](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonVoice), [DVoice](https://github.com/speechbrain/speechbrain/tree/develop/recipes/DVoice), [KsponSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/KsponSpeech), [LibriSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech), [MEDIA](https://github.com/speechbrain/speechbrain/tree/develop/recipes/MEDIA), [RescueSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/RescueSpeech), [Switchboard](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Switchboard), [TIMIT](https://github.com/speechbrain/speechbrain/tree/develop/recipes/TIMIT), [Tedlium2](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Tedlium2), [Voicebank](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Voicebank) | [CTC](https://www.cs.toronto.edu/~graves/icml_2006.pdf), [Transducers](https://arxiv.org/pdf/1211.3711.pdf?origin=publication_detail), [Transformers](https://arxiv.org/abs/1706.03762), [Seq2Seq](http://zhaoshuaijiang.com/file/Hybrid_CTC_Attention_Architecture_for_End-to-End_Speech_Recognition.pdf), [Beamsearch techniques for CTC](https://arxiv.org/pdf/1911.01629.pdf),[seq2seq](https://arxiv.org/abs/1904.02619.pdf),[transducers](https://www.merl.com/publications/docs/TR2017-190.pdf)), [Rescoring](https://arxiv.org/pdf/1612.02695.pdf), [Conformer](https://arxiv.org/abs/2005.08100), [Branchformer](https://arxiv.org/abs/2207.02971), [Hyperconformer](https://arxiv.org/abs/2305.18281), [Kaldi2-FST](https://github.com/k2-fsa/k2) |
| Speaker Recognition      | [VoxCeleb](https://github.com/speechbrain/speechbrain/tree/develop/recipes/VoxCeleb) | [ECAPA-TDNN](https://arxiv.org/abs/2005.07143), [ResNET](https://arxiv.org/pdf/1910.12592.pdf), [Xvectors](https://www.danielpovey.com/files/2018_icassp_xvectors.pdf), [PLDA](https://ieeexplore.ieee.org/document/6639151), [Score Normalization](https://www.sciencedirect.com/science/article/abs/pii/S1051200499903603) |
| Speech Separation      | [WSJ0Mix](https://github.com/speechbrain/speechbrain/tree/develop/recipes/WSJ0Mix), [LibriMix](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriMix), [WHAM!](https://github.com/speechbrain/speechbrain/tree/develop/recipes/WHAMandWHAMR), [WHAMR!](https://github.com/speechbrain/speechbrain/tree/develop/recipes/WHAMandWHAMR), [Aishell1Mix](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Aishell1Mix), [BinauralWSJ0Mix](https://github.com/speechbrain/speechbrain/tree/develop/recipes/BinauralWSJ0Mix) | [SepFormer](https://arxiv.org/abs/2010.13154), [RESepFormer](https://arxiv.org/abs/2206.09507), [SkiM](https://arxiv.org/abs/2201.10800), [DualPath RNN](https://arxiv.org/abs/1910.06379), [ConvTasNET](https://arxiv.org/abs/1809.07454) |
| Speech Enhancement      | [DNS](https://github.com/speechbrain/speechbrain/tree/develop/recipes/DNS), [Voicebank](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Voicebank) | [SepFormer](https://arxiv.org/abs/2010.13154), [MetricGAN](https://arxiv.org/abs/1905.04874), [MetricGAN-U](https://arxiv.org/abs/2110.05866), [SEGAN](https://arxiv.org/abs/1703.09452), [spectral masking](http://staff.ustc.edu.cn/~jundu/Publications/publications/Trans2015_Xu.pdf), [time masking](http://staff.ustc.edu.cn/~jundu/Publications/publications/Trans2015_Xu.pdf) |
| Text-to-Speech      | [LJSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LJSpeech), [LibriTTS](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriTTS) | [Tacotron2](https://arxiv.org/abs/1712.05884), [Zero-Shot Multi-Speaker Tacotron2](https://arxiv.org/abs/2112.02418), [FastSpeech2](https://arxiv.org/abs/2006.04558) |
| Vocoding      | [LJSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LJSpeech), [LibriTTS](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriTTS) | [HiFiGAN](https://arxiv.org/abs/2010.05646), [DiffWave](https://arxiv.org/abs/2009.09761)
| Spoken Language Understanding | [MEDIA](https://github.com/speechbrain/speechbrain/tree/develop/recipes/MEDIA), [SLURP](https://github.com/speechbrain/speechbrain/tree/develop/recipes/SLURP), [Fluent Speech Commands](https://github.com/speechbrain/speechbrain/tree/develop/recipes/fluent-speech-commands), [Timers-and-Such](https://github.com/speechbrain/speechbrain/tree/develop/recipes/timers-and-such)  | [Direct SLU](https://arxiv.org/abs/2104.01604), [Decoupled SLU](https://arxiv.org/abs/2104.01604), [Multistage SLU](https://arxiv.org/abs/2104.01604) |
| Speech-to-Speech Translation  | [CVSS](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CVSS) | [Discrete Hubert](https://arxiv.org/pdf/2106.07447.pdf), [HiFiGAN](https://arxiv.org/abs/2010.05646), [wav2vec2](https://arxiv.org/abs/2006.11477) |
| Speech Translation  | [Fisher CallHome (Spanish)](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Fisher-Callhome-Spanish), [IWSLT22(lowresource)](https://github.com/speechbrain/speechbrain/tree/develop/recipes/IWSLT22_lowresource) | [wav2vec2](https://arxiv.org/abs/2006.11477) |
| Emotion Classification      | [IEMOCAP](https://github.com/speechbrain/speechbrain/tree/develop/recipes/IEMOCAP), [ZaionEmotionDataset](https://github.com/speechbrain/speechbrain/tree/develop/recipes/ZaionEmotionDataset) | [ECAPA-TDNN](https://arxiv.org/abs/2005.07143), [wav2vec2](https://arxiv.org/abs/2006.11477), [Emotion Diarization](https://arxiv.org/abs/2306.12991) |
| Language Identification | [VoxLingua107](https://github.com/speechbrain/speechbrain/tree/develop/recipes/VoxLingua107), [CommonLanguage](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonLanguage)| [ECAPA-TDNN](https://arxiv.org/abs/2005.07143) |
| Voice Activity Detection  | [LibriParty](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriParty) | [CRDNN](https://arxiv.org/abs/2106.04624) |
| Sound Classification  | [ESC50](https://github.com/speechbrain/speechbrain/tree/develop/recipes/ESC50), [UrbanSound](https://github.com/speechbrain/speechbrain/tree/develop/recipes/UrbanSound8k) | [CNN14](https://github.com/ranchlai/sound_classification), [ECAPA-TDNN](https://arxiv.org/abs/2005.07143) |
| Self-Supervised Learning | [CommonVoice](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonVoice), [LibriSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech) | [wav2vec2](https://arxiv.org/abs/2006.11477) |
| Interpretability | [ESC50](https://github.com/speechbrain/speechbrain/tree/develop/recipes/ESC50) | [Learning-to-Interpret (L2I)](https://proceedings.neurips.cc/paper_files/paper/2022/file/e53280d73dd5389e820f4a6250365b0e-Paper-Conference.pdf), [Non-Negative Matrix Factorization (NMF)](https://proceedings.neurips.cc/paper_files/paper/2022/file/e53280d73dd5389e820f4a6250365b0e-Paper-Conference.pdf), [PIQ](https://arxiv.org/abs/2303.12659) |
| Speech Generation | [AudioMNIST](https://github.com/speechbrain/speechbrain/tree/develop/recipes/AudioMNIST) | [Diffusion](https://arxiv.org/abs/2006.11239), [Latent Diffusion](https://arxiv.org/abs/2112.10752) |
| Metric Learning | [REAL-M](https://github.com/speechbrain/speechbrain/tree/develop/recipes/REAL-M/sisnr-estimation), [Voicebank](https://github.com/speechbrain/speechbrain/tree/develop/recipes/Voicebank) | [Blind SNR-Estimation](https://arxiv.org/abs/2002.08909), [PESQ Learning](https://arxiv.org/abs/2110.05866) |
| Alignment | [TIMIT](https://github.com/speechbrain/speechbrain/tree/develop/recipes/TIMIT) | [CTC](https://www.cs.toronto.edu/~graves/icml_2006.pdf), [Viterbi](https://www.cs.cmu.edu/~cga/behavior/rabiner1.pdf), [Forward Forward](https://www.cs.cmu.edu/~cga/behavior/rabiner1.pdf) |
| Diarization | [AMI](https://github.com/speechbrain/speechbrain/tree/develop/recipes/AMI) | [ECAPA-TDNN](https://arxiv.org/abs/2005.07143), [X-vectors](https://www.danielpovey.com/files/2018_icassp_xvectors.pdf), [Spectral Clustering](http://www.ifp.illinois.edu/~hning2/papers/Ning_spectral.pdf) |

## 📝 Text Processing
| Tasks        | Datasets           | Technologies/Models  |
| ------------- |-------------| -----|
| Language Modeling | [CommonVoice](https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonVoice), [LibriSpeech](https://github.com/speechbrain/speechbrain/tree/unstable-v0.6/recipes/LibriSpeech)| [n-grams](https://web.stanford.edu/~jurafsky/slp3/3.pdf), [RNNLM](https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf), [TransformerLM](https://arxiv.org/abs/1706.03762) |
| Response Generation | [MultiWOZ](https://github.com/speechbrain/speechbrain/tree/unstable-v0.6/recipes/MultiWOZ/response_generation)| [GPT2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), [Llama2](https://arxiv.org/abs/2307.09288) |
| Grapheme-to-Phoneme | [LibriSpeech](https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech) | [RNN](https://arxiv.org/abs/2207.13703), [Transformer](https://arxiv.org/abs/2207.13703), [Curriculum Learning](https://arxiv.org/abs/2207.13703), [Homograph loss](https://arxiv.org/abs/2207.13703) |

## 🔍 Additional Features

SpeechBrain includes a range of native functionalities that enhance the development of Conversational AI technologies. Here are some examples:

- **Training Orchestration:** The `Brain` class serves as a fully customizable tool for managing training and evaluation loops over data. It simplifies training loops while providing the flexibility to override any part of the process.

- **Hyperparameter Management:** A YAML-based hyperparameter file specifies all hyperparameters, from individual numbers (e.g., learning rate) to complete objects (e.g., custom models). This elegant solution drastically simplifies the training script.

- **Dynamic Dataloader:** Enables flexible and efficient data reading.

- **GPU Training:** Supports single and multi-GPU training, including distributed training.

- **Dynamic Batching:** On-the-fly dynamic batching enhances the efficient processing of variable-length signals.

- **Mixed-Precision Training:** Accelerates training through mixed-precision techniques.

- **Efficient Data Reading:** Reads large datasets efficiently from a shared Network File System (NFS) via [WebDataset](https://github.com/webdataset/webdataset).

- **Hugging Face Integration:** Interfaces seamlessly with [HuggingFace](https://huggingface.co/speechbrain) for popular models such as wav2vec2 and Hubert.

- **Orion Integration:** Interfaces with [Orion](https://github.com/Epistimio/orion) for hyperparameter tuning.

- **Speech Augmentation Techniques:** Includes SpecAugment, Noise, Reverberation, and more.

- **Data Preparation Scripts:** Includes scripts for preparing data for supported datasets.

SpeechBrain is rapidly evolving, with ongoing efforts to support a growing array of technologies in the future.

## 📊 Performance

- SpeechBrain integrates a variety of technologies, including those that achieves competitive or state-of-the-art performance.

- For a comprehensive overview of the achieved performance across different tasks, datasets, and technologies, please visit [here](https://github.com/speechbrain/speechbrain/blob/develop/PERFORMANCE.md).

#
# 📜 License

- SpeechBrain is released under the [Apache License, version 2.0](https://www.apache.org/licenses/LICENSE-2.0), a popular BSD-like license.
- You are free to redistribute SpeechBrain for both free and commercial purposes, with the condition of retaining license headers. Unlike the GPL, the Apache License is not viral, meaning you are not obligated to release modifications to the source code.

#
# 🔮Future Plans

We have ambitious plans for the future, with a focus on the following priorities:

- **Scale Up:** Our aim is to provide comprehensive recipes and technologies for training massive models on extensive datasets.

- **Scale Down:** While scaling up delivers unprecedented performance, we recognize the challenges of deploying large models in production scenarios. We are focusing on real-time, streamable, and small-footprint Conversational AI.

#
# 🤝 Contributing

- SpeechBrain is a community-driven project, led by a core team with the support of numerous international collaborators.
- We welcome contributions and ideas from the community. For more information, check [here](https://speechbrain.github.io/contributing.html).

#
# 🙏 Sponsors

- SpeechBrain is an academically driven project and relies on the passion and enthusiasm of its contributors.
- As we cannot rely on the resources of a large company, we deeply appreciate any form of support, including donations or collaboration with the core team.
- If you're interested in sponsoring SpeechBrain, please reach out to us at speechbrainproject@gmail.com.
- A heartfelt thank you to all our sponsors, including the current ones:

[](https://speechbrain.github.io/img/hf.ico)
[](https://viadialog.com/en/)
[](https://europe.naverlabs.com/)

[](https://www.ovhcloud.com/en-ca/)
[](https://usa.baidu.com/)
[](https://research.samsung.com/aicenter_cambridge)

[](https://mila.quebec/en/)
[](https://www.concordia.ca/)
[](https://lia.univ-avignon.fr/)
#
# 📖 Citing SpeechBrain

If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:

```bibtex
@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}
```


# Avalanche: an End-to-End Library for Continual Learning
**[Avalanche Website](https://avalanche.continualai.org)** | **[Getting Started](https://avalanche.continualai.org/getting-started)** | **[Examples](https://avalanche.continualai.org/examples)** | **[Tutorial](https://avalanche.continualai.org/from-zero-to-hero-tutorial)** | **[API Doc](https://avalanche-api.continualai.org)** | **[Paper](https://arxiv.org/abs/2104.00405)** | **[Twitter](https://twitter.com/AvalancheLib)**

[![unit test](https://github.com/ContinualAI/avalanche/actions/workflows/unit-test.yml/badge.svg)](https://github.com/ContinualAI/avalanche/actions/workflows/unit-test.yml)
[![syntax checking](https://github.com/ContinualAI/avalanche/actions/workflows/syntax.yml/badge.svg)](https://github.com/ContinualAI/avalanche/actions/workflows/syntax.yml)
[![docstring coverage](https://github.com/ContinualAI/avalanche-report/blob/main/badge/interrogate-badge.svg)](https://github.com/ContinualAI/avalanche-report/blob/main/docstring_coverage/documentation-coverage.txt)
[![Coverage Status](https://coveralls.io/repos/github/ContinualAI/avalanche/badge.svg)](https://coveralls.io/github/ContinualAI/avalanche)

**Avalanche** is an *end-to-end Continual Learning library* based on **Pytorch**, born within [ContinualAI](https://www.continualai.org/) with the unique goal of providing a shared and collaborative
open-source (MIT licensed) codebase for fast prototyping, training and reproducible evaluation of continual learning algorithms.

> :warning: Looking for **continual learning baselines**? In the [CL-Baseline](https://github.com/ContinualAI/continual-learning-baselines) sibling project based on Avalanche we reproduce seminal papers results you can directly use in **your experiments**!

Avalanche can help Continual Learning researchers in several ways:

- *Write less code, prototype faster & reduce errors*
- *Improve reproducibility, modularity and reusability*
- *Increase code efficiency, scalability & portability*
- *Augment impact and usability of your research products*

The library is organized into four main modules:

- [Benchmarks](avalanche/benchmarks): This module maintains a uniform API for data handling: mostly generating a stream of data from one or more datasets. It contains all the major CL benchmarks (similar to what has been done for torchvision).
- [Training](avalanche/training): This module provides all the necessary utilities concerning model training. This includes simple and efficient ways of implement new continual learning strategies as well as a set of pre-implemented CL baselines and state-of-the-art algorithms you will be able to use for comparison!
- [Evaluation](avalanche/evaluation): This module provides all the utilities and metrics that can help evaluate a CL algorithm with respect to all the factors we believe to be important for a continually learning system. It also includes advanced logging and plotting features, including native Tensorboard support.
- [Models](avalanche/models): This module provides utilities to implement model expansion and task-aware models, along with a set of pre-trained models and popular architectures that can be used for your continual learning experiment (similar to what has been done in torchvision.models).
- [Logging](avalanche/logging): It includes advanced logging and plotting features, including native stdout, file and TensorBoard support (How cool it is to have a complete, interactive dashboard, tracking your experiment metrics in real-time with a single line of code?)

_Avalanche_ the first experiment of an **End-to-end Library** for reproducible continual learning research & development where you can find benchmarks, algorithms, evaluation metrics and much more, in the same place.

Let's make it together :people_holding_hands: a wonderful ride! :balloon:

Check out below how you can start using Avalanche! :point_down:

Quick Example
----------------

```python
import torch
from torch.nn import CrossEntropyLoss
from torch.optim import SGD

from avalanche.benchmarks.classic import PermutedMNIST
from avalanche.models import SimpleMLP
from avalanche.training import Naive

# Config
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# model
model = SimpleMLP(num_classes=10)

# CL Benchmark Creation
perm_mnist = PermutedMNIST(n_experiences=3)
train_stream = perm_mnist.train_stream
test_stream = perm_mnist.test_stream

# Prepare for training & testing
optimizer = SGD(model.parameters(), lr=0.001, momentum=0.9)
criterion = CrossEntropyLoss()

# Continual learning strategy
cl_strategy = Naive(
    model, optimizer, criterion, train_mb_size=32, train_epochs=2,
    eval_mb_size=32, device=device)

# train and test loop over the stream of experiences
results = []
for train_exp in train_stream:
    cl_strategy.train(train_exp)
    results.append(cl_strategy.eval(test_stream))
```

Current Release
----------------

Avalanche is a framework in constant development. Thanks to the support of the [ContinualAI](https://www.continualai.org/) community and its active members we are quickly extending its features and improve its usability based on the demands of our research community!

A the moment, Avalanche is in **Beta**. We support [several *Benchmarks*, *Strategies* and *Metrics*](https://avalanche.continualai.org/getting-started/alpha-version), that make it, we believe, the best tool out there for your continual learning research! 💪

**You can install Avalanche by running `pip install avalanche-lib`.**
This will install the core Avalanche package. You can install Avalanche with extra packages to enable more functionalities.
Look [here](https://avalanche.continualai.org/getting-started/how-to-install) for a more complete guide on the different ways available to install Avalanche.

Getting Started
----------------

We know that learning a new tool may be tough at first. This is why we made Avalanche as easy as possible to learn with a set of resources that will help you along the way.
For example, you may start with our 5-minutes guide that will let you acquire the basics about Avalanche and how you can use it in your research project:

- [Getting Started Guide](https://avalanche.continualai.org/getting-started)

We have also prepared for you a large set of examples & snippets you can plug-in directly into your code and play with:

- [Avalanche Examples](https://avalanche.continualai.org/examples)

Having completed these two sections, you will already feel with superpowers ⚡, this is why we have also created an in-depth tutorial that will cover all the aspects of Avalanche in
detail and make you a true Continual Learner! :woman_student:

- [From Zero to Hero Tutorial](https://avalanche.continualai.org/from-zero-to-hero-tutorial)

Cite Avalanche
----------------

If you use Avalanche in your research project, please remember to cite our JMLR-MLOSS paper [https://jmlr.org/papers/v24/23-0130.html](https://jmlr.org/papers/v24/23-0130.html). This will help us make Avalanche better known in the machine learning community, ultimately making a better tool for everyone:

```
@article{JMLR:v24:23-0130,
  author  = {Antonio Carta and Lorenzo Pellegrini and Andrea Cossu and Hamed Hemati and Vincenzo Lomonaco},
  title   = {Avalanche: A PyTorch Library for Deep Continual Learning},
  journal = {Journal of Machine Learning Research},
  year    = {2023},
  volume  = {24},
  number  = {363},
  pages   = {1--6},
  url     = {http://jmlr.org/papers/v24/23-0130.html}
}
```

you can also cite the previous [CLVision @ CVPR2021](https://sites.google.com/view/clvision2021/overview?authuser=0) workshop paper: ["Avalanche: an End-to-End Library for Continual Learning"](https://arxiv.org/abs/2104.00405).

```
@InProceedings{lomonaco2021avalanche,
    title={Avalanche: an End-to-End Library for Continual Learning},
    author={Vincenzo Lomonaco and Lorenzo Pellegrini and Andrea Cossu and Antonio Carta and Gabriele Graffieti and Tyler L. Hayes and Matthias De Lange and Marc Masana and Jary Pomponi and Gido van de Ven and Martin Mundt and Qi She and Keiland Cooper and Jeremy Forest and Eden Belouadah and Simone Calderara and German I. Parisi and Fabio Cuzzolin and Andreas Tolias and Simone Scardapane and Luca Antiga and Subutai Amhad and Adrian Popescu and Christopher Kanan and Joost van de Weijer and Tinne Tuytelaars and Davide Bacciu and Davide Maltoni},
    booktitle={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition},
    series={2nd Continual Learning in Computer Vision Workshop},
    year={2021}
}
```

Maintained by ContinualAI Lab
----------------

*Avalanche* is the flagship open-source collaborative project of [ContinualAI](https://www.continualai.org/): a non-profit research organization and the largest open community on Continual Learning for AI.

Do you have a question, do you want to report an issue or simply ask for a new feature? Check out the [Questions & Issues](https://avalanche.continualai.org/questions-and-issues/ask-your-question) center. Do you want to improve Avalanche yourself? Follow these simple rules on [How to Contribute](https://avalanche.continualai.org/from-zero-to-hero-tutorial/09_contribute-to-avalanche).

The Avalanche project is maintained by the collaborative research team [ContinualAI Lab](https://www.continualai.org/lab/) and used extensively by the Units of the [ContinualAI Research (CLAIR)](https://www.continualai.org/research/) consortium, a research network of the major continual learning stakeholders around the world.

We are always looking for new awesome members willing to join the ContinualAI Lab, so check out our [official website](https://www.continualai.org/lab/) if you want to learn more about us and our activities, or [contact us](https://avalanche.continualai.org/contacts-and-links/the-team#contacts).

Learn more about the [Avalanche team and all the people who made it great](https://avalanche.continualai.org/contacts-and-links/the-team)!


# NLPretext

[![CI status](https://github.com/artefactory/NLPretext/actions/workflows/ci.yml/badge.svg?branch%3Amain&event%3Apush)](https://github.com/artefactory/NLPretext/actions/workflows/ci.yml?query=branch%3Amain)
[![CD status](https://github.com/artefactory/NLPretext/actions/workflows/cd.yml/badge.svg?event%3Arelease)](https://github.com/artefactory/NLPretext/actions/workflows/cd.yml?query=event%3Arelease)
[![Python Version](https://img.shields.io/badge/Python-3.8-informational.svg)](#supported-python-versions)
[![Dependencies Status](https://img.shields.io/badge/dependabots-active-informational.svg)](https://github.com/artefactory/NLPretext}/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Aapp%2Fdependabot)

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Security: bandit](https://img.shields.io/badge/security-bandit-informational.svg)](https://github.com/PyCQA/bandit)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-informational?logo=pre-commit&logoColor=white)](https://github.com/artefactory/NLPretext}/blob/main/.pre-commit-config.yaml)
[![Semantic Versions](https://img.shields.io/badge/%F0%9F%9A%80-semantic%20versions-informational.svg)](https://github.com/artefactory/NLPretext/releases)
[![Documentation](https://img.shields.io/badge/doc-sphinx-informational.svg)](https://github.com/artefactory/NLPretext}/tree/main/docs)
[![License](https://img.shields.io/badge/License-Apache%20Software%20License%202.0-informational.svg)](https://github.com/artefactory/NLPretext}/blob/main/LICENSE)

All the goto functions you need to handle NLP use-cases, integrated in NLPretext

# TL;DR

> *Working on an NLP project and tired of always looking for the same silly preprocessing functions on the web?*  :tired_face:

> *Need to efficiently extract email adresses from a document? Hashtags from tweets? Remove accents from a French post?* :disappointed_relieved:

**NLPretext got you covered!** :rocket:

NLPretext packages in a **unique** library all the text **preprocessing** functions you need to **ease** your NLP project.

:mag: Quickly explore below our preprocessing pipelines and individual functions referential.

* [Default preprocessing pipeline](#default_pipeline)
* [Custom preprocessing pipeline](#custom_pipeline)
* [Replacing phone numbers](#replace_phone_numbers)
* [Removing hashtags](#remove_hashtags)
* [Extracting emojis](#extract_emojis)
* [Data augmentation](#data_augmentation)

Cannot find what you were looking for? Feel free to open an [issue]((https://github.com/artefactory/nlpretext/issues) ).

# Installation

### Supported Python Versions

- Main version supported : `3.8`
- Other supported versions : `3.9`, `3.10`

We strongly advise you to do the remaining steps in a virtual environnement.

To install this library from PyPi, run the following command:

```bash
pip install nlpretext
```

or with `Poetry`

```bash
poetry add nlpretext
```

# Usage

## Default pipeline

Need to preprocess your text data but no clue about what function to use and in which order? The default preprocessing pipeline got you covered:

```python
from nlpretext import Preprocessor
text = "I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris \n"
preprocessor = Preprocessor()
text = preprocessor.run(text)
print(text)
# "I just got the best dinner in my life!!! I recommend"
```

## Create your custom pipeline

Another possibility is to create your custom pipeline if you know exactly what function to apply on your data, here's an example:

```python
from nlpretext import Preprocessor
from nlpretext.basic.preprocess import (normalize_whitespace, remove_punct, remove_eol_characters,
remove_stopwords, lower_text)
from nlpretext.social.preprocess import remove_mentions, remove_hashtag, remove_emoji
text = "I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris \n"
preprocessor = Preprocessor()
preprocessor.pipe(lower_text)
preprocessor.pipe(remove_mentions)
preprocessor.pipe(remove_hashtag)
preprocessor.pipe(remove_emoji)
preprocessor.pipe(remove_eol_characters)
preprocessor.pipe(remove_stopwords, args={'lang': 'en'})
preprocessor.pipe(remove_punct)
preprocessor.pipe(normalize_whitespace)
text = preprocessor.run(text)
print(text)
# "dinner life recommend"
```

Take a look at all the functions that are available [here](https://github.com/artefactory/NLPretext/tree/master/nlpretext) in the ```preprocess.py``` scripts in the different folders: basic, social, token.

## Load text data

Pre-processing text data is useful only if you have loaded data to process! Importing text data as strings in your code can be really simple if you have short texts contained in a local .txt, but it can quickly become difficult if you want to load a lot of texts, stored in multiple formats and divided in multiple files. Hopefully, you can use NLPretext's TextLoader class to easily import text data.
while it is not mandatory our textLoader work best with dask, make sure to have the librairy installed if you want the best performances.

```python
from nlpretext.textloader import TextLoader
files_path = "local_folder/texts/text.txt"
text_loader = TextLoader(use_dask=True)
text_dataframe = text_loader.read_text(files_path)
print(text_dataframe.text.values.tolist())
# ["I just got the best dinner in my life!!!",  "I recommend", "It was awesome"]
```

File path can be provided as string, list of strings, with or without wildcards. It also supports imports from cloud providers, if your machine is authentified on a project.

```python
text_loader = TextLoader(text_column="name_of_text_column_in_your_data")

local_file_path = "local_folder/texts/text.csv" # File from local folder
local_corpus_path = ["local_folder/texts/text_1.csv", "local_folder/texts/text_2.csv", "local_folder/texts/text_3.csv"] # Multiple files from local folder

gcs_file_path = "gs://my-bucket/texts/text.json" # File from GCS
s3_file_path = "s3://my-bucket/texts/text.json" # File from S3
hdfs_file_path = "hdfs://folder/texts/text.txt" # File from HDFS
azure_file_path = "az://my-bucket/texts/text.parquet" # File from Azure

gcs_corpus_path = "gs://my-bucket/texts/text_*.json" # Multiple files from GCS with wildcard

text_dataframe_1 = text_loader.read_text(local_file_path)
text_dataframe_2 = text_loader.read_text(local_corpus_path)
text_dataframe_3 = text_loader.read_text(gcs_file_path)
text_dataframe_4 = text_loader.read_text(s3_file_path)
text_dataframe_5 = text_loader.read_text(hdfs_file_path)
text_dataframe_6 = text_loader.read_text(azure_file_path)
text_dataframe_7 = text_loader.read_text(gcs_corpus_path)

```

You can also specify a Preprocessor if you want your data to be directly pre-processed when loaded.
```python
text_loader = TextLoader(text_column="text_col")
preprocessor = Preprocessor()

file_path = "local_folder/texts/text.csv" # File from local folder

raw_text_dataframe = text_loader.read_text(local_file_path)
preprocessed_text_dataframe = text_loader.read_text(local_file_path, preprocessor=preprocessor)

print(raw_text_dataframe.text_col.values.tolist())
# ["These   texts are not preprocessed",  "This is bad ## "]

print(preprocessed_text_dataframe.text_col.values.tolist())
# ["These texts are not preprocessed",  "This is bad"]
```

## Individual Functions

### Replacing emails

```python
from nlpretext.basic.preprocess import replace_emails
example = "I have forwarded this email to obama@whitehouse.gov"
example = replace_emails(example, replace_with="*EMAIL*")
print(example)
# "I have forwarded this email to *EMAIL*"
```

### Replacing phone numbers

```python
from nlpretext.basic.preprocess import replace_phone_numbers
example = "My phone number is 0606060606"
example = replace_phone_numbers(example, country_to_detect=["FR"], replace_with="*PHONE*")
print(example)
# "My phone number is *PHONE*"
```

### Removing Hashtags

```python
from nlpretext.social.preprocess import remove_hashtag
example = "This restaurant was amazing #food #foodie #foodstagram #dinner"
example = remove_hashtag(example)
print(example)
# "This restaurant was amazing"
```

### Extracting emojis

```python
from nlpretext.social.preprocess import extract_emojis
example = "I take care of my skin 😀"
example = extract_emojis(example)
print(example)
# [':grinning_face:']
```

## Data augmentation

The augmentation module helps you to **generate new texts** based on your given examples by modifying some words in the initial ones and to **keep associated entities unchanged**, if any, in the case of **NER tasks**. If you want words other than entities to remain unchanged, you can specify it within the `stopwords` argument. Modifications depend on the chosen method, the ones currently supported by the module are **substitutions with synonyms** using Wordnet or BERT from the [`nlpaug`](https://github.com/makcedward/nlpaug) library.

```python
from nlpretext.augmentation.text_augmentation import augment_text
example = "I want to buy a small black handbag please."
entities = [{'entity': 'Color', 'word': 'black', 'startCharIndex': 22, 'endCharIndex': 27}]
example = augment_text(example, method=”wordnet_synonym”, entities=entities)
print(example)
# "I need to buy a small black pocketbook please."
```

# 📈 Releases

You can see the list of available releases on the [GitHub Releases](https://github.com/artefactory/NLPretext}/releases) page.

We follow [Semantic Versions](https://semver.org/) specification.

We use [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.

For Pull Requests, these labels are configured, by default:

|               **Label**               |  **Title in Releases**  |
| :-----------------------------------: | :---------------------: |
|       `enhancement`, `feature`        |       🚀 Features       |
| `bug`, `refactoring`, `bugfix`, `fix` | 🔧 Fixes & Refactoring  |
|       `build`, `ci`, `testing`        | 📦 Build System & CI/CD |
|              `breaking`               |   💥 Breaking Changes   |
|            `documentation`            |    📝 Documentation     |
|            `dependencies`             | ⬆ Dependencies updates |

GitHub creates the `bug`, `enhancement`, and `documentation` labels automatically. Dependabot creates the `dependencies` label. Create the remaining labels on the Issues tab of the GitHub repository, when needed.## 🛡 License

[![License](https://img.shields.io/github/license/artefactory/NLPretext)](https://github.com/artefactory/NLPretext}/blob/main/LICENSE)

This project is licensed under the terms of the `Apache Software License 2.0` license. See [LICENSE](https://github.com/artefactory/NLPretext}/blob/main/LICENSE) for more details.## 📃 Citation

```
@misc{nlpretext,
  author = {artefactory},
  title = {All the goto functions you need to handle NLP use-cases, integrated in NLPretext},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/artefactory/NLPretext}}}
}
```

# Project Organization
------------

    .
    ├── .github/workflows           <- Where the CI and CD lives
    ├── datasets/external           <- Bash scripts to download external datasets
    ├── docker                      <- All you need to build a Docker image from that package
    ├── docs                        <- Sphinx HTML documentation
    ├── nlpretext                   <- Main Package. This is where the code lives
    │   ├── preprocessor.py         <- Main preprocessing script
    │   ├── text_loader.py          <- Main loading script
    │   ├── augmentation            <- Text augmentation script
    │   ├── basic                   <- Basic text preprocessing
    │   ├── cli                     <- Command lines that can be used
    │   ├── social                  <- Social text preprocessing
    │   ├── token                   <- Token text preprocessing
    │   ├── textloader              <- File loading
    │   ├── _config                 <- Where the configuration and constants live
    │   └── _utils                  <- Where preprocessing utils scripts lives
    ├── references                  <- assets
    ├── tests                       <- Where the tests lives
    ├── .gitignore
    ├── .pre-commit-config.yaml     <- Pre-commit configuration
    ├── CODE_OF_CONDUCT.md          <- Code of conduct guidelines
    ├── CONTRIBUTING.md             <- Contribution guidelines
    ├── LICENSE
    ├── Makefile
    ├── pyproject.toml              <- Package build configuration
    ├── README.md                   <- The top-level README for developers using this project.
    └── SECURITY.md

# Credits

- [textacy](https://github.com/chartbeat-labs/textacy) for the following basic preprocessing functions:
    - `fix_bad_unicode`
    - `normalize_whitespace`
    - `unpack_english_contractions`
    - `replace_urls`
    - `replace_emails`
    - `replace_numbers`
    - `replace_currency_symbols`
    - `remove_punct`
    - `remove_accents`
    - `replace_phone_numbers` *(with some modifications of our own)*