[![CI](https://img.shields.io/github/actions/workflow/status/emdgroup/baybe/ci.yml?style=flat-square&label=CI&labelColor=0f69af&color=ffdcb9)](https://github.com/emdgroup/baybe/actions/workflows/ci.yml)
[![Regular](https://img.shields.io/github/actions/workflow/status/emdgroup/baybe/regular.yml?style=flat-square&label=Regular%20Check&labelColor=0f69af&color=ffdcb9)](https://github.com/emdgroup/baybe/actions/workflows/regular.yml)
[![Docs](https://img.shields.io/github/actions/workflow/status/emdgroup/baybe/docs.yml?style=flat-square&label=Docs&labelColor=0f69af&color=ffdcb9)](https://github.com/emdgroup/baybe/actions/workflows/docs.yml)

[![Supports Python](https://img.shields.io/pypi/pyversions/baybe?style=flat-square&label=Supports%20Python&labelColor=96d7d2&color=ffdcb9)](https://pypi.org/project/baybe/)
[![PyPI version](https://img.shields.io/pypi/v/baybe.svg?style=flat-square&label=PyPI%20Version&labelColor=96d7d2&color=ffdcb9)](https://pypi.org/project/baybe/)
[![Issues](https://img.shields.io/github/issues/emdgroup/baybe?style=flat-square&label=Issues&labelColor=96d7d2&color=ffdcb9)](https://github.com/emdgroup/baybe/issues/)
[![PRs](https://img.shields.io/github/issues-pr/emdgroup/baybe?style=flat-square&label=PRs&labelColor=96d7d2&color=ffdcb9)](https://github.com/emdgroup/baybe/pulls/)
[![License](https://shields.io/badge/License-Apache%202.0-green.svg?style=flat-square&labelColor=96d7d2&color=ffdcb9)](http://www.apache.org/licenses/LICENSE-2.0)

[![Logo](https://raw.githubusercontent.com/emdgroup/baybe/main/docs/_static/banner2.svg)](https://github.com/emdgroup/baybe/)

Homepage
 •
User Guide
 •
Documentation
 •
Contribute

# BayBE — A Bayesian Back End for Design of Experiments

The Bayesian Back End (**BayBE**) is a general-purpose toolbox for Bayesian Design
of Experiments, focusing on additions that enable real-world experimental campaigns.

Besides functionality to perform a typical recommend-measure loop, BayBE's highlights are:
- ✨ Custom parameter encodings: Improve your campaign with domain knowledge
- 🧪 Built-in chemical encodings: Improve your campaign with chemical knowledge
- 🎯 Single and multiple targets with min, max and match objectives
- ⚙ Custom surrogate models: For specialized problems or active learning
- 🎭 Hybrid (mixed continuous and discrete) spaces
- 🚀 Transfer learning: Mix data from multiple campaigns and accelerate optimization
- 📈 Comprehensive backtest, simulation and imputation utilities: Benchmark and find your best settings
- 📝 Fully typed and hypothesis-tested: Robust code base
- 🔄 All objects are fully de-/serializable: Useful for storing results in databases or use in wrappers like APIs

## ⚡ Quick Start

Let us consider a simple experiment where we control three parameters and want to
maximize a single target called `Yield`.

First, install BayBE into your Python environment:
```bash
pip install baybe
```
For more information on this step, see our
[detailed installation instructions](#installation).

### Defining the Optimization Objective

In BayBE's language, the `Yield` can be represented as a `NumericalTarget`,
which we wrap into a `SingleTargetObjective`:

```python
from baybe.targets import NumericalTarget
from baybe.objectives import SingleTargetObjective

target = NumericalTarget(
    name="Yield",
    mode="MAX",
)
objective = SingleTargetObjective(target=target)
```
In cases where we are confronted with multiple (potentially conflicting) targets,
the `DesirabilityObjective` can be used instead. It allows to define additional
settings, such as how these targets should be balanced.
For more details, see the
[objectives section](https://emdgroup.github.io/baybe/userguide/objectives.html)
of the user guide.

### Defining the Search Space

Next, we inform BayBE about the available "control knobs", that is, the underlying
system parameters we can tune to optimize our targets. This also involves specifying
their values/ranges and other parameter-specific details.

For our example, we assume that we can control three parameters – `Granularity`,
`Pressure[bar]`, and `Solvent` – as follows:

```python
from baybe.parameters import (
    CategoricalParameter,
    NumericalDiscreteParameter,
    SubstanceParameter,
)

parameters = [
    CategoricalParameter(
        name="Granularity",
        values=["coarse", "medium", "fine"],
        encoding="OHE",  # one-hot encoding of categories
    ),
    NumericalDiscreteParameter(
        name="Pressure[bar]",
        values=[1, 5, 10],
        tolerance=0.2,  # allows experimental inaccuracies up to 0.2 when reading values
    ),
    SubstanceParameter(
        name="Solvent",
        data={
            "Solvent A": "COC",
            "Solvent B": "CCC",  # label-SMILES pairs
            "Solvent C": "O",
            "Solvent D": "CS(=O)C",
        },
        encoding="MORDRED",  # chemical encoding via mordred package
    ),
]
```

For more parameter types and their details, see the
[parameters section](https://emdgroup.github.io/baybe/userguide/parameters.html)
of the user guide.

Additionally, we can define a set of constraints to further specify allowed ranges and
relationships between our parameters. Details can be found in the
[constraints section](https://emdgroup.github.io/baybe/userguide/constraints.html) of the user guide.
In this example, we assume no further constraints.

With the parameter definitions at hand, we can now create our
`SearchSpace` based on the Cartesian product of all possible parameter values:

```python
from baybe.searchspace import SearchSpace

searchspace = SearchSpace.from_product(parameters)
```

See the [search spaces section](https://emdgroup.github.io/baybe/userguide/searchspace.html)
of our user guide for more information on the structure of search spaces
and alternative ways of construction.

### Optional: Defining the Optimization Strategy

As an optional step, we can specify details on how the optimization should be
conducted. If omitted, BayBE will choose a default setting.

For our example, we combine two recommenders via a so-called meta recommender named
`TwoPhaseMetaRecommender`:

1. In cases where no measurements have been made prior to the interaction with BayBE,
   a selection via `initial_recommender` is used.
2. As soon as the first measurements are available, we switch to `recommender`.

For more details on the different recommenders, their underlying algorithmic
details, and their configuration settings, see the
[recommenders section](https://emdgroup.github.io/baybe/userguide/recommenders.html)
of the user guide.

```python
from baybe.recommenders import (
    BotorchRecommender,
    FPSRecommender,
    TwoPhaseMetaRecommender,
)

recommender = TwoPhaseMetaRecommender(
    initial_recommender=FPSRecommender(),  # farthest point sampling
    recommender=BotorchRecommender(),  # Bayesian model-based optimization
)
```

### The Optimization Loop

We can now construct a campaign object that brings all pieces of the puzzle together:

```python
from baybe import Campaign

campaign = Campaign(searchspace, objective, recommender)
```

With this object at hand, we can start our experimentation cycle.
In particular:

* We can ask BayBE to `recommend` new experiments.
* We can `add_measurements` for certain experimental settings to the campaign's
  database.

Note that these two steps can be performed in any order.
In particular, available measurements can be submitted at any time and also several
times before querying the next recommendations.

```python
df = campaign.recommend(batch_size=3)
print(df)
```

```none
   Granularity  Pressure[bar]    Solvent
15      medium            1.0  Solvent D
10      coarse           10.0  Solvent C
29        fine            5.0  Solvent B
```

Note that the specific recommendations will depend on both the data
already fed to the campaign and the random number generator seed that is used.

After having conducted the corresponding experiments, we can add our measured
targets to the table and feed it back to the campaign:

```python
df["Yield"] = [79.8, 54.1, 59.4]
campaign.add_measurements(df)
```

With the newly arrived data, BayBE can produce a refined design for the next iteration.
This loop would typically continue until a desired target value has been achieved in
the experiment.

### Advanced Example: Chemical Substances
BayBE has several modules to go beyond traditional approaches. One such example is the
use of custom encodings for categorical parameters. Chemical encodings for substances
are a special built-in case of this that comes with BayBE.

In the following picture you can see
the outcome for treating the solvent, base and ligand in a direct arylation reaction
optimization (from [Shields, B.J. et al.](https://doi.org/10.1038/s41586-021-03213-y)) with
chemical encodings compared to one-hot and a random baseline:
![Substance Encoding Example](./examples/Backtesting/full_lookup_light.svg)

(installation)=
## 💻 Installation
### From Package Index
The easiest way to install BayBE is via PyPI:

```bash
pip install baybe
```

A certain released version of the package can be installed by specifying the
corresponding version tag in the form `baybe==x.y.z`.

### From GitHub
If you need finer control and would like to install a specific commit that has not been
released under a certain version tag, you can do so by installing BayBE directly from
GitHub via git and specifying the corresponding
[git ref](https://pip.pypa.io/en/stable/topics/vcs-support/#git).

For instance, to install the latest commit of the main branch, run:

```bash
pip install git+https://github.com/emdgroup/baybe.git@main
```

### From Local Clone

Alternatively, you can install the package from your own local copy.
First, clone the repository, navigate to the repository root folder, check out the
desired commit, and run:

```bash
pip install .
```

A developer would typically also install the package in editable mode ('-e'),
which ensures that changes to the code do not require a reinstallation.

```bash
pip install -e .
```

If you need to add additional dependencies, make sure to use the correct syntax
including `''`:

```bash
pip install -e '.[dev]'
```

### Optional Dependencies
There are several dependency groups that can be selected during pip installation, like
```bash
pip install 'baybe[test,lint]' # will install baybe with additional dependency groups `test` and `lint`
```
To get the most out of `baybe`, we recommend to install at least
```bash
pip install 'baybe[chem,simulation]'
```

The available groups are:
- `chem`: Cheminformatics utilities (e.g. for the `SubstanceParameter`).
- `docs`: Required for creating the documentation.
- `examples`: Required for running the examples/streamlit.
- `lint`: Required for linting and formatting.
- `mypy`: Required for static type checking.
- `onnx`: Required for using custom surrogate models in [ONNX format](https://onnx.ai).
- `simulation`: Enabling the [simulation](https://emdgroup.github.io/baybe/_autosummary/baybe.simulation.html) module.
- `test`: Required for running the tests.
- `dev`: All of the above plus `tox` and `pip-audit`. For code contributors.

## 📡 Telemetry
BayBE collects anonymous usage statistics **only** for employees of Merck KGaA,
Darmstadt, Germany and/or its affiliates. The recording of metrics is turned off for
all other users and is impossible due to a VPN block. In any case, the usage statistics
do **not** involve logging of recorded measurements, targets/parameters or their names
or any project information that would allow for reconstruction of details. The user and
host machine names are anonymized with via truncated hashing.
- You can verify the above statements by studying the open-source code in the
  `telemetry` module.
- You can always deactivate all telemetry by setting the environment variable
  `BAYBE_TELEMETRY_ENABLED` to `false` or `off`. For details please consult
  [this page](https://emdgroup.github.io/baybe/userguide/envvars.html#telemetry).
- If you want to be absolutely sure, you can uninstall internet related packages such
  as `opentelemetry*` or its secondary dependencies from the environment. Due to the
  inability of specifying opt-out dependencies, these are installed by default, but the
  package works without them.

## 👨🏻🔧 Maintainers

- Martin Fitzner (Merck KGaA, Darmstadt, Germany), [Contact](mailto:martin.fitzner@merckgroup.com), [Github](https://github.com/Scienfitz)
- Adrian Šošić (Merck Life Science KGaA, Darmstadt, Germany), [Contact](mailto:adrian.sosic@merckgroup.com), [Github](https://github.com/AdrianSosic)
- Alexander Hopp (Merck KGaA, Darmstadt, Germany) [Contact](mailto:alexander.hopp@merckgroup.com), [Github](https://github.com/AVHopp)

## 🛠 Known Issues
A list of know issues can be found [here](https://emdgroup.github.io/baybe/known_issues.html).

## 📄 License

Copyright 2022-2024 Merck KGaA, Darmstadt, Germany
and/or its affiliates. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


    Note: this is the v3 branch, which is currently in beta. See the docs for v3. If needed the 2.x branch is here, but is in bugfix only mode.

    pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands. Think ansible but Python instead of YAML, and a lot faster.

---

    Getting Started •
    Examples Repo •
    Chat on Matrix

    Documentation •
    Help & Support •
    Contributing

---

Why pyinfra? Design features include:

+ 🚀 **Super fast** execution over thousands of hosts with predictable performance.
+ 🚨 **Instant debugging** with realtime stdin/stdout/stderr output (`-vvv`).
+ 🔄 **Idempotent operations** that enable diffs and dry runs before making changes.
+ 📦 **Extendable** with the entire Python package ecosystem.
+ 💻 **Agentless execution** against anything with shell access.
+ 🔌 **Integrated** with connectors for Docker, Terraform, Vagrant and more.

## Quickstart

Install pyinfra with `pip`:

```
pip install pyinfra
```

Now you can execute commands on hosts via SSH:

```sh
pyinfra my-server.net exec -- echo "hello world"
```

Or target Docker containers, the local machine, and other [connectors](https://docs.pyinfra.com/page/connectors.html):

```sh
pyinfra @docker/ubuntu exec -- echo "Hello world"
pyinfra @local exec -- echo "Hello world"
```

As well as executing commands you can define state using [operations](https://docs.pyinfra.com/page/operations.html):

```sh
# Install iftop apt package if not present
pyinfra @docker/ubuntu apt.packages iftop update=true _sudo=true
```

Which can then be saved as a Python file like `deploy.py`:

```py
from pyinfra.operations import apt

apt.packages(
    name="Ensure iftop is installed",
    packages=['iftop'],
    update=True,
    _sudo=True,
)
```

The hosts can also be saved in a file, for example `inventory.py`:

```py
targets = ["@docker/ubuntu", "my-test-server.net"]
```

And executed together:

```sh
pyinfra inventory.py deploy.py
```

Now you know the building blocks of pyinfra! By combining inventory, operations and Python code you can deploy anything.

See the more detailed [getting started](https://docs.pyinfra.com/page/getting-started.html) or [using operations](https://docs.pyinfra.com/page/using-operations.html) guides. See how to use [inventory & data](https://docs.pyinfra.com/page/inventory-data.html), [global arguments](https://docs.pyinfra.com/page/arguments.html) and [the CLI](https://docs.pyinfra.com/page/cli.html) or check out the [documented examples](https://docs.pyinfra.com/page/examples.html).

---


# FESTIM
[![NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org/)
[![CI](https://github.com/festim-dev/FESTIM/actions/workflows/ci.yml/badge.svg)](https://github.com/festim-dev/FESTIM/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/festim-dev/FESTIM/branch/master/graph/badge.svg?token=AK3A9CV2D3)](https://codecov.io/gh/festim-dev/FESTIM)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square)](https://github.com/psf/black)
![GitHub stars](https://img.shields.io/github/stars/festim-dev/FESTIM.svg?logo=github&label=Stars&logoColor=white)
![GitHub forks](https://img.shields.io/github/forks/festim-dev/FESTIM.svg?logo=github&label=Forks&logoColor=white)

FESTIM (Finite Elements Simulation of Tritium in Materials) is a tool for modeling hydrogen transport in materials.
It simulates the diffusion and trapping of hydrogen, coupled to heat transfer with [FEniCS](https://fenicsproject.org).

## Getting started

The installation instructions are available [here](https://festim.readthedocs.io/en/latest/installation.html).

## Documentation

Documentation can be viewed at https://festim.readthedocs.io/

## Tutorials

Visit the FESTIM-workshop to view the available tutorials https://github.com/festim-dev/FESTIM-workshop

## Contact

For questions on how to use FESTIM, visit our [Discourse page](https://festim.discourse.group/)
or join our [Slack](https://join.slack.com/t/festimworkspace/shared_invite/zt-246hw8d6o-htWASLsbdosUo_2nRKCf9g) channel

## Contributing

The guidelines for contributions can be found in the documentation: https://festim.readthedocs.io/en/latest/devguide/index.html

## Roadmap

You can follow the public roadmap on the [FESTIM projects page](https://github.com/festim-dev/FESTIM/projects?query=is%3Aopen).
Moreover, you can check out the development branch (with support of FEniCSx) on the [_fenicsx_](https://github.com/festim-dev/FESTIM/tree/fenicsx) branch.


# Standalone previewer for VapourSynth scripts

Fork of Endilll's [vapoursynth-preview](https://github.com/Endilll/vapoursynth-preview) (not maintained anymore)

This program is meant to be paired with a code editor with integrated terminal like Visual Studio Code.

# Prerequisites

1. [Python](https://www.Python.org/downloads) (3.12+ required)
   - Make sure to install Python to your `PATH`.
1. [VapourSynth](https://github.com/vapoursynth/vapoursynth/releases) (R65+ required)

# Installation

Install latest stable via pypi:

```bash
pip install vspreview
```

Install latest git:

```bash
pip install -U git+https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview.git
```

# Usage

It can be used by running `vspreview script.vpy` or your preferred way in [your IDE](#ide-integration).

[Keyboard Shortcuts](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview/blob/master/docs/accessibility/keybinds.rst)

[Saved Frame Filename Variables](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview/tree/master/docs/save_frame_placeholders.md)

# IDE Integration

- [Visual Studio Code](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview/tree/master/docs/installation/install_vscode.rst)
- [Vim](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview/tree/master/docs/installation/install_vim.rst)
- [Notepad++](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview/tree/master/docs/installation/install_notepad++.rst)

# Plugins

You can install external plugins by using the following command:

```bash
vspreview install [plugin-name]
```

A list of plugins can be found in [this repo](https://github.com/Jaded-Encoding-Thaumaturgy/vs-preview-plugins). `plugin-name` is the name of the plugin directory in the repo.

To develop new plugins or manually install them, create a .ppy file inside the global plugins directory. You can find the path to that in the storage file. You can add additional search paths by adding them to a .pth file inside the global plugins directory, just like you can with python path and site-packages.


# SO TOD Library

![Build and Test](https://github.com/simonsobs/sotodlib/workflows/Build%20and%20Test/badge.svg) [![Documentation Status](https://readthedocs.org/projects/sotodlib/badge/?version=latest)](https://sotodlib.readthedocs.io/en/latest/?badge=latest)

Tools for working with real and simulated time ordered data.

Full documentation here:

https://sotodlib.readthedocs.io/en/latest/


[![Netlify Status](https://api.netlify.com/api/v1/badges/733ffbee-a738-432b-963c-5ef65ff16302/deploy-status)](https://app.netlify.com/sites/educates-docs/deploys)

Educates Training Platform
==========================

The Educates project provides a system for hosting interactive workshop
environments in Kubernetes, or on top of a local container runtime. It can be
used for self paced or supervised workshops. It can also be useful where you
need to package up demos of applications hosted in Kubernetes or a local
container runtime.

Educates documentation
----------------------

For detailed instructions on how to deploy and make use of Educates see the
[Educates user documentation](https://docs.educates.dev/).

Getting help with Educates
--------------------------

If you have questions about using Educates, use the `#educates` channel under
the [Kubernetes community Slack](https://kubernetes.slack.com/).

If you have found a bug in Educates or want to request a feature, you can use
our [GitHub issue
tracker](https://github.com/vmware-tanzu-labs/educates-training-platform/issues).

Contributing to Educates
------------------------

If you would like to contribute to Educates, check out our [contribution
guidelines](CONTRIBUTING.md) and [developer
documenation](developer-docs/README.md).


[![Contributors][contributors-shield]][contributors-url]
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![MIT License][license-shield]][license-url]
[![LinkedIn][linkedin-shield]][linkedin-url]

PhysioLabXR

    A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality Experiments

    Explore the docs »

    Official website
    ·
    Report Bug
    ·
    Request Feature

  Table of Contents

      About The Project

Built With

      Getting Started

Prerequisites

Installation

Usage

Roadmap

Contributing

License

Contact

Acknowledgments

## About The Project

[![PhysioLabXR Screen Shot][product-screenshot]](physiolabxr/_media/readme/AllPlottingFormat.png)

**PhysioLabXR** is a Python-based App for visualizing, recording, and processing (i.e., make prediction)
data streams. PhysioLabXR can help you build novel interaction interface like BCIs as well as aid you in
running experiments. It works best with multi-modal (e.g., combining EEG and eyetracking, camera with speech), high-throughput (~500Mbps/sec), real-time data streams.

[//]: # (Here's a blank template to get started: To avoid retyping too much info. Do a search and replace with your text editor for the following: `github_username`, `repo_name`, `twitter_handle`, `linkedin_username`, `email_client`, `email`, `project_title`, `project_description`)

[//]: # ()
[//]: # (
(back to top)
)

Check out the paper on [Journal of Open Source Software](https://doi.org/10.21105/joss.05854): [![DOI](https://joss.theoj.org/papers/10.21105/joss.05854/status.svg)](https://doi.org/10.21105/joss.05854). Cite:

```
@article{Li2024, doi = {10.21105/joss.05854}, url = {https://doi.org/10.21105/joss.05854}, year = {2024}, publisher = {The Open Journal}, volume = {9}, number = {93}, pages = {5854}, author = {Ziheng 'Leo' Li and Haowen 'John' Wei and Ziwen Xie and Yunxiang Peng and June Pyo Suh and Steven Feiner and Paul Sajda}, title = {PhysioLabXR: A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality Experiments}, journal = {Journal of Open Source Software} }
```

### Built With

* [![Python][Python.org]][Python-url]
* [![QT][QT.io]][QT-url]
* [![PyQtGraph][pyqtgraph.org]][pyqtgraph-url]
* [![NumPy][numpy.org]][numpy-url]
* [![ZMQ][zeromq.org]][zeromq-url]
* [![LSL][LSL.org]][LSL-url]

(back to top)

## Getting Started

### Download and run the executable

Download the latest release exe from [here](https://github.com/PhysioLabXR/PhysioLabXR/releases), available for Windows, Mac, and Linux.

### Install with pip

PhysioLabXR currently supports Python 3.9, 3.10, and 3.11. Support for Python 3.12 is coming soon.

Install PhysioLabXR's [PYPI distribution](https://pypi.org/project/physiolabxr/) with

  ```sh
    pip install physiolabxr
  ```
Then run with

  ```sh
    physiolabxr
  ```

### Run from Source

Alternatively, you can clone the repo and run from source.

  ```sh
    git clone https://github.com/PhysioLabXR/PhysioLabXR.git
    cd PhysioLabXR
    pip install -r requirements.txt
  ```

The entry point to PhysioLabXR is `physiolabxr.py`, located in the folder named "physiolabxr". From the root folder, you can run it by:

  ```sh
    python physiolabxr/physiolabxr.py
  ```

(back to top)

## Usage

_For more examples, please refer to the tutorials in the [documentation](https://physiolabxrdocs.readthedocs.io/en/latest/index.html)_

Tutorials have examples for:

* Real-time fixation detection ([link](https://physiolabxrdocs.readthedocs.io/en/latest/FixationDetection.html))
* Real-time multi-modal event-related-potential classification with EEG and pupillometry
* P300 speller in Unity
* Stroop task with PsychoPy (link [https://physiolabxrdocs.readthedocs.io/en/latest/PsychoPy.html])

More are coming soon!

(back to top)

## License

Distributed under the 3-Clause BSD License. See `LICENSE.txt` for more information.

© 2023 The Trustees of Columbia University in the City of New York.  This work may be reproduced, distributed, and otherwise exploited for academic non-commercial purposes only.  To obtain a license to this work for commercial purposes, please contact Columbia Technology Ventures at techventures@columbia.edu.

(back to top)

## Contact

[Ziheng 'Leo' Li](https://www.linkedin.com/in/ziheng-leo-li/) - zl2990@columbia.edu

(back to top)

## Acknowledgments

We would like to express our gratitude for the support from our colleagues at Columbia University and Worcester Polytechnic Institute.
We would also like to thank all the community members who have contributed to *PhysioLabXR*.

[//]: # (* []())

[//]: # (* []())

(back to top)

[contributors-shield]: https://img.shields.io/github/contributors/apocalyvec/renalabapp.svg?style=for-the-badge
[contributors-url]: https://github.com/apocalyvec/renalabapp/graphs/contributors
[forks-shield]: https://img.shields.io/github/forks/apocalyvec/renalabapp.svg?style=for-the-badge
[forks-url]: https://github.com/apocalyvec/renalabapp/network/members
[stars-shield]: https://img.shields.io/github/stars/apocalyvec/renalabapp.svg?style=for-the-badge
[stars-url]: https://github.com/apocalyvec/renalabapp/stargazers
[issues-shield]: https://img.shields.io/github/issues/apocalyvec/renalabapp.svg?style=for-the-badge
[issues-url]: https://github.com/apocalyvec/renalabapp/issues
[license-shield]: https://img.shields.io/github/license/apocalyvec/renalabapp.svg?style=for-the-badge
[license-url]: https://github.com/apocalyvec/renalabapp/blob/master/LICENSE.txt
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
[linkedin-url]: https://www.linkedin.com/in/ziheng-leo-li/
[product-screenshot]: physiolabxr/_media/readme/AllPlottingFormat.png

[Python.org]: https://img.shields.io/badge/Python-000000?style=for-the-badge&logo=python&color=3776AB&logoColor=white
[Python-url]: https://python.org/

[QT.io]: https://img.shields.io/badge/QT-000000?style=for-the-badge&logo=qt
[QT-url]: https://qt.io/

[zeromq.org]: https://img.shields.io/badge/zeromq-000000?style=for-the-badge&logo=zeromq&color=DF0000
[zeromq-url]: https://qt.io/

[lsl.org]: https://img.shields.io/badge/lsl-000000?style=for-the-badge&color=lightgrey
[lsl-url]: https://labstreaminglayer.org

[pyqtgraph.org]: https://img.shields.io/badge/pyqtgraph-000000?style=for-the-badge&color=bbbbff
[pyqtgraph-url]: https://www.pyqtgraph.org/

[numpy.org]: https://img.shields.io/badge/numpy-000000?style=for-the-badge&logo=numpy&color=013243
[numpy-url]: https://www.numpy.org/

[Next-url]: https://python.org/
[React.js]: https://img.shields.io/badge/React-20232A?style=for-the-badge&logo=react&logoColor=61DAFB
[React-url]: https://reactjs.org/
[Vue.js]: https://img.shields.io/badge/Vue.js-35495E?style=for-the-badge&logo=vuedotjs&logoColor=4FC08D
[Vue-url]: https://vuejs.org/
[Angular.io]: https://img.shields.io/badge/Angular-DD0031?style=for-the-badge&logo=angular&logoColor=white
[Angular-url]: https://angular.io/
[Svelte.dev]: https://img.shields.io/badge/Svelte-4A4A55?style=for-the-badge&logo=svelte&logoColor=FF3E00
[Svelte-url]: https://svelte.dev/
[Laravel.com]: https://img.shields.io/badge/Laravel-FF2D20?style=for-the-badge&logo=laravel&logoColor=white
[Laravel-url]: https://laravel.com
[Bootstrap.com]: https://img.shields.io/badge/Bootstrap-563D7C?style=for-the-badge&logo=bootstrap&logoColor=white
[Bootstrap-url]: https://getbootstrap.com
[JQuery.com]: https://img.shields.io/badge/jQuery-0769AD?style=for-the-badge&logo=jquery&logoColor=white
[JQuery-url]: https://jquery.com


# aind-data-schema

[![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
![Code Style](https://img.shields.io/badge/code%20style-black-black)
[![Documentation Status](https://readthedocs.org/projects/aind-data-schema/badge/?version=latest)](https://aind-data-schema.readthedocs.io/en/latest/?badge=latest)

A library that defines [AIND](https://alleninstitute.org/what-we-do/brain-science/research/allen-institute-neural-dynamics/) data schema and validates JSON files.

User documentation available on [readthedocs](https://aind-data-schema.readthedocs.io/en/latest/).

## Overview

This repository contains the schemas needed to ingest and validate metadata that are essential to ensuring [AIND](https://alleninstitute.org/what-we-do/brain-science/research/allen-institute-neural-dynamics/) data collection is completely reproducible. Our general approach is to semantically version core schema classes and include those version numbers in serialized metadata so that we can flexibly evolve the schemas over time without requiring difficult data migrations. In the future, we will provide a browsable list of these classes rendered to [JSON Schema](https://json-schema.org/), including all historic versions.

Be aware that this package is still under heavy preliminary development. Expect breaking changes regularly, although we will communicate these through semantic versioning.

A simple example:

```python
import datetime

from aind_data_schema.core.subject import BreedingInfo, Housing, Subject
from aind_data_schema_models.organizations import Organization
from aind_data_schema_models.species import Species

t = datetime.datetime(2022, 11, 22, 8, 43, 00)

s = Subject(
   species=Species.MUS_MUSCULUS,
   subject_id="12345",
   sex="Male",
   date_of_birth=t.date(),
   genotype="Emx1-IRES-Cre;Camk2a-tTA;Ai93(TITL-GCaMP6f)",
   housing=Housing(home_cage_enrichment=["Running wheel"], cage_id="123"),
   background_strain="C57BL/6J",
   source=Organization.AI,
   breeding_info=BreedingInfo(
         breeding_group="Emx1-IRES-Cre(ND)",
         maternal_id="546543",
         maternal_genotype="Emx1-IRES-Cre/wt; Camk2a-tTa/Camk2a-tTA",
         paternal_id="232323",
         paternal_genotype="Ai93(TITL-GCaMP6f)/wt",
   ),
)

s.write_standard_file() # writes subject.json
```

```json
{
   "describedBy": "https://raw.githubusercontent.com/AllenNeuralDynamics/aind-data-schema/main/src/aind_data_schema/core/subject.py",
   "schema_version": "0.5.6",
   "subject_id": "12345",
   "sex": "Male",
   "date_of_birth": "2022-11-22",
   "genotype": "Emx1-IRES-Cre;Camk2a-tTA;Ai93(TITL-GCaMP6f)",
   "species": {
      "name": "Mus musculus",
      "abbreviation": null,
      "registry": {
         "name": "National Center for Biotechnology Information",
         "abbreviation": "NCBI"
      },
      "registry_identifier": "10090"
   },
   "alleles": [],
   "background_strain": "C57BL/6J",
   "breeding_info": {
      "breeding_group": "Emx1-IRES-Cre(ND)",
      "maternal_id": "546543",
      "maternal_genotype": "Emx1-IRES-Cre/wt; Camk2a-tTa/Camk2a-tTA",
      "paternal_id": "232323",
      "paternal_genotype": "Ai93(TITL-GCaMP6f)/wt"
   },
   "source": {
      "name": "Allen Institute",
      "abbreviation": "AI",
      "registry": {
         "name": "Research Organization Registry",
         "abbreviation": "ROR"
      },
      "registry_identifier": "03cpe7c52"
   },
   "rrid": null,
   "restrictions": null,
   "wellness_reports": [],
   "housing": {
      "cage_id": "123",
      "room_id": null,
      "light_cycle": null,
      "home_cage_enrichment": [
         "Running wheel"
      ],
      "cohoused_subjects": []
   },
   "notes": null
}
```

## Installing and Upgrading

To install the latest version:
```
pip install aind-data-schema
```

Every merge to the `main` branch is automatically tagged with a new major/minor/patch version and uploaded to PyPI. To upgrade to the latest version:
```
pip install aind-data-schema --upgrade
```

## Issues and Discussions
If you've found a bug in the schemas or would like to make a minor change, open an [issue](https://github.com/AllenNeuralDynamics/aind-data-schema/issues) and please use the provided [templates](https://github.com/AllenNeuralDynamics/aind-metadata-mapper/issues/new/choose).
If you'd like to propose a large change or addition, or generally have a question about how things work, head start a new [Discussion](https://github.com/AllenNeuralDynamics/aind-data-schema/discussions)!

## Controlled Vocabularies

Controlled vocabularies and other enumerated lists are maintained in a separate repository: [aind-data-schema-models](https://github.com/AllenNeuralDynamics/aind-data-schema-models). This allows us to specify these lists without changing aind-data-schema. Controlled vocabularies include lists of organizations, manufacturers, species, modalities, platforms, units, harp devices, and registries.

To upgrade to the latest data models version:
```
pip install aind-data-schema-models --upgrade
```

## Contributing
Contributions are more than welcome for this project! If you'd like to develop the code, please follow the standards outlined in the [contribution guide](https://github.com/AllenNeuralDynamics/aind-data-schema/blob/main/CONTRIBUTING.md).

### Documentation

To generate the rst files source files for documentation, run:

```
sphinx-apidoc -o docs/source/ src
```

Then to create the documentation html files, run:
```
sphinx-build -b html docs/source/ docs/build/html
```

More info on sphinx installation can be found here: https://www.sphinx-doc.org/en/master/usage/installation.html


.. a location that the doc/index.rst uses for including this file
.. before_header

.. image:: https://github.com/codespaces/badge.svg
  :width: 160
  :target: https://codespaces.new/gymrek-lab/TRTools

.. image:: https://github.com/gymrek-lab/trtools/workflows/Tests/badge.svg
    :target: https://github.com/gymrek-lab/trtools/workflows/Tests/badge.svg

.. image:: https://codecov.io/gh/gymrek-lab/TRTools/branch/master/graph/badge.svg
  :target: https://codecov.io/gh/gymrek-lab/TRTools

.. a location that the doc/index.rst uses for including this file
.. after_header

TRTools
=======

.. a location that the doc/index.rst uses for including this file
.. after_title

TRTools includes a variety of utilities for filtering, quality control and analysis of tandem repeats downstream of genotyping them from next-generation sequencing. It supports multiple recent genotyping tools (see below).

See full documentation and examples at https://trtools.readthedocs.io/en/stable/.

If you use TRTools in your work, please cite: Nima Mousavi, Jonathan Margoliash, Neha Pusarla, Shubham Saini, Richard Yanicky, Melissa Gymrek. (2020) TRTools: a toolkit for genome-wide analysis of tandem repeats. Bioinformatics. (https://doi.org/10.1093/bioinformatics/btaa736)

Install
-------

Note: TRTools supports Python versions 3.8 and up. We do not officially support python version 3.7 as it is `end of life <https://devguide.python.org/versions/#status-of-python-versions>`_, but we believe TRTools likely works with it from previous testing results.

With conda
^^^^^^^^^^

::

        conda install -c conda-forge -c bioconda trtools

Optionally install :code:`bcftools` which is used to prepare input files for TRTools (and :code:`ART` which is used by simTR) by running:

::

        conda install -c conda-forge -c bioconda bcftools art

With pip
^^^^^^^^

First install :code:`htslib` (which contains :code:`tabix` and :code:`bgzip`). Optionally install :code:`bcftools`.
These are used to prepare input files for TRTools and aren't installed by pip.

Then run:

::

        pip install --upgrade pip
        pip install trtools

Note: TRTools installation may fail for pip version 10.0.1, hence the need to upgrade pip first

Note: if you will run or test :code:`simTR`, you will also need to install `ART <https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm>`_. The simTR tests will only run if the executable :code:`art_illumina` is found on your :code:`PATH`. If it has been installed, :code:`which art_illumina` should return a path.

From source
^^^^^^^^^^^

To install from source (only recommended for development) clone the TRTools repository from `github <https://github.com/gymrek-lab/TRTools/>`_ and checkout the branch you're interested in::

        git clone -b master https://github.com/gymrek-lab/TRTools
        cd TRTools/

Now, create 1) a conda environment with our development tools and 2) a virtual environment with our dependencies and an editable install of TRTools::

        conda env create -n trtools -f dev-env.yml
        conda run -n trtools poetry install

Now, whenever you'd like to run/import pytest or TRTools, you will first need to activate both environments::

        conda activate trtools
        poetry shell

With Docker
^^^^^^^^^^^

Please refer to `the biocontainers registry for TRTools <https://biocontainers.pro/tools/trtools>`_ for all of our images. To use the most recent release, run the following command::

        docker pull quay.io/biocontainers/trtools:latest

Tools
-----
TRTools includes the following tools.

* `mergeSTR <https://trtools.readthedocs.io/en/stable/source/mergeSTR.html>`_: a tool to merge VCF files across multiple samples genotyped using the same tool
* `dumpSTR <https://trtools.readthedocs.io/en/stable/source/dumpSTR.html>`_: a tool for filtering VCF files with TR genotypes
* `qcSTR <https://trtools.readthedocs.io/en/stable/source/qcSTR.html>`_: a tool for generating various quality control plots for a TR callset
* `statSTR <https://trtools.readthedocs.io/en/stable/source/statSTR.html>`_: a tool for computing various statistics on VCF files
* `compareSTR <https://trtools.readthedocs.io/en/stable/source/compareSTR.html>`_: a tool for comparing TR callsets
* `associaTR <https://trtools.readthedocs.io/en/stable/source/associaTR.html>`_: a tool for testing TR length-phenotype associations (e.g., running a TR GWAS)
* `prancSTR <https://trtools.readthedocs.io/en/stable/source/prancSTR.html>`_: a tool for identifying somatic mosacisim at TRs. Currently only compatible with HipSTR VCF files. (*beta mode*)
* `simTR <https://trtools.readthedocs.io/en/stable/source/simTR.html>`_: a tool for simulating next-generation sequencing reads from TR regions. (*beta mode*)

Type :code:`<command> --help` to see a full set of options.

It additionally includes a python library, :code:`trtools`, which can be accessed from within Python scripts. e.g.::

        import trtools.utils.utils as stls
        allele_freqs = {5: 0.5, 6: 0.5} # 50% of alleles have 5 repeat copies, 50% have 6
        stls.GetHeterozygosity(allele_freqs) # should return 0.5

Usage
-----

We recommend new users start with the example commands described in the `command-line interface for each tool <https://trtools.readthedocs.io/en/stable/UTILITIES.html>`_.
We also suggest going through our `vignettes <https://trtools.readthedocs.io/en/stable/VIGNETTES.html>`_ that walk through some example workflows using TRTools.

Supported TR Callers
--------------------
TRTools supports VCFs from the following TR genotyping tools:

* AdVNTR_
* ExpansionHunter_
* GangSTR_ version 2.4 or higher
* HipSTR [`main repo <https://github.com/tfwillems/HipSTR>`_] [`Gymrek Lab repo <https://github.com/gymrek-lab/hipstr>`_]
* PopSTR_ version 2 or higher

See our description of the `features and example use-cases <https://trtools.readthedocs.io/en/stable/CALLERS.html>`_ of each of these tools.

..
    please ensure this list of links remains the same as the one in the main README

.. _AdVNTR: https://advntr.readthedocs.io/en/latest/
.. _ExpansionHunter: https://github.com/Illumina/ExpansionHunter
.. _GangSTR: https://github.com/gymreklab/gangstr
.. _HipSTR: https://hipstr-tool.github.io/HipSTR/
.. _PopSTR: https://github.com/DecodeGenetics/popSTR

Testing
-------
After you've installed TRTools, we recommend running our tests to confirm that TRTools works properly on your system. Just execute the following::

        test_trtools.sh

Development Notes
-----------------

* TRTools only currently supports diploid genotypes. Haploid calls, such as those on male chrX or chrY, are not yet supported but should be coming soon.

Contact Us
----------
Please submit an issue on the `trtools github <https://github.com/gymrek-lab/TRTools>`_

.. _Contributing:

Contributing
------------
We appreciate contributions to TRTools. If you would like to contribute a fix or new feature, follow these guidelines:

1. Consider `discussing <https://github.com/gymrek-lab/TRTools/issues>`_ your solution with us first so we can provide help or feedback if necessary.
#. Install TRTools from source `as above <From source_>`_.
#. Fork the TRTools repository.
#. Create a branch off of :code:`master` titled with the name of your feature.
#. Make your changes.
#. If you need to add a dependency or update the version of a dependency, you can use the :code:`poetry add` command.

    * You should specify a `version constraint <https://python-poetry.org/docs/master/dependency-specification#version-constraints>`_ when adding a dependency. Use the oldest version compatible with your code. Don't worry if you're not sure at first, since you can (and should!) always update it later. For example, to specify a version of :code:`numpy>=1.23.0`, you can run :code:`poetry add 'numpy>=1.23.0'`.
    * Afterwards, double-check that the :code:`poetry.lock` file contains 1.23.0 in it. **All of our dependencies should be locked to their minimum versions at all times.** To downgrade to a specific version of :code:`numpy` in our lock file, you can explicitly add the version via :code:`poetry add 'numpy==1.23.0'`, manually edit the pyproject.toml file to use a :code:`>=` sign in front of the version number, and then run :code:`poetry lock --no-update`.

#. Document your changes.

   * Ensure all functions, modules, classes etc. conform to `numpy docstring standards <https://numpydoc.readthedocs.io/en/latest/format.html>`_.

    If applicable, update the REAMDEs in the directories of the files you changed with new usage information.

   * New doc pages for `the website <https://trtools.readthedocs.io/en/stable/>`_ can be created under :code:`<project-root>/doc` and linked to as appropriate.
   * If you have added significant amounts of documentation in any of these ways, build the documentation locally to ensure it looks good.

    :code:`cd` to the :code:`doc` directory and run :code:`make clean && make html`, then view :code:`doc/_build/html/index.html` and navigate from there

#. Add tests to test any new functionality. Add them to the :code:`tests/` folder in the directory of the code you modified.

   * :code:`cd` to the root of the project and run :code:`poetry run pytest --cov=. --cov-report term-missing` to make sure that (1) all tests pass and (2) any code you have added is covered by tests. (Code coverage may **not** go down).
   * :code:`cd` to the root of the project and run :code:`nox` to make sure that the tests pass on all versions of python that we support.

#. Submit a pull request (PR) **to the master branch** of the central repository with a description of what changes you have made. Prefix the title of the PR according to the `conventional commits spec <https://www.conventionalcommits.org>`_.
   A member of the TRTools team will reply and continue the contribution process from there, possibly asking for additional information/effort on your part.

   * If you are reviewing a pull request, please double-check that the PR addresses each item in `our PR checklist <https://github.com/gymrek-lab/TRTools/blob/master/.github/pull_request_template.md>`_

Publishing
----------
If you are a TRTools maintainer and wish to publish changes and distribute them to PyPI and bioconda, please see PUBLISHING.rst in the root of the git repo.
If you are a community member and would like that to happen, contact us (see above).


[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![testing: pytest](https://img.shields.io/badge/testing-pytest-blue.svg)](https://docs.pytest.org/en/latest/contents.html)
[![license: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-lightgrey)](http://www.apache.org/licenses/LICENSE-2.0)
[![DELV - Terms of Service](https://img.shields.io/badge/DELV-Terms_of_Service-orange)](https://delv-public.s3.us-east-2.amazonaws.com/delv-terms-of-service.pdf)

[![testing: coverage](https://codecov.io/gh/delvtech/agent0/branch/main/graph/badge.svg?token=1S60MD42ZP)](https://app.codecov.io/gh/delvtech/agent0?displayType=list)
[![docs: build](https://readthedocs.org/projects/agent0/badge/?version=latest)](https://agent0.readthedocs.io/en/latest/?badge=latest)

# [DELV](https://delv.tech) repo for market simulation and analysis

This docs page can be found via [https://agent0.readthedocs.io/en/latest/](https://agent0.readthedocs.io/en/latest/).

## Quickstart

This repo contains general purpose code for interacting with Ethereum smart contracts.
However, it was bulit for the primary use case of trading on [Hyperdrive](https://hyperdrive.delv.tech) markets.

First, install [Foundry](https://book.getfoundry.sh/getting-started/installation) and [Docker](https://docs.docker.com/engine/install/).

Next, using a Python 3.10 environment, you can install agent0 via pip:

```sh
pip install --upgrade agent0
```

Finally, you can execute Hyperdrive trades in a simulated blockchain environment:

```python
import datetime
from fixedpointmath import FixedPoint
from agent0 import LocalHyperdrive, LocalChain

# Initialize
chain = LocalChain()
hyperdrive = LocalHyperdrive(chain)
hyperdrive_agent0 = chain.init_agent(base=FixedPoint(100_000), eth=FixedPoint(10), pool=hyperdrive)

# Run trades
chain.advance_time(datetime.timedelta(weeks=1))
open_long_event = hyperdrive_agent0.open_long(base=FixedPoint(100), eth=FixedPoint(10))
chain.advance_time(datetime.timedelta(weeks=5))
close_event = hyperdrive_agent0.close_long(
    maturity_time=open_long_event.maturity_time, bonds=open_long_event.bond_amount
)

# Analyze
pool_info = hyperdrive.get_pool_info(coerce_float=True)
pool_info.plot(x="block_number", y="longs_outstanding", kind="line")
```

See our [tutorial notebook](examples/tutorial.ipynb) and [examples notebook](examples/short_examples.ipynb) for more information, including details on executing trades on remote chains.

## Install

Please refer to [INSTALL.md](INSTALL.md) for more advanced install options.

## Deployment

Please refer to [BUILD.md](BUILD.md).

## Testing

We deploy a local anvil chain to run system tests.
Therefore, you must [install foundry](https://github.com/foundry-rs/foundry#installatio://github.com/foundry-rs/foundry#installation) as a prerequisite for running tests.

Testing is achieved with [py.test](https://docs.pytest.org/en/latest/contents.html).
You can run all tests from the repository root directory by running `python -m pytest`, or you can pick a specific test with `python -m pytest {path/to/test_file.py}`.
General integration-level tests are in the `tests` folder, while more isolated or unit tests are colocated with the files they are testing and end with a `_test.py` suffix.

## Contributions

Please refer to [CONTRIBUTING.md](CONTRIBUTING.md).

## Coverage

To run coverage locally you can follow these steps:

```bash
pip install coverage
coverage run -m pytest
coverage html
```

then just open `htmlcov/index.html` to view the report!

## Number format

We frequently use 18-decimal [fixed-point precision numbers](https://github.com/delvtech/fixedpointmath#readme) for arithmetic.

## Disclaimer

This project is a work-in-progress.
The language used in this code and documentation is not intended to, and does not, have any particular financial, legal, or regulatory significance.

---

Copyright © 2024  DELV

Licensed under the Apache License, Version 2.0 (the "OSS License").

By accessing or using this code, you signify that you have read, understand and agree to be bound by and to comply with the [OSS License](http://www.apache.org/licenses/LICENSE-2.0) and [DELV's Terms of Service](https://delv-public.s3.us-east-2.amazonaws.com/delv-terms-of-service.pdf). If you do not agree to those terms, you are prohibited from accessing or using this code.

Unless required by applicable law or agreed to in writing, software distributed under the OSS License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the OSS License and the DELV Terms of Service for the specific language governing permissions and limitations.


# DeepRank2

|     Badges     |                                                                                |
| :------------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|  **fairness**  | [![fair-software.eu](https://img.shields.io/badge/fair--software.eu-%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F-green)](https://fair-software.eu) [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6403/badge)](https://bestpractices.coreinfrastructure.org/projects/6403)                                                                                |
|  **package**   | [![PyPI version](https://badge.fury.io/py/deeprank2.svg)](https://badge.fury.io/py/deeprank2) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/b1bde03fc0334e07b0cd8a69ce2adeb3)](https://app.codacy.com/gh/DeepRank/deeprank2/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)                                                                                |
|    **docs**    | [![Documentation Status](https://readthedocs.org/projects/deeprank2/badge/?version=latest)](https://deeprank2.readthedocs.io/en/latest/?badge=latest) [![RSD](https://img.shields.io/badge/RSD-deeprank2-pink)](https://research-software-directory.org/software/deeprankcore) [![DOI](https://zenodo.org/badge/450496579.svg)](https://zenodo.org/badge/latestdoi/450496579) [![DOI](https://joss.theoj.org/papers/10.21105/joss.05983/status.svg)](https://doi.org/10.21105/joss.05983)                                                                        |
|   **tests**    | [![Build Status](https://github.com/DeepRank/deeprank2/actions/workflows/build-repo.yml/badge.svg)](https://github.com/DeepRank/deeprank2/actions) ![Linting status](https://github.com/DeepRank/deeprank2/actions/workflows/linting.yml/badge.svg?branch=main) [![Coverage Status](https://coveralls.io/repos/github/DeepRank/deeprank2/badge.svg?branch=main)](https://coveralls.io/github/DeepRank/deeprank2?branch=main) ![Python](https://img.shields.io/badge/python-3.10-blue.svg)  |
| **running on** | ![Ubuntu](https://img.shields.io/badge/Ubuntu-E95420?style=for-the-badge&logo=ubuntu&logoColor=white)                                                                                |
|  **license**   | [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/license/apache-2-0/)                                                                                |

## Overview

DeepRank2 is an open-source deep learning (DL) framework for data mining of protein-protein interfaces (PPIs) or single-residue variants (SRVs). This package is an improved and unified version of three previously developed packages: [DeepRank](https://github.com/DeepRank/deeprank), [DeepRank-GNN](https://github.com/DeepRank/Deeprank-GNN), and [DeepRank-Mut](https://github.com/DeepRank/DeepRank-Mut).

As input, DeepRank2 takes [PDB-formatted](https://www.cgl.ucsf.edu/chimera/docs/UsersGuide/tutorials/pdbintro.html) atomic structures, and map them to graphs, where nodes can represent either residues or atoms, as chosen by the user, and edges represent the interactions between them. DeepRank2 has the option to choose between two types of queries as input for the featurization phase:

- PPIs, for mining interaction patterns within protein-protein complexes, implemented by the `ProteinProteinInterfaceQuery` class;
- SRVs, for mining mutation phenotypes within protein structures, implemented by the `SingleResidueVariantQuery` class.

The physico-chemical and geometrical features are then computed and assigned to each node and edge. The user can choose which features to generate from several pre-existing options defined in the package, or define custom features modules, as explained in the documentation. The graphs can then be mapped to 3D-grids as well. The generated data can be used for training neural networks. DeepRank2 also offers a pre-implemented training pipeline, using either [CNNs](https://en.wikipedia.org/wiki/Convolutional_neural_network) (for 3D-grids) or [GNNs](https://en.wikipedia.org/wiki/Graph_neural_network) (for graphs), as well as output exporters for evaluating performances.

Main features:

- Predefined atom-level and residue-level feature types
  - e.g. atom/residue type, charge, size, potential energy
  - All features' documentation is available [here](https://deeprank2.readthedocs.io/en/latest/features.html)
- Predefined target types
  - binary class, CAPRI categories, DockQ, RMSD, and FNAT
  - Detailed docking scores documentation is available [here](https://deeprank2.readthedocs.io/en/latest/docking.html)
- Flexible definition of both new features and targets
- Features generation for both graphs and 3D-grids
- Efficient data storage in HDF5 format
- Support for both classification and regression (based on [PyTorch](https://pytorch.org/) and [PyTorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/))

📚 [Documentation](https://deeprank2.rtfd.io/)

📣 [Discussions](https://github.com/DeepRank/deeprank2/discussions)

## Table of contents

- [DeepRank2](#deeprank2)
  - [Overview](#overview)
  - [Table of contents](#table-of-contents)
  - [Installation](#installation)
    - [Containerized Installation](#containerized-installation)
    - [Local/remote installation](#localremote-installation)
      - [YML file installation (recommended)](#yml-file-installation-recommended)
      - [Manual installation (customizable)](#manual-installation-customizable)
      - [Testing DeepRank2 installation](#testing-deeprank2-installation)
  - [Contributing](#contributing)
  - [Using DeepRank2](#using-deeprank2)
    - [Data generation](#data-generation)
    - [Datasets](#datasets)
      - [GraphDataset](#graphdataset)
      - [GridDataset](#griddataset)
    - [Training](#training)
      - [Run a pre-trained model on new data](#run-a-pre-trained-model-on-new-data)
  - [Computational performances](#computational-performances)
  - [Package development](#package-development)

## Installation

There are two ways to install DeepRank2:

1. In a [dockerized container](#containerized-installation). This allows you to use DeepRank2, including all the notebooks within the container (a protected virtual space), without worrying about your operating system or installation of dependencies.
   - We recommend this installation for inexperienced users and to learn to use or test our software, e.g. using the provided [tutorials](tutorials/TUTORIAL.md). However, resources might be limited in this installation and we would not recommend using it for large datasets or on high-performance computing facilities.
2. [Local installation](#localremote-installation) on your system. This allows you to use the full potential of DeepRank2, but requires a few additional steps during installation.
   - We recommend this installation for more experienced users, for larger projects, and for (potential) [contributors](#contributing) to the codebase.

### Containerized Installation

In order to try out the package without worrying about your OS and without the need of installing all the required dependencies, we created a `Dockerfile` that can be used for taking care of everything in a suitable container.

For this, you first need to install [Docker](https://docs.docker.com/engine/install/) on your system. Then run the following commands. You may need to have sudo permission for some steps, in which case the commands below can be preceded by `sudo`:

```bash
# Clone the DeepRank2 repository and enter its root directory
git clone https://github.com/DeepRank/deeprank2
cd deeprank2

# Build and run the Docker image
docker build -t deeprank2 .
docker run -p 8888:8888 deeprank2
```

Next, open a browser and go to `http://localhost:8888` to access the application running inside the Docker container. From there you can use DeepRank2, e.g. to run the tutorial notebooks.

More details about the tutorials' contents can be found [here](https://github.com/DeepRank/deeprank2/blob/main/tutorials/TUTORIAL.md). Note that in the docker container only the raw PDB files are downloaded, which needed as a starting point for the tutorials. You can obtain the processed HDF5 files by running the `data_generation_xxx.ipynb` notebooks. Because Docker containers are limited in memory resources, we limit the number of data points processed in the tutorials. Please [install the package locally](#localremote-installation) to fully leverage its capabilities.

If after running the tutorials you want to remove the (quite large) Docker image from your machine, you must first [stop the container](https://docs.docker.com/engine/reference/commandline/stop/) and can then [remove the image](https://docs.docker.com/engine/reference/commandline/image_rm/). More general information about Docker can be found on the [official website docs](https://docs.docker.com/get-started/).

### Local/remote installation

Local installation is formally only supported on the latest stable release of ubuntu, for which widespread automated testing through continuous integration workflows has been set up. However, it is likely that the package runs smoothly on other operating systems as well.

Before installing DeepRank2 please ensure you have [GCC](https://gcc.gnu.org/install/) installed: if running `gcc --version` gives an error, run `sudo apt-get install gcc`.

#### YML file installation (recommended)

You can use the provided YML file for creating a [conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) containing the latest stable release of DeepRank2 and all its dependencies.
This will install the CPU-only version of DeepRank2 on Python 3.10.
Note that this will not work for MacOS. Do the [Manual Installation](#manual-installation) instead.

```bash
# Clone the DeepRank2 repository and enter its root directory
git clone https://github.com/DeepRank/deeprank2
cd deeprank2

# Ensure you are in your base environment
conda activate
# Create the environment
conda env create -f env/deeprank2.yml
# Activate the environment
conda activate deeprank2
# Install the latest deeprank2 release
pip install deeprank2
```

See instructions below to [test](#testing-deeprank2-installation) that the installation was succesful.

#### Manual installation (customizable)

If you want to use the GPUs, choose a specific python version, are a MacOS user, or if the YML installation was not successful, you can install the package manually. We advise to do this inside a [conda virtual environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).

You can first remove from `env/deeprank2.yml` the packages that cannot be installed properly, or the ones that you want to install differently (e.g., pytorch-related packages if you wish to install the CUDA version), and then proceed with the environment creation by using the edited YML file: `conda env create -f env/deeprank2.yml`. Then activate the environment, and proceed with installing the missing packages, which might fall into the following list. If you have any issues during installation of dependencies, please refer to the official documentation for each package (linked below), as our instructions may be out of date (last tested on 19 Feb 2024):

- [MSMS](https://anaconda.org/bioconda/msms): [Here](https://ssbio.readthedocs.io/en/latest/instructions/msms.html) for MacOS with M1 chip users.
- [PyTorch](https://pytorch.org/get-started/locally/)
  - Pytorch regularly publishes updates and not all newest versions will work stably with DeepRank2. Currently, the package is tested on ubuntu using [PyTorch 2.1.1](https://pytorch.org/get-started/previous-versions/#v211).
  - We support torch's CPU library as well as CUDA.
- [PyG](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) and its optional dependencies: `torch_scatter`, `torch_sparse`, `torch_cluster`, `torch_spline_conv`.
  - The exact command to install pyg will depend on the version of pytorch you are using. Please refer to the source's installation instructions (we recommend using the pip installation for this as it also shows the command for the dependencies).
- [FreeSASA](https://freesasa.github.io/python/).

Finally install deeprank2 itself: `pip install deeprank2`.

Alternatively, get the latest updates by cloning the repo and installing the editable version of the package with:

```bash
git clone https://github.com/DeepRank/deeprank2
cd deeprank2
pip install -e .'[test]'
```

The `test` extra is optional, and can be used to install test-related dependencies, useful during development.

#### Testing DeepRank2 installation

You can check that all components were installed correctly, using `pytest`. We especially recommend doing this in case you installed DeepRank2 and its dependencies manually (the latter option above).

The quick test should be sufficient to ensure that the software works, while the full test (a few minutes) will cover a much broader range of settings to ensure everything is correct.

First run `pip install pytest`, if you did not install it above. Then run `pytest tests/test_integration.py` for the quick test or just `pytest` for the full test (expect a few minutes to run).

## Contributing

If you would like to contribute to the package in any way, please see [our guidelines](CONTRIBUTING.rst).

## Using DeepRank2

The following section serves as a first guide to start using the package, using protein-protein Interface (PPI) queries as example. For an enhanced learning experience, we provide in-depth [tutorial notebooks](https://github.com/DeepRank/deeprank2/tree/main/tutorials) for generating PPI data, generating SVR data, and for the training pipeline.
For more details, see the [extended documentation](https://deeprank2.rtfd.io/).

### Data generation

For each protein-protein complex (or protein structure containing a missense variant), a `Query` can be created and added to the `QueryCollection` object, to be processed later on. Two subtypes of `Query` exist: `ProteinProteinInterfaceQuery` and `SingleResidueVariantQuery`.

A `Query` takes as inputs:

- A `.pdb` file, representing the molecular structure.
- The resolution (`"residue"` or `"atom"`), i.e. whether each node should represent an amino acid residue or an atom.
- `chain_ids`, the chain ID or IDs (generally single capital letter(s)).
  - `SingleResidueVariantQuery` takes a single ID, which represents the chain containing the variant residue.
  - `ProteinProteinInterfaceQuery` takes a pair of ids, which represent the chains between which the interface exists.
  - Note that in either case this does not limit the structure to residues from this/these chain/s. The structure contained in the `.pdb` can thus have any number of chains, and residues from these chains will be included in the graphs and 3D-grids produced by DeepRank2 (if they are within the `influence_radius`).
- Optionally, the correspondent position-specific scoring matrices (PSSMs), in the form of `.pssm` files.

```python
from deeprank2.query import QueryCollection, ProteinProteinInterfaceQuery

queries = QueryCollection()

# Append data points
queries.add(ProteinProteinInterfaceQuery(
    pdb_path = "tests/data/pdb/1ATN/1ATN_1w.pdb",
    resolution = "residue",
    chain_ids = ["A", "B"],
    targets = {
        "binary": 0
    },
    pssm_paths = {
        "A": "tests/data/pssm/1ATN/1ATN.A.pdb.pssm",
        "B": "tests/data/pssm/1ATN/1ATN.B.pdb.pssm"
    }
))
queries.add(ProteinProteinInterfaceQuery(
    pdb_path = "tests/data/pdb/1ATN/1ATN_2w.pdb",
    resolution = "residue",
    chain_ids = ["A", "B"],
    targets = {
        "binary": 1
    },
    pssm_paths = {
        "A": "tests/data/pssm/1ATN/1ATN.A.pdb.pssm",
        "B": "tests/data/pssm/1ATN/1ATN.B.pdb.pssm"
    }
))
queries.add(ProteinProteinInterfaceQuery(
    pdb_path = "tests/data/pdb/1ATN/1ATN_3w.pdb",
    resolution = "residue",
    chain_ids = ["A", "B"],
    targets = {
        "binary": 0
    },
    pssm_paths = {
        "A": "tests/data/pssm/1ATN/1ATN.A.pdb.pssm",
        "B": "tests/data/pssm/1ATN/1ATN.B.pdb.pssm"
    }
))

```

The user is free to implement a custom query class. Each implementation requires the `build` method to be present.

The queries can then be processed into graphs only or both graphs and 3D-grids, depending on which kind of network will be used later for training.

```python
from deeprank2.features import components, conservation, contact, exposure, irc, surfacearea
from deeprank2.utils.grid import GridSettings, MapMethod

feature_modules = [components, conservation, contact, exposure, irc, surfacearea]

# Save data into 3D-graphs only
hdf5_paths = queries.process(
    "<output_folder>/<prefix_for_outputs>",
    feature_modules = feature_modules)

# Save data into graphs and 3D-grids
hdf5_paths = queries.process(
    "<output_folder>/<prefix_for_outputs>",
    feature_modules = feature_modules,
    grid_settings = GridSettings(
        # the number of points on the x, y, z edges of the cube
        points_counts = [20, 20, 20],
        # x, y, z sizes of the box in Å
        sizes = [1.0, 1.0, 1.0]),
    grid_map_method = MapMethod.GAUSSIAN)
```

### Datasets

Data can be split in sets implementing custom splits according to the specific application. Assuming that the training, validation and testing ids have been chosen (keys of the HDF5 file/s), then the `DeeprankDataset` objects can be defined.

#### GraphDataset

For training GNNs the user can create a `GraphDataset` instance:

```python
from deeprank2.dataset import GraphDataset

node_features = ["bsa", "res_depth", "hse", "info_content", "pssm"]
edge_features = ["distance"]
target = "binary"
train_ids = [<ids>]
valid_ids = [<ids>]
test_ids = [<ids>]

# Creating GraphDataset objects
dataset_train = GraphDataset(
    hdf5_path = hdf5_paths,
    subset = train_ids,
    node_features = node_features,
    edge_features = edge_features,
    target = target
)
dataset_val = GraphDataset(
    hdf5_path = hdf5_paths,
    subset = valid_ids,
    train_source = dataset_train
)
dataset_test = GraphDataset(
    hdf5_path = hdf5_paths,
    subset = test_ids,
    train_source = dataset_train
)
```

#### GridDataset

For training CNNs the user can create a `GridDataset` instance:

```python
from deeprank2.dataset import GridDataset

features = ["bsa", "res_depth", "hse", "info_content", "pssm", "distance"]
target = "binary"
train_ids = [<ids>]
valid_ids = [<ids>]
test_ids = [<ids>]

# Creating GraphDataset objects
dataset_train = GridDataset(
    hdf5_path = hdf5_paths,
    subset = train_ids,
    features = features,
    target = target
)
dataset_val = GridDataset(
    hdf5_path = hdf5_paths,
    subset = valid_ids,
    train_source = dataset_train,
)
dataset_test = GridDataset(
    hdf5_path = hdf5_paths,
    subset = test_ids,
    train_source = dataset_train,
)
```

### Training

Let's define a `Trainer` instance, using for example of the already existing `GINet`. Because `GINet` is a GNN, it requires a dataset instance of type `GraphDataset`.

```python
from deeprank2.trainer import Trainer
from deeprank2.neuralnets.gnn.naive_gnn import NaiveNetwork

trainer = Trainer(
    NaiveNetwork,
    dataset_train,
    dataset_val,
    dataset_test
)

```

The same can be done using a CNN, for example `CnnClassification`. Here a dataset instance of type `GridDataset` is required.

```python
from deeprank2.trainer import Trainer
from deeprank2.neuralnets.cnn.model3d import CnnClassification

trainer = Trainer(
    CnnClassification,
    dataset_train,
    dataset_val,
    dataset_test
)
```

By default, the `Trainer` class creates the folder `./output` for storing predictions information collected later on during training and testing. `HDF5OutputExporter` is the exporter used by default, but the user can specify any other implemented exporter or implement a custom one.

Optimizer (`torch.optim.Adam` by default) and loss function can be defined by using dedicated functions:

```python
import torch

trainer.configure_optimizers(torch.optim.Adamax, lr = 0.001, weight_decay = 1e-04)

```

Then the `Trainer` can be trained and tested; the best model in terms of validation loss is saved by default, and the user can modify so or indicate where to save it using the `train()` method parameter `filename`.

```python
trainer.train(
    nepoch = 50,
    batch_size = 64,
    validate = True,
    filename = "<my_folder/model.pth.tar>")
trainer.test()

```

#### Run a pre-trained model on new data

If you want to analyze new PDB files using a pre-trained model, the first step is to process and save them into HDF5 files [as we have done above](#data-generation).

Then, the `DeeprankDataset` instance for the newly processed data can be created. Do this by specifying the path for the pre-trained model in `train_source`, together with the path to the HDF5 files just created. Note that there is no need of setting the dataset's parameters, since they are inherited from the information saved in the pre-trained model. Let's suppose that the model has been trained with `GraphDataset` objects:

```python
from deeprank2.dataset import GraphDataset

dataset_test = GraphDataset(
    hdf5_path = "<output_folder>/<prefix_for_outputs>",
    train_source = "<pretrained_model_path>"
)
```

Finally, the `Trainer` instance can be defined and the new data can be tested:

```python
from deeprank2.trainer import Trainer
from deeprank2.neuralnets.gnn.naive_gnn import NaiveNetwork
from deeprank2.utils.exporters import HDF5OutputExporter

trainer = Trainer(
    NaiveNetwork,
    dataset_test = dataset_test,
    pretrained_model = "<pretrained_model_path>",
    output_exporters = [HDF5OutputExporter("<output_folder_path>")]
)

trainer.test()
```

For more details about how to run a pre-trained model on new data, see the [docs](https://deeprank2.readthedocs.io/en/latest/getstarted.html#run-a-pre-trained-model-on-new-data).

## Computational performances

We measured the efficiency of data generation in DeepRank2 using the tutorials' [PDB files](https://zenodo.org/record/8187806) (~100 data points per data set), averaging the results run on Apple M1 Pro, using a single CPU.
Parameter settings were: atomic resolution, `distance_cutoff` of 5.5 Å, radius (for SRV only) of 10 Å. The [features modules](https://deeprank2.readthedocs.io/en/latest/features.html) used were `components`, `contact`, `exposure`, `irc`, `secondary_structure`, `surfacearea`, for a total of 33 features for PPIs and 26 for SRVs (the latter do not use `irc` features).

|      |            Data processing speed
[seconds/structure]             |                   Memory
[megabyte/structure]                    |
| ---- | :--------------------------------------------------------------------: | :--------------------------------------------------------------------: |
| PPIs | graph only: **2.99** (std 0.23)
graph+grid: **11.35** (std 1.30) | graph only: **0.54** (std 0.07)
graph+grid: **16.09** (std 0.44) |
| SRVs | graph only: **2.20** (std 0.08)
graph+grid: **2.85** (std 0.10)  | graph only: **0.05** (std 0.01)
graph+grid: **17.52** (std 0.59) |

## Package development

If you're looking for developer documentation, go [here](https://github.com/DeepRank/deeprank2/blob/dev/README.dev.md).


Tensile is a tool for creating benchmark-driven backend libraries for GEMMs, GEMM-like problems (such as batched GEMM), and general N-dimensional tensor contractions on a GPU.
The Tensile library is mainly used as backend library to rocBLAS.
Tensile acts as the performance backbone for a wide variety of 'compute' applications running on AMD GPUs.

See [Tensile Wiki](https://github.com/ROCm/Tensile/wiki) for documentation.


# Resonite Locale
This repository contains the locale assets for the core UI of [Resonite](https://resonite.com) and allows anyone to contribute translations. The contents of this repository will be periodically merged with the public build released on Steam (STEAM) and other platforms.

## Localization Status
Czech [cs] - 70.6% - Missing keys: 669
German [de] - 100.0% - Missing keys: 0
English (United Kingdom) [en-gb] - 1.8% - Missing keys: 2231
English [en] - 100.0% - Missing keys: 0
Esperanto [eo] - 71.9% - Missing keys: 638
Spanish [es] - 73.4% - Missing keys: 604
Estonian [et] - 36.2% - Missing keys: 1449
Finnish [fi] - 84.3% - Missing keys: 356
French [fr] - 99.6% - Missing keys: 8
Icelandic [is] - 28.1% - Missing keys: 1634
Japanese [ja] - 99.6% - Missing keys: 8
Korean [ko] - 99.6% - Missing keys: 10
Dutch [nl] - 45.5% - Missing keys: 1239
Norwegian [no] - 69.1% - Missing keys: 703
Polish [pl] - 76.1% - Missing keys: 543
Portuguese (Brazil) [pt-br] - 69.0% - Missing keys: 704
Russian [ru] - 100.0% - Missing keys: 0
Swedish [sv] - 28.0% - Missing keys: 1635
Thai [th] - 1.8% - Missing keys: 2232
Turkish [tr] - 35.7% - Missing keys: 1461
Chinese (China) [zh-cn] - 99.6% - Missing keys: 8
Chinese (Taiwan) [zh-tw] - 69.3% - Missing keys: 698

Total keys: 2272

## How To Contribute
If you'd like to contribute translations, create a fork of the repository, make the changes and **once they are ready** to be merged create a Pull Request, so the contributions can be checked and merged. You don't need to translate everything at once, if you cover part of the UI, the changes can be merged, with more translations coming later.

### Translating the Store descriptions
If you're like, you can help translate the store descriptions as well (this is used on Steam for example), but we consider those highly optional since it's quite a lot of text. If you don't want to translate those, don't worry about them! The store descriptions do not count towards the translation completeness percentage and are provided in separate files.

If you do translate them and you haven't added a credit yet, put your name in the regular .json file for translations of in-game strings, even if you haven't translated any in-game strings.

### If you're contributing a new language
1. [Create a new Issue](https://github.com/Yellow-Dog-Man/Locale) for given language in format "Language [lang-code]", for example "English [en]", which will help coordinate efforts of different translators.
1. Verify that our fork of ICU MessageFormat.NET has a pluralizer for your language, by checking the "AddStandardPluralizers()" function in this file: https://github.com/Yellow-Dog-Man/messageformat.net/blob/master/src/Jeffijoe.MessageFormat/Formatting/Formatters/PluralFormatter.cs

If you can't find your language code in this file, please make a Issue either on in this repository.

Alternatively you can implement the pluralizer yourself based on the reference from the Unicode CLDR repository: https://github.com/unicode-org/cldr/blob/master/common/supplemental/plurals.xml and make a pull request for it to be merged with our fork or MessageFormat.NET

3) Add a new [lang-code].json file to your fork. We highly recommend creating a skeleton file first without any translation strings, just containing the Locale and Authors and creating a pull request, so it's clearer to other contributors that translations are being worked on by someone.

### Contributing translations to a language
If you'd like to contribute translations for existing language file (or one you have just created), we recommend the following:

1) Make a fork of the repository or your own branch.
2) Update the language file, either by modifying the translation strings or adding new ones for missing translations.
3) Ensure you do not have any left-over English strings in the file. Your file should only contain actually translated strings. Any missing strings will automatically fallback (see below for details)
4) Ensure your modified translation file works correctly inside Resonite (see below how to test)
5) Create a Pull Request for your translations to be merged into the main repository. After merging they will be available publicly in the next public build of Resonite.

As we develop our platform, we'll be constantly adding new strings in English or modifying the existing ones. We recommend watching the repository for activity through GitHub, so you can get notified when there are changes and new strings to be translated.

### Collaborating & Conflicts
Sometimes when translating, multiple users can translate the same area/section at the same time. This results in a Conflict, as the Resonite Team usually cannot speak the language. We're unable to decide which PR should be merged. When a conflict occurs any PRs involved will be marked with a Conflicts label. Please work with other translators of the language to resolve any conflicts so that the team can proceed to merge PRs clearly.

### Testing your translation
As you work on the translation we recommend that you periodically check it inside of Resonite. This will not only help ensure that you don't have any syntax errors, but also make sure that the strings are correct in the context.

To test the translation, find folder where Resonite.exe is installed (on Steam, you can do so by right clicking Resonite, going to Properties -> LOCAL FILES -> BROWSE LOCAL FILES...) and then locate the "Locale" folder. Simply place your modified file into this folder and Resonite will load it up.

By default, Resonite uses your system locale to determine which file to load. You can override this by going to Settings and changing the "Override Locale" to a different language code.

- You can edit the translation file on the fly without shutting down Resonite. To force it to reload, change the locale to "en" and then back to your own.
- Note that while most UI will change language immediately, not all of it would. Simply close and reopen the UI dialog to load the translated strings
- If the string is showing in English, you probably have a typo in the string key. It needs to match exactly
- If the translation isn't loading in Resonite, it is likely JSON syntax error preventing it from being loaded
- If you see "ERROR!!!" instead of your translated string, you have a syntax error in the particular string. Check Resonite's log file, which will contain details.

### If you use an external tool to do the translation and the JSON structure is mangled
You can use the python script in this repository: CleanJSON.py

For example to clean the french json, `./CleanJSON.py --en en.json --lang fr.json --out fr.json.cleaned`

```usage: CleanJSON.py [-h] [--en en_path] [--lang lang_path] [--out out_path]

This script will reformat a Babel style JSON for locales to match the en.json
baseline formatting for git changes purposes.

optional arguments:
  -h, --help        show this help message and exit
  --en en_path      The path to the en.json locale file.
  --lang lang_path  The path to the LANG.json locale file to clean.
  --out out_path    The path to save the formatted file.
```

## Do's and Don'ts
- Make sure the .json locale file is UTF8 encoded
- Always keep the "Dummy" : "Dummy" entry at the bottom of the file. This way you don't have to remember to remove the comma at the end of the last entry every time
- Copy & Paste the whole content of the file into this online validator to ensure you don't have any syntax errors: https://jsonformatter.curiousconcept.com/
- **DO** use spaces instead of tabs to keep the formatting of all documents consistent
- **DO** check if other users are making modifications to the same locale as you are in Issues and coordinate. If you send changes that conflict with other users, it's hard to resolve them on our end since we don't understand the language.
- **DO** keep the same style of formatting and coloring where it's present in the text to ensure it stays consistent across locales. Coloring in particular can be contextual and used in other parts of the UI and changing or removing it would cause user confusion.

- **DON'T** update the Localization Status section of this document, it is automatically generated when changes are merged
- **DON'T** convert the formatting of the entire document. This creates major merge conflicts and makes it hard to track what was actually changed, plus it introduces inconsistencies
- **DON'T** correct mistakes in the string keys, only report them. They will be fixed by a script, which will apply the correction to all locales at once.
- **DON'T** submit purely machine translated locales. Those often result in odd and confusing results for user interfaces. Using machine translation as basis for manual translation is ok.
- **DON'T** Submit strings from PRs that have the "New Strings" label until that PR is merged. The development team may need to update or change these strings as a part of their inclusion and this can cause conflicts. Wait till the PR is merged and THEN work on the new strings.

## The ICU MessageFormat Syntax for translation strings
Resonite uses the ICU MessageFormat Syntax defined by the Unicode organization for its localized strings. This offers high amount of flexibility on how you translate strings and ensures that you can correctly follow the grammar rules of your language, particularly with regards to pluralization (e.g. displaying "1 item" vs "1 items"). This is why it's important to ensure that your language has a pluralizer implemented in our fork of MessageFormat.NET

To learn more about the ICU MessageFormat Syntax check the following links:
https://unicode-org.github.io/icu/userguide/format_parse/messages/

Formatting guide with examples (the C# version of the library currently doesn't implement all the formatters, but they will be added as needed):
https://messageformat.github.io/messageformat/page-guide

Language pluralization rules:
https://github.com/unicode-org/cldr/blob/master/common/supplemental/plurals.xml

Typically most strings are straight up replacements, with no complex syntax. Some use just a simple variable replacement (e.g. {name}), which will replace part of the message with given variable.

For cases when the structure of sentence changes based on the value of a number, you'll see pattern {variable, plural, ...}. Each language has a set of plural categories, like zero, one, two, few, many and other. Some languages have only "other", some (like English) have "one" and "other", while other languages have multiple. Make sure that you familiarize yourself with the plural categories in your language (using the links above) so you can correctly translate strings using this syntax.

Another common syntax is using the {variable, select, ...} form. This lets you match the variable against specific values and provide translated versions of each. You can either replace a single word (e.g. "Server status is {status, select, good {Good} bad {Not Good} }") or the whole sentence (e.g. "{status, select, good {Server is good!} bad {Oh no, servers are down!}}") depending on what works better in your language.

Please let us know if you have any questions or are unsure about certain things.

## Language codes and fallbacks
We uses the IETF language tags (https://en.wikipedia.org/wiki/IETF_language_tag) to load locales. These consist of a single primary language tag (typically two-letter language code from ISO 639-1 or a three-letter code from ISO 639-2 (1998), ISO 639-3 (2007) or ISO 639-5 (2008)) and and optional region sub-tag with country code.

When loading locale file, we will first check for the most specific locale file. Then it will load any missing strings from the general locale file and last it will load any missing strings from the English locale.

For example if your system locale is British English (en-gb), it will first look for "en-gb.json" file and then for the more general one "en.json".

We recommend putting most translations into the general language file (single two letter or three letter code) and if necessary only put specific overrides into the more specific language file. That way, most translations can be shared across variants of the language if possible.

Any strings you don't translate at all will also fall back into their English variants, so you don't have to worry about missing some of them, they can be translated later (or by another contributor). This also ensures that newly added strings in the English will show up and can be gradually translated as they come.

## CI
When you open a PR some tooling will run to validate that your JSON is valid and that your changes do not include any keys that are not found in the english file. If these checks fail, we cannot merge your PR so make sure to look into why the failure occurs.

## FAQ
### What if I find string that cannot be translated?
While majority of Resonite's UI has been converted to the localization system, there are likely a few stragglers and some parts that aren't translatable right now. If you encounter such place, create an Issue on this repository, ideally with screenshot of the non-translatable part, so we can convert it as well.

Currently there are a few known parts that cannot be translated, but are planned to be supported at a later date:
- Enumerations (e.g. certain tool options that cycle through several options)
- Component names and categories (component names will still show original for technical reasons, but will show optional translation for non-English languages)
- ProtoFlux node names and categories (same as above)
- Component fields (those will only show optional translated names on hover once tooltip system is implemented)

### Any other questions?
If you have questions or are unsure about something, you can create an Issue on this repository to ask a question