# Overview
This repository is a monorepo for managing the configuration and deployment of services managed by MIT Open Learning engineering. It uses a combination of Pulumi and PyInfra to build a pure-python deployment stack to enable more developer-friendly access to creating and modifying the systems that power the applications that we build and serve.

All infrastructure provisioning performed via Pulumi is located under `src/ol-infrastructure/` and all configuration management written in PyInfra lives under `src/bilder`.

# Infrastructure Management

## Getting Started
This is a Pulumi project, so the first step is to install the Pulumi CLI along with the relevant language
protocol. Instructions [here](https://www.pulumi.com/docs/get-started/install/).

To install and manage the relevant dependencies this project uses [Poetry](https://python-poetry.org/). After installing
the Poetry CLI simply run `poetry install`.

We use the S3 state backend for Pulumi, so after installing the Pulumi CLI run `pulumi login s3://mitol-pulumi-state`.

## Structure

This is a monorepo of the Pulumi code that we use to manage the infrastructure that powers MIT Open Learning

Component resources are located under `src/ol-infrastructure/components/` with a descriptive name that makes it evident what the intent of the
component is.

Concrete implementations of infrastructure are located under `src/ol-infrastructure/infrastructure/` with a descriptive name that makes it
evident what is contained in that project.

Management of resources that rely on foundational infrastructure being provisioned, but
which supports the operation of applications is located under
`src/ol-infrastructure/substructure`. An Example of this includes Consul prepared queries.

Provisioning of all of the resources needed to support and deploy a specific application
is located under `src/ol-infrastructure/applications/` with a name that indicates the
application being managed (e.g. `concourse`).

Each component or concrete infrastructure that is more complex than a single resource will include a `diagram.py` file
that uses the [diagrams](https://diagrams.mingrammer.com/) package to illustrate the system structure that it creates.

## Nomenclature

Pulumi organizes code into `Projects` which represent a deployable unit. Within a project they have a concept of
`Stacks` which are often used as a mapping for different environments. Each module underneath `src/ol-infrastructure/infrastructure/` and
`src/ol-infrastructure/applications/` is its own `Project`, meaning that it will have a `Pulumi.yaml` definition of that project. Each `stack`
has its own yaml file in which the configuration for that stack is defined.

## Conventions

Stack names should be a dot-separated namespaced representation of the project path, suffixed with an environment
specifier in the form of QA or Production. The capitalization is important as it will be used directly to interpolate
into tag objects. It is easier to start with QA and Production and then call `.lower()` than it is to build a dictionary
mapping the lowercase versions to their properly capitalized representation.

The dotted namespace allows for peaceful coexistence of multiple projects within a single state backend, as well as
allowing for use of [stack references](https://www.pulumi.com/docs/tutorials/aws/aws-py-stackreference/) between
projects.

The infrastructure components should be properly namespaced to match the stack names. For example, the project for
managing VPC networking in AWS is located at `src/ol-infrastructure/infrastructure/aws/network/` and the corresponding stacks are defined as
`aws.network.QA` and `aws.network.Production`.

## Executing

In order to run a deployment, you need to specify the project and the stack that you would like to deploy. From the root
of the repository, you can run `pulumi -C src/ol_infrastructure/path/to/module/ up`. If you haven't already selected the
stack, it will ask you to interactively select the stack which you are deploying.

## Adding a new Project

For each deployable unit of work we need to have a Pulumi project defined. The Pulumi CLI has a `new` command, but that
introduces extra files that we don't want. The minimum necessary work to signal that a given directory is a project is
the presence of a `Pulumi.yaml` file and a `__main__.py` where the deployment code is located. The contents of the
`Pulumi.yaml` file should follow this structure:

```
name: ol-infrastructure-dagster-application # Change the name here to be descriptive of the purpose of this project
runtime: python # not necessary to change
description: Pulumi project for deploying the stack of services needed by the Dagster ETL framework # update the description accordingly
backend:
  url: s3://mitol-pulumi-state/ # should not be changed
```

Create the directory path according to the conventions detailed above and then create these files. Once that is done you
will need to create the stack definitions (again according to the above conventions). To do this, `cd` to the target
directory and run the command `pulumi stack init --secrets-provider=awskms://alias/infrastructure-secrets-qa <your.dotted.stack.name.QA>`

Now you're ready to start writing the code that will define the target deployment.

# Configuration Management
In order to streamline the management of infrastructure required to support the services
run by MIT Open Learning we have implemented our configuration management logic using
PyInfra. This has the benefit of being pure Python, allowing us to take advantage of the
broad ecosystem that it provides including linting, testing, and integrations.

## Project Structure
Any given service is unlikely to be composed of a single service, instead requiring a
variety of components to be used together. In order to make the composition of a running
system easier to reason about, we decompose the responsibilities for any given piece of
technology into a self-contained component. These components live under the
`src/bilder/components/` directory. Each component provide at minimum the logic needed to
install and configure the associated service. In addition, each component should also
provide the logic needed to upgrade an instance of the service, the SystemD unit files
needed to register the service, and any logic required to perform maintenance on the
associated technology.

## Configuration Objects
A common requirement across all configuration management systems is a method to provide
settings at build/runtime to populate config files, determine which logical paths to
take, etc. A problem that often comes from managing these settings is uncertainty around
the constraints for each value, and the ability to easily merge default values with user
supplied information. In order to reduce the complexity of this situation and provide
useful constraints we use [Pydantic](https://pydantic-docs.helpmanual.io/) models with
relevant validators to contain the values needed by each component.

Each model defined for a given component will inherit from
`ol_configuration_management.lib.model_helpers.OLBaseSettings`. This class inherits from
the Pydantic `BaseSettings` class and sets the `case_sensitive` model setting to
`False`. This allows us to easily use environment variables to override attributes of a
model at build time.

## Directory Locations
When faced with a decision of where to locate any given files or directories we default
to following the [Filesystem Hierarchy
Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard). This typically
means that applications or services which are deployed by downloading an archive or
cloning a software repository should go under `/opt`, configuration files under `/etc/`,
and data files under `/var/lib`.


  Project Board |
  Community Site |
  #openverse @ Slack |
  Handbook |
  Design Handbook |
  Releases mockups |
  Design Library

  Openverse is a search engine for openly-licensed media.

# Openverse

Openverse is a powerful search engine for GPL-compatible images, audio, and
more. Openverse is live at [openverse.org](https://openverse.org).

- [Catalog](catalog/) | The Apache Airflow-powered system for downloading and
  storing Openverse's metadata
- [Ingestion server](ingestion_server/) | The mechanism for refreshing the data
  from the catalog to the API
- [API](api/) | The Django REST API for querying the database, used by the
  frontend
- [Frontend](frontend/) | The public search engine at
  [openverse.org](https://openverse.org), built with Vue and Nuxt
- [Automations](automations/) | Scripts used for various workflows around
  Openverse repositories and processes
- [Utilities](utilities/) | Scripts or utilities which are useful across
  multiple projects or don't necessarily fit into a specific project.

This repository also contains the following directories.

- [Brand](brand/) | Brand assets for Openverse such as logo and icon and
  guidelines for using these assets
- [Templates](templates/) | Jinja templates that can be rendered into common
  scaffolding code for the project

## Keep in touch

You can keep in touch with the project via the following channels:

- GitHub
  - [Issues](https://github.com/WordPress/openverse/issues/)
  - [PRs](https://github.com/WordPress/openverse/pulls/)
  - [Discussions](https://github.com/WordPress/openverse/discussions/)
  - [Project Board](https://github.com/orgs/WordPress/projects/3)
- [Community Site](https://make.wordpress.org/openverse/)
- #openverse channel in the
  [Making WordPress Chat](https://make.wordpress.org/chat/)
  - Weekly Development Chat (
    [Mondays @ 15:00 UTC](https://everytimezone.com/s/d2e71015))
  - Monthly Prioritisation Meeting (first Wednesday of every month @ 15:00 UTC)

## Documentation

To use the Openverse API, please refer to the
[API consumer documentation](https://api.openverse.org/v1/).

## Contributing

Pull requests are welcome! Feel free to
[join us on Slack](https://make.wordpress.org/chat/) and discuss the project
with the engineers and community members on #openverse.

You are welcome to take any open issue in the tracker labelled
[`help wanted`](https://github.com/WordPress/openverse/labels/help%20wanted) or
[`good first issue`](https://github.com/WordPress/openverse/labels/good%20first%20issue);
**there's no need to ask for permission in advance**. Other issues are open for
contribution as well, but may be less accessible or well-defined in comparison
to those that are explicitly labelled.

See the
[contribution guide](https://docs.openverse.org/general/contributing.html) for
details.

## Acknowledgments

Openverse, previously known as CC Search, was conceived and built at
[Creative Commons](https://creativecommons.org). We thank them for their
commitment to open source and openly licensed content, with particular thanks to
previous team members @ryanmerkley, @janetpkr, @lizadaly, @sebworks, @pa-w,
@kgodey, @annatuma, @mathemancer, @aldenstpage, @brenoferreira, and @sclachar,
along with their
[community of volunteers](https://opensource.creativecommons.org/community/community-team/).


# Introduction

The icefall project contains speech-related recipes for various datasets
using [k2-fsa](https://github.com/k2-fsa/k2) and [lhotse](https://github.com/lhotse-speech/lhotse).

You can use [sherpa](https://github.com/k2-fsa/sherpa), [sherpa-ncnn](https://github.com/k2-fsa/sherpa-ncnn) or [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) for deployment with models
in icefall; these frameworks also support models not included in icefall; please refer to respective documents for more details.

You can try pre-trained models from within your browser without the need
to download or install anything by visiting this [huggingface space](https://huggingface.co/spaces/k2-fsa/automatic-speech-recognition).
Please refer to [document](https://k2-fsa.github.io/icefall/huggingface/spaces.html) for more details.

# Installation

Please refer to [document](https://k2-fsa.github.io/icefall/installation/index.html)
for installation.

# Recipes

Please refer to [document](https://k2-fsa.github.io/icefall/recipes/index.html)
for more details.

## ASR: Automatic Speech Recognition

### Supported Datasets
  - [yesno][yesno]

  - [Aidatatang_200zh][aidatatang_200zh]
  - [Aishell][aishell]
  - [Aishell2][aishell2]
  - [Aishell4][aishell4]
  - [Alimeeting][alimeeting]
  - [AMI][ami]
  - [CommonVoice][commonvoice]
  - [Corpus of Spontaneous Japanese][csj]
  - [GigaSpeech][gigaspeech]
  - [LibriCSS][libricss]
  - [LibriSpeech][librispeech]
  - [Libriheavy][libriheavy]
  - [Multi-Dialect Broadcast News Arabic Speech Recognition][mgb2]
  - [PeopleSpeech][peoplespeech]
  - [SPGISpeech][spgispeech]
  - [Switchboard][swbd]
  - [TIMIT][timit]
  - [TED-LIUM3][tedlium3]
  - [TAL_CSASR][tal_csasr]
  - [Voxpopuli][voxpopuli]
  - [XBMU-AMDO31][xbmu-amdo31]
  - [WenetSpeech][wenetspeech]

More datasets will be added in the future.

### Supported Models

The [LibriSpeech][librispeech] recipe supports the most comprehensive set of models, you are welcome to try them out.

#### CTC
  - TDNN LSTM CTC
  - Conformer CTC
  - Zipformer CTC

#### MMI
  - Conformer MMI
  - Zipformer MMI

#### Transducer
  - Conformer-based Encoder
  - LSTM-based Encoder
  - Zipformer-based Encoder
  - LSTM-based Predictor
  - [Stateless Predictor](https://research.google/pubs/rnn-transducer-with-stateless-prediction-network/)

#### Whisper
  - [OpenAi Whisper](https://arxiv.org/abs/2212.04356) (We support fine-tuning on AiShell-1.)

If you are willing to contribute to icefall, please refer to [contributing](https://k2-fsa.github.io/icefall/contributing/index.html) for more details.

We would like to highlight the performance of some of the recipes here.

### [yesno][yesno]

This is the simplest ASR recipe in `icefall` and can be run on CPU.
Training takes less than 30 seconds and gives you the following WER:

```
[test_set] %WER 0.42% [1 / 240, 0 ins, 1 del, 0 sub ]
```
We provide a Colab notebook for this recipe: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tIjjzaJc3IvGyKiMCDWO-TSnBgkcuN3B?usp=sharing)

### [LibriSpeech][librispeech]

Please see [RESULTS.md](https://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/RESULTS.md)
for the **latest** results.

#### [Conformer CTC](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/conformer_ctc)

|     | test-clean | test-other |
|-----|------------|------------|
| WER | 2.42       | 5.73       |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1huyupXAcHsUrKaWfI83iMEJ6J0Nh0213?usp=sharing)

#### [TDNN LSTM CTC](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/tdnn_lstm_ctc)

|     | test-clean | test-other |
|-----|------------|------------|
| WER | 6.59       | 17.69      |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1-iSfQMp2So-We_Uu49N4AAcMInB72u9z?usp=sharing)

#### [Transducer (Conformer Encoder + LSTM Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/transducer)

|               | test-clean | test-other |
|---------------|------------|------------|
| greedy_search | 3.07       | 7.51       |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_u6yK9jDkPwG_NLrZMN2XK7Aeq4suMO2?usp=sharing)

#### [Transducer (Conformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/transducer)

|                                       | test-clean | test-other |
|---------------------------------------|------------|------------|
| modified_beam_search (`beam_size=4`) | 2.56       | 6.27       |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CO1bXJ-2khDckZIW8zjOPHGSKLHpTDlp?usp=sharing)

#### [Transducer (Zipformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/librispeech/ASR/zipformer)

WER (modified_beam_search `beam_size=4` unless further stated)

1. LibriSpeech-960hr

| Encoder         | Params | test-clean | test-other | epochs  | devices    |
|-----------------|--------|------------|------------|---------|------------|
| Zipformer       | 65.5M  | 2.21       | 4.79       | 50      | 4 32G-V100 |
| Zipformer-small | 23.2M  | 2.42       | 5.73       | 50      | 2 32G-V100 |
| Zipformer-large | 148.4M | 2.06       | 4.63       | 50      | 4 32G-V100 |
| Zipformer-large | 148.4M | 2.00       | 4.38       | 174     | 8 80G-A100 |

2. LibriSpeech-960hr + GigaSpeech

| Encoder         | Params | test-clean | test-other |
|-----------------|--------|------------|------------|
| Zipformer       | 65.5M   | 1.78       | 4.08       |

3. LibriSpeech-960hr + GigaSpeech + CommonVoice

| Encoder         | Params | test-clean | test-other |
|-----------------|--------|------------|------------|
| Zipformer       | 65.5M   | 1.90       | 3.98       |

### [GigaSpeech][gigaspeech]

#### [Conformer CTC](https://github.com/k2-fsa/icefall/tree/master/egs/gigaspeech/ASR/conformer_ctc)

|     |  Dev  | Test  |
|-----|-------|-------|
| WER | 10.47 | 10.58 |

#### [Transducer (pruned_transducer_stateless2)](https://github.com/k2-fsa/icefall/tree/master/egs/gigaspeech/ASR/pruned_transducer_stateless2)

Conformer Encoder + Stateless Predictor + k2 Pruned RNN-T Loss

|                      |  Dev  | Test  |
|----------------------|-------|-------|
|    greedy_search     | 10.51 | 10.73 |
|   fast_beam_search   | 10.50 | 10.69 |
| modified_beam_search | 10.40 | 10.51 |

#### [Transducer (Zipformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/gigaspeech/ASR/zipformer)

|                      |  Dev  | Test  |
|----------------------|-------|-------|
|    greedy_search     | 10.31 | 10.50 |
|   fast_beam_search   | 10.26 | 10.48 |
| modified_beam_search | 10.25 | 10.38 |

### [Aishell][aishell]

#### [TDNN LSTM CTC](https://github.com/k2-fsa/icefall/tree/master/egs/aishell/ASR/tdnn_lstm_ctc)

|     | test  |
|-----|-------|
| CER | 10.16 |

We provide a Colab notebook to test the pre-trained model:  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1jbyzYq3ytm6j2nlEt-diQm-6QVWyDDEa?usp=sharing)

#### [Transducer (Conformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/aishell/ASR/transducer_stateless)

|     | test |
|-----|------|
| CER | 4.38 |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/14XaT2MhnBkK-3_RqqWq3K90Xlbin-GZC?usp=sharing)

#### [Transducer (Zipformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/aishell/ASR/zipformer)

WER (modified_beam_search `beam_size=4`)

| Encoder         | Params | dev | test | epochs  |
|-----------------|--------|-----|------|---------|
| Zipformer       | 73.4M  | 4.13| 4.40 | 55      |
| Zipformer-small | 30.2M  | 4.40| 4.67 | 55      |
| Zipformer-large | 157.3M | 4.03| 4.28 | 56      |

### [Aishell4][aishell4]

#### [Transducer (pruned_transducer_stateless5)](https://github.com/k2-fsa/icefall/tree/master/egs/aishell4/ASR/pruned_transducer_stateless5)

1 Trained with all subsets:
|     |   test     |
|-----|------------|
| CER |   29.08    |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1z3lkURVv9M7uTiIgf3Np9IntMHEknaks?usp=sharing)

### [TIMIT][timit]

#### [TDNN LSTM CTC](https://github.com/k2-fsa/icefall/tree/master/egs/timit/ASR/tdnn_lstm_ctc)

|   |TEST|
|---|----|
|PER| 19.71% |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Hs9DA4V96uapw_30uNp32OMJgkuR5VVd?usp=sharing)

#### [TDNN LiGRU CTC](https://github.com/k2-fsa/icefall/tree/master/egs/timit/ASR/tdnn_ligru_ctc)

|   |TEST|
|---|----|
|PER| 17.66% |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1z3lkURVv9M7uTiIgf3Np9IntMHEknaks?usp=sharing)

### [TED-LIUM3][tedlium3]

#### [Transducer (Conformer Encoder + Stateless Predictor)](https://github.com/k2-fsa/icefall/tree/master/egs/tedlium3/ASR/transducer_stateless)

|                                      |  dev  |  test  |
|--------------------------------------|-------|--------|
| modified_beam_search (`beam_size=4`) |  6.91 |  6.33  |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MmY5bBxwvKLNT4A2DJnwiqRXhdchUqPN?usp=sharing)

#### [Transducer (pruned_transducer_stateless)](https://github.com/k2-fsa/icefall/tree/master/egs/tedlium3/ASR/pruned_transducer_stateless)

|                                      |  dev  |  test  |
|--------------------------------------|-------|--------|
| modified_beam_search (`beam_size=4`) |  6.77 |  6.14  |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1je_1zGrOkGVVd4WLzgkXRHxl-I27yWtz?usp=sharing)

### [Aidatatang_200zh][aidatatang_200zh]

#### [Transducer (pruned_transducer_stateless2)](https://github.com/k2-fsa/icefall/tree/master/egs/aidatatang_200zh/ASR/pruned_transducer_stateless2)

|                      |  Dev  | Test  |
|----------------------|-------|-------|
|    greedy_search     | 5.53  | 6.59  |
|   fast_beam_search   | 5.30  | 6.34  |
| modified_beam_search | 5.27  | 6.33  |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wNSnSj3T5oOctbh5IGCa393gKOoQw2GH?usp=sharing)

### [WenetSpeech][wenetspeech]

#### [Transducer (pruned_transducer_stateless2)](https://github.com/k2-fsa/icefall/tree/master/egs/wenetspeech/ASR/pruned_transducer_stateless2)

|                      |  Dev  | Test-Net | Test-Meeting |
|----------------------|-------|----------|--------------|
|    greedy_search     | 7.80  |  8.75    |  13.49       |
|   fast_beam_search   | 7.94  |  8.74    |  13.80       |
| modified_beam_search | 7.76  |  8.71    |  13.41       |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1EV4e1CHa1GZgEF-bZgizqI9RyFFehIiN?usp=sharing)

#### [Transducer **Streaming** (pruned_transducer_stateless5) ](https://github.com/k2-fsa/icefall/tree/master/egs/wenetspeech/ASR/pruned_transducer_stateless5)

|                      |  Dev  | Test-Net | Test-Meeting |
|----------------------|-------|----------|--------------|
| greedy_search | 8.78 | 10.12 | 16.16 |
| fast_beam_search| 9.01 | 10.47 | 16.28 |
| modified_beam_search | 8.53| 9.95 | 15.81 |

### [Alimeeting][alimeeting]

#### [Transducer (pruned_transducer_stateless2)](https://github.com/k2-fsa/icefall/tree/master/egs/alimeeting/ASR/pruned_transducer_stateless2)

|                      |  Eval  | Test-Net |
|----------------------|--------|----------|
|    greedy_search     | 31.77  |  34.66   |
|   fast_beam_search   | 31.39  |  33.02   |
| modified_beam_search | 30.38  |  34.25   |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tKr3f0mL17uO_ljdHGKtR7HOmthYHwJG?usp=sharing)

### [TAL_CSASR][tal_csasr]

#### [Transducer (pruned_transducer_stateless5)](https://github.com/k2-fsa/icefall/tree/master/egs/tal_csasr/ASR/pruned_transducer_stateless5)

The best results for Chinese CER(%) and English WER(%) respectively (zh: Chinese, en: English):
|decoding-method | dev | dev_zh | dev_en | test | test_zh | test_en |
|--|--|--|--|--|--|--|
|greedy_search| 7.30 | 6.48 | 19.19 |7.39| 6.66 | 19.13|
|fast_beam_search| 7.18 | 6.39| 18.90 |  7.27| 6.55 | 18.77|
|modified_beam_search| 7.15 | 6.35 | 18.95 | 7.22| 6.50 | 18.70 |

We provide a Colab notebook to test the pre-trained model: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1DmIx-NloI1CMU5GdZrlse7TRu4y3Dpf8?usp=sharing)

## TTS: Text-to-Speech

### Supported Datasets

  - [LJSpeech][ljspeech]
  - [VCTK][vctk]

### Supported Models

  - [VITS](https://arxiv.org/abs/2106.06103)

# Deployment with C++

Once you have trained a model in icefall, you may want to deploy it with C++ without Python dependencies.

Please refer to

  - https://k2-fsa.github.io/icefall/model-export/export-with-torch-jit-script.html
  - https://k2-fsa.github.io/icefall/model-export/export-onnx.html
  - https://k2-fsa.github.io/icefall/model-export/export-ncnn.html

for how to do this.

We also provide a Colab notebook, showing you how to run a torch scripted model in [k2][k2] with C++.
Please see: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1BIGLWzS36isskMXHKcqC9ysN6pspYXs_?usp=sharing)

[yesno]: egs/yesno/ASR
[librispeech]: egs/librispeech/ASR
[aishell]: egs/aishell/ASR
[aishell2]: egs/aishell2/ASR
[aishell4]: egs/aishell4/ASR
[timit]: egs/timit/ASR
[tedlium3]: egs/tedlium3/ASR
[gigaspeech]: egs/gigaspeech/ASR
[aidatatang_200zh]: egs/aidatatang_200zh/ASR
[wenetspeech]: egs/wenetspeech/ASR
[alimeeting]: egs/alimeeting/ASR
[tal_csasr]: egs/tal_csasr/ASR
[ami]: egs/ami
[swbd]: egs/swbd/ASR
[k2]: https://github.com/k2-fsa/k2
[commonvoice]: egs/commonvoice/ASR
[csj]: egs/csj/ASR
[libricss]: egs/libricss/SURT
[libriheavy]: egs/libriheavy/ASR
[mgb2]: egs/mgb2/ASR
[peoplespeech]: egs/peoplespeech/ASR
[spgispeech]: egs/spgispeech/ASR
[voxpopuli]: egs/voxpopuli/ASR
[xbmu-amdo31]: egs/xbmu-amdo31/ASR

[vctk]: egs/vctk/TTS
[ljspeech]: egs/ljspeech/TTS


# Scan and prepare your document for [Paperless](https://github.com/paperless-ngx/paperless-ngx)

The main goal of this project is to have some productive process from the document scanning to
[Paperless](https://github.com/paperless-ngx/paperless-ngx).
For that we need to prepare the documents some tools that need many resources, then the idea to do it
in the background and ideally on another host like a NAS.
A consequence of that it's a not easy to put it in place, but then you will be really productive.
The interface between the user and the process is the `scan` command to do the initial scan, and the file system
to verify that the result is OK (and do some advance operations describe below) and validate it.

## Features

- Scan the images optionally by using the Automatic Document Feeder
- Easily scan double-sided images using the Automatic Document Feeder
- Extract the DPI from the TIFF images
- Change the images levels
- Remove the area out of the image
- Deskew the images
- Crop the images
- Sharpen the images (disable by default)
- Dither the images (disable by default)
- Auto rotate the images by using tesseract (To have the text on the right side)
- Optimize the images using `pngquant`, `optipng`, `ps2pdf` or `jpeg` (using quality from GraphicsMagick convert)
- Assisted split, used to split a prospectus page in more pages (Requires to modify the YAML...)
- Append credit cart, used to have the two faces of a credit cart on the same page
- Be able to copy the OCR result from the PDF
- Scan the QR code and Bar code and add a new page with the values (separate process)
- Manage the empty lines in the QR code (replace by a pipe (`|`) in the PDF,
  and run `scan --convert-clipboard` to scan your clipboard to do the inverse transform)

## Requirements

On the desktop:

- [Python](https://www.python.org/) >= 3.6
- The [scanimage](http://www.sane-project.org/) command, on Windows it should be able to use another command,
  but it's never be tested.
  This command would be an adapter that interpret the following arguments:
  `--batch`, `--batch-start`, `--batch-increment`, `--batch-count`,
  `--batch` for the destination file name template (`%d` is replaced by the page number),
  and the others for the `auto_bash`.

On the NAS:

- [Docker](https://www.docker.com/)

## Install

Scan-to-paperless requires a desktop and a server part, the two parts communicate through the scan folder.

The server part is where the document were processed, and the desktop part is
from where we want we will scan the document, on witch one the scanner is connected.

The scan folder should be synchronized between the desktop and the server,
I use [Syncthing](https://syncthing.net/) for that.

### On the desktop

```bash
$ python3 -m pip install scan-to-paperless
$ echo PATH=$PATH:~/venv/bin >> ~/.bashrc
$ echo source <(register-python-argcomplete scan) >> ~/.bashrc
$ echo source <(register-python-argcomplete scan-progress-status) >> ~/.bashrc
```

Create the configuration file on `<home_config>/scan-to-paperless.yaml` (on Linux it's `~/.config/scan-to-paperless.yaml`), with:

```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/sbrunner/scan-to-paperless/master/scan_to_paperless/config_schema.json

scan_folder: /home/sbrunner/Paperless/scan/
scanimage_arguments: # Additional argument passed to the scanimage command
  - --device=... # Use `scanimage --list` to get the possible values
  - --format=png
  - --mode=color
  - --resolution=300
default_args:
  auto_mask: {}
  auto_cut: {}
  run_pngquant: true
  cut_white: 200 # cut the near white color to have a uniform background
  dpi: 300 # Not necessary if the scanner generate a tiff file
  tesseract_lang: fra+eng # The used languages for the OCR
```

[Full config documentation](./config.md)

### On the NAS

The Docker support is required, Personally I use a [Synology DiskStation DS918+](https://www.synology.com/products/DS918+),
and you can get the \*.syno.json files to configure your Docker services.

Otherwise, use:

```bash
SCAN_FOLDER=<scan_folder>
CONSUME_FOLDER=<consume_folder>
docker run --name=scan-to-paperless --restart=unless-stopped --detach \
  --volume=${SCAN_FOLDER}:/source \
  --volume=${CONSUME_FOLDER}:/destination \
  sbrunner/scan-to-paperless
```

You can set the environment variable `PROGRESS` to `TRUE` to get all the intermediate images.

To stop run:

```bash
docker stop scan-to-paperless
docker rm scan-to-paperless
```

### Repertory link

You should find a way to synchronize or using sharing to link the scan folder on your desktop and on your NAS.

You should also link to consume folder to `paperless-ngx` probably just by using the same folder.

## Usage

1. Use the `scan` command to import your document, to scan your documents.

2. The document is transferred to your NAS (I use [Syncthing](https://syncthing.net/)).

3. The documents will be processed on the NAS.

4. Use `scan-process-status` to know the status of your documents.

5. Validate your documents.

6. If your happy with that remove the `REMOVE_TO_CONTINUE` file.
   (To restart the process remove one of the generated images, to cancel the job just remove the folder).

7. The process will continue his job and import the document in `paperless-ngx`.

## Job config file

In the `config.yaml` file present in the document folder, you can find some information generated during
the processing and some can be modified.

E.g. you can modify an image angle to fix the skew, then remove a generated image for force to regenerate
the images.

[Full job config documentation](./process.md)

## Advance feature

### Add a mask

If your scanner add some margin around the scanned image it will relay case some issue the skew and the
content detection.

To solve that you can add a black and white image named `mask.png` in the root folder and draw in black the
part that should not be taken in account.

Scan to Paperless is also able to create a mask automatically, to enable is with the default configuration,
just add `args` name `auto_mask` with an empty dictionary (`{}`).

See also: [The documentation](./config.md#definitions)

Configuration note:

By default, the options `lower_hsv_color` and `upper_hsv_color` select the page (white).
Yon can also select the scanner background, for that you also should set the option `inverse_mask` to `true`
and the option `de_noise_morphology` to `false`.

### Mask the image

If your scanner add some margin around the scanned image you can definitively mask them.

To solve that you can add a black and white image named `cut.png` in the root folder and draw in black the
part that should not be taken in account.

Scan to Paperless is also able to create a mask automatically, to enable is with the default configuration,
just add `args` name `auto_cut` with an empty dictionary (`{}`).

See also: [The documentation](./config.md#definitions)

### Double sized scanning

1. Pour your sheets on the Automatic Document Feeder.

2. Run `scan` with the option `--mode=double`.

3. Press enter to start scanning the first side of all sheets.

4. Put again all your sheets on the Automatic Document Feeder without turning them.

The scan utils will rotate and reorder all the sheets to get a good document.

### Credit card scanning

The options `--append-credit-card` will append all the sheets vertically to have the booth face of the credit card on the same page.

### Assisted split

1. Do your scan as usual with the extra option `--assisted-split`.

2. After the process do his first pass you will have images with lines and numbers.
   The lines represent the detected potential split of the image, the length indicate the strength of the detection.
   In your config you will have something like:

```yaml
assisted_split:
-   destinations:
    -   4 # Page number of the left part of the image
    -   1 # Same for the right page of the image
        image: image-1.png # name of the image
        limits:
    -   margin: 0 # Margin around the split
        name: 0 # Number visible on the generated image
        value: 375 # The position of the split (can be manually edited)
        vertical: true # Will split the image vertically
    -   ...
        source: /source/975468/7-assisted-split/image-1.png
-   ...

```

Edit your config file, you should have one more destination than the limits.
If you put destination like that: 2.1, it means that it will be the first part of the page 2 and the 2.2 will be the second part.

3. Delete the file `REMOVE_TO_CONTINUE`.

4. After the process do his first pass you will have the final generated images.

5. If it's OK delete the file `REMOVE_TO_CONTINUE`.

## The scan modes configuration

First of all the `scanimage` command and arguments can be configured with the `scanimage` and
`scanimage_argumentss` options in the configuration file (`~/.config/scan-to-paperless.yaml`).

In this file there is also a `modes` section that can configure each modes.

See also: [The documentation](./config.md)

## Extends an existing configuration

To create the `preset` configuration file it can be useful to extends an existing configuration.
For that you can use the `extends` (and `merge_strategies`) option in the configuration file.

See also: [The documentation](./config.md)

## Server configuration

Environment variable:

- `SCAN_SOURCE_FOLDER`: The main input folder for the scan process.
- `SCAN_CODES_FOLDER`: The input folder for the codes (QR code ad Barcode) detection and add a new page.
- `SCAN_FINAL_FOLDER`: The final folder for the scan process.
- `SCAN_CODES_DPI`: The used DPI to decode the codes.
- `SCAN_CODES_PDF_DPI`: The used PDF DPI to create the codes document.
- `SCAN_CODES_FONT_NAME`: The used font of code number.
- `SCAN_CODES_FONT_SIZE`: The used font size of code number.
- `SCAN_CODES_MARGIN_TOP`: The top margin of code number.
- `SCAN_CODES_MARGIN_LEFT`:The left margin of code number.
- `TIME`: Print the elapsed time.
- `PROGRESS`: Save some intermediate files, don't clean the folder at the end.

## Contributing

Install the pre-commit hooks:

```bash
pip install pre-commit
pre-commit install --allow-missing-config
```


  VKBottle

    Кастомизируемый, быстрый и удобный фреймворк для работы с VK API

## Hello World

```python
from vkbottle.bot import Bot

bot = Bot("GroupToken")

@bot.on.message()
async def handler(_) -> str:
    return "Hello world!"

bot.run_forever()
```

[Смотреть больше примеров!](https://github.com/vkbottle/vkbottle/tree/master/examples)

## Документация

[Туториал для новичков](https://vkbottle.rtfd.io/ru/latest/tutorial/)\
[Техническая документация](https://vkbottle.rtfd.io/ru/latest)

## Установка

Установить новейшую версию можно командой:

```shell
pip install vkbottle
```

Если вы ищете старые версии:

- [`3.x`](https://github.com/vkbottle/vkbottle/tree/v3.0)
- [`2.x`](https://github.com/vkbottle/vkbottle/tree/v2.0)

## Contributing

ПР поддерживаются! Перед созданием пулл реквеста ознакомьтесь с [CONTRIBUTION_GUIDE](CONTRIBUTION_GUIDE.md). Нам приятно видеть ваш вклад в развитие фреймворка.\
Задавайте вопросы в блоке Issues или на нашем [**форуме в Telegram**](https://t.me/botoforum)!

Мейнтейнеры: [click](https://github.com/vkbottle/vkbottle/graphs/contributors)

## Лицензия

Copyright © 2019-2023 [timoniq](https://github.com/timoniq).\
Copyright © 2022 [FeeeeK](https://github.com/FeeeeK).\
Этот проект имеет [MIT](https://github.com/vkbottle/vkbottle/blob/master/LICENSE) лицензию.


# About

Source for Google Click to Deploy solutions listed on Google Cloud Marketplace.

# Disclaimer

This is not an officially supported Google product.

# :warning: About log4j Vulnerabilities

We are constantly working on updating the scripts used in this repository to use non-impacted versions of the OSS that are used here. So updates are constanly being made in this repository.

Below is the list of solutions, in this repo, currently affected by the log4j CVEs, or using a log4j version < 2.16

## Virtual Machines

| Solution | Notes |
| --- | --- |
| elasticsearch | Vulnerable |
| logstash | Vulnerable |
| magento | Vulnerable (uses elasticsearch) |
| sonarqube | Vulnerable (uses elasticsearch) |
| liferay | Vulnerable |

## Kubernetes Applications

| Solution | Notes |
| --- | --- |
| activemq | Uses log4j 1.x |
| elastic-gke-logging | Vulnerable |
| elasticsearch | Vulnerable |
| kafka | Uses log4j 1.x |
| magento |  Vulnerable (uses elasticsearch) |
| sonarqube | Vulnerable (uses elasticsearch) |
| zookeeper | Vulnerable (uses elasticsearch) |

# Cloud Build CI

This repository uses Cloud Build for continuous integration. Each type of application has its own configuration file.

For detailed information on each configuration, see the following documentations:

*   [Docker images](docker/README.md#cloud-build-ci)
*   [K8s applications](k8s/README.md#cloud-build-ci)
*   [VM applications](vm/README.md#cloud-build-ci)

## GCB custom worker pools

The Cloud Build configurations use Google Cloud Build (GCB) custom worker pools.

If you want to create a new worker pool, run the following command:

```shell
gcloud beta builds worker-pools create gcb-workers-pool-e2 \
  --project=[PROJECT_ID] \
  --peered-network=projects/[NETWORK_PROJECT_NUMBER]/global/networks/default \
  --region=us-central1 \
  --worker-machine-type=e2-standard-2
```

Where:

*   `[PROJECT_ID]` is the GCP project ID where you want to create your custom worker pool.
*   `[NETWORK_PROJECT_NUMBER]` is the project number of the Cloud project that holds your VPC network.

For more information, see the
[gcloud beta builds worker-pools commands](https://cloud.google.com/sdk/gcloud/reference/beta/builds/worker-pools/).


Create an internal MLOps platform for your entire machine learning team.

  [![PyPi][pypi-shield]][pypi-url]
  [![PyPi][pypiversion-shield]][pypi-url]
  [![PyPi][downloads-shield]][downloads-url]
  [![Contributors][contributors-shield]][contributors-url]
  [![License][license-shield]][license-url]

[pypi-shield]: https://img.shields.io/pypi/pyversions/zenml?color=281158

[pypi-url]: https://pypi.org/project/zenml/

[pypiversion-shield]: https://img.shields.io/pypi/v/zenml?color=361776

[downloads-shield]: https://img.shields.io/pypi/dm/zenml?color=431D93

[downloads-url]: https://pypi.org/project/zenml/

[codecov-shield]: https://img.shields.io/codecov/c/gh/zenml-io/zenml?color=7A3EF4

[codecov-url]: https://codecov.io/gh/zenml-io/zenml

[contributors-shield]: https://img.shields.io/github/contributors/zenml-io/zenml?color=7A3EF4

[contributors-url]: https://github.com/zenml-io/zenml/graphs/contributors

[license-shield]: https://img.shields.io/github/license/zenml-io/zenml?color=9565F6

[license-url]: https://github.com/zenml-io/zenml/blob/main/LICENSE

[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[linkedin-url]: https://www.linkedin.com/company/zenml/

[twitter-shield]: https://img.shields.io/twitter/follow/zenml_io?style=for-the-badge

[twitter-url]: https://twitter.com/zenml_io

[slack-shield]: https://img.shields.io/badge/-Slack-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[slack-url]: https://zenml.io/slack-invite

[build-shield]: https://img.shields.io/github/workflow/status/zenml-io/zenml/Build,%20Lint,%20Unit%20&%20Integration%20Test/develop?logo=github&style=for-the-badge

[build-url]: https://github.com/zenml-io/zenml/actions/workflows/ci.yml

---

## 🤸 Quickstart
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/zenml-io/zenml/blob/main/examples/quickstart/quickstart.ipynb)

[Install ZenML](https://docs.zenml.io/getting-started/installation) via [PyPI](https://pypi.org/project/zenml/). Python 3.8 - 3.11 is required:

```bash
pip install "zenml[server]" notebook
```

Take a tour with the guided quickstart by running:

```bash
zenml go
```

## 🪄 Simple, integrated, End-to-end MLOps

### Create machine learning pipelines with minimal code changes

ZenML is a MLOps framework intended for data scientists or ML engineers looking to standardize machine learning practices. Just add `@step` and `@pipeline` to your existing Python functions to get going. Here is a toy example:

```python
from zenml import pipeline, step

@step  # Just add this decorator
def load_data() -> dict:
    training_data = [[1, 2], [3, 4], [5, 6]]
    labels = [0, 1, 0]
    return {'features': training_data, 'labels': labels}

@step
def train_model(data: dict) -> None:
    total_features = sum(map(sum, data['features']))
    total_labels = sum(data['labels'])

    print(f"Trained model using {len(data['features'])} data points. "
          f"Feature sum is {total_features}, label sum is {total_labels}")

@pipeline  # This function combines steps together
def simple_ml_pipeline():
    dataset = load_data()
    train_model(dataset)

if __name__ == "__main__":
    run = simple_ml_pipeline()  # call this to run the pipeline

```

![Running a ZenML pipeline](/docs/book/.gitbook/assets/readme_basic_pipeline.gif)

### Deploy workloads easily on your production infrastructure

The framework is a gentle entry point for practitioners to build complex ML pipelines with little knowledge required of the underlying infrastructure complexity. ZenML pipelines can be run on AWS, GCP, Azure, Airflow, Kubeflow and even on Kubernetes without having to change any code or know underlying internals.

```python
from zenml.config import ResourceSettings, DockerSettings

@step(
  settings={
    "resources": ResourceSettings(memory="16GB", gpu_count="1", cpu_count="8"),
    "docker": DockerSettings(parent_image="pytorch/pytorch:1.12.1-cuda11.3-cudnn8-runtime")
  }
)
def training(...):
	...
```

```bash
zenml stack set k8s  # Set a stack with kubernetes orchestrator
python run.py
```

![Workloads with ZenML](/docs/book/.gitbook/assets/readme_compute.gif)

### Track models, pipeline, and artifacts

Create a complete lineage of who, where, and what data and models are produced.

You’ll be able to find out who produced which model, at what time, with which data, and on which version of the code. This guarantees full reproducibility and auditability.

```python
from zenml import Model

@step(model=Model(name="classification"))
def trainer(training_df: pd.DataFrame) -> Annotated["model", torch.nn.Module]:
	...
```

![Exploring ZenML Models](/docs/book/.gitbook/assets/readme_mcp.gif)

### Purpose built for machine learning with integration to you favorite tools

While ZenML brings a lot of value of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.

```python
from bentoml._internal.bento import bento

@step(on_failure=alert_slack, experiment_tracker="mlflow")
def train_and_deploy(training_df: pd.DataFrame) -> bento.Bento
	mlflow.autolog()
	...
	return bento
```

![Exploring ZenML Integrations](/docs/book/.gitbook/assets/readme_integrations.gif)

## 🖼 Learning

The best way to learn about ZenML is the [docs](https://docs.zenml.io/). We recommend beginning with the [Starter Guide](https://docs.zenml.io/user-guide/starter-guide) to get up and running quickly.

If you are a visual learner, this 11-minute video tutorial is also a great start:

[![Introductory Youtube Video](docs/book/.gitbook/assets/readme_youtube_thumbnail.png)](https://www.youtube.com/watch?v=wEVwIkDvUPs)

And finally, here are some other examples and use cases for inspiration:

1. [E2E Batch Inference](examples/e2e/): Feature engineering, training, and inference pipelines for tabular machine learning.
2. [Basic NLP with BERT](examples/e2e_nlp/): Feature engineering, training, and inference focused on NLP.
3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents): Using Langchain to create a simple RAG pipeline.
4. [Huggingface Model to Sagemaker Endpoint](https://github.com/zenml-io/zenml-projects/tree/main/huggingface-sagemaker): Automated MLOps on Amazon Sagemaker and HuggingFace
5. [LLMops](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide): Complete guide to do LLM with ZenML

## 🔋 Deploy ZenML

For full functionality ZenML should be deployed on the cloud to
enable collaborative features as the central MLOps interface for teams.

Currently, there are two main ways to deploy ZenML:

- **ZenML Cloud**: With [ZenML Cloud](cloud.zenml.io/?utm_source=readme&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link),
you can make use of a control plane to create ZenML servers, also known as tenants.
These tenants are managed and maintained by ZenML’s dedicated team, alleviating
the burden of server management from your end.
- **Self-hosted deployment**: Alternatively, you have the flexibility to [deploy
ZenML on your own self-hosted environment](https://docs.zenml.io/getting-started/deploying-zenml#deploying-a-zenml-server).
This can be achieved through various methods, including using our CLI, Docker,
Helm, or HuggingFace Spaces.

## Use ZenML with VS Code

ZenML has a [VS Code extension](https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode) that allows you to inspect your stacks and pipeline runs directly from your editor. The extension also allows you to switch your stacks without needing to type any CLI commands.

  🖥 VS Code Extension in Action!

## 🗺 Roadmap

ZenML is being built in public. The [roadmap](https://zenml.io/roadmap) is a regularly updated source of truth for the ZenML community to understand where the product is going in the short, medium, and long term.

ZenML is managed by a [core team](https://zenml.io/company) of developers that are responsible for making key decisions and incorporating feedback from the community. The team oversees feedback via various channels,
and you can directly influence the roadmap as follows:

- Vote on your most wanted feature on our [Discussion
board](https://zenml.io/discussion).
- Start a thread in our [Slack channel](https://zenml.io/slack).
- [Create an issue](https://github.com/zenml-io/zenml/issues/new/choose) on our GitHub repo.

## 🙌 Contributing and Community

We would love to develop ZenML together with our community! The best way to get
started is to select any issue from the `[good-first-issue`
label](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22)
and open up a Pull Request!

If you
would like to contribute, please review our [Contributing
Guide](CONTRIBUTING.md) for all relevant details.

## 🆘 Getting Help

The first point of call should
be [our Slack group](https://zenml.io/slack-invite/).
Ask your questions about bugs or specific use cases, and someone from
the [core team](https://zenml.io/company) will respond.
Or, if you
prefer, [open an issue](https://github.com/zenml-io/zenml/issues/new/choose) on
our GitHub repo.

## 📜 License

ZenML is distributed under the terms of the Apache License Version 2.0.
A complete version of the license is available in the [LICENSE](LICENSE) file in
this repository. Any contribution made to this project will be licensed under
the Apache License Version 2.0.

      Join our

    Slack Community  and be part of the ZenML family.

    Features
    ·
    Roadmap
    ·
    Report Bug
    ·
    Sign up for Cloud
    ·
    Read Blog
    ·
    Contribute to Open Source
    ·
    Projects Showcase

    🎉 Version 0.58.1 is out. Check out the release notes
    here.

    🖥 Download our VS Code Extension here.


Welcome to the HADDOCK3-Beta version.

The `main` branch represents the latest state of HADDOCK v3. Currently,
stable beta version.

[![unit tests](https://github.com/haddocking/haddock3/workflows/tests/badge.svg?branch=main)](https://github.com/haddocking/haddock3/actions?workflow=tests)
[![build](https://github.com/haddocking/haddock3/workflows/build/badge.svg?branch=main)](https://github.com/haddocking/haddock3/actions?workflow=build)
[![docs](https://github.com/haddocking/haddock3/workflows/pages/badge.svg?branch=main)](https://github.com/haddocking/haddock3/actions?workflow=pages)

[![Codacy Badge](https://app.codacy.com/project/badge/Grade/e11e7f45400f4e8589cdf5941f95233a)](https://app.codacy.com/gh/haddocking/haddock3/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/e11e7f45400f4e8589cdf5941f95233a)](https://app.codacy.com/gh/haddocking/haddock3/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)

[![fair-software.eu](https://img.shields.io/badge/fair--software.eu-%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F%20%20%E2%97%8F-green)](https://fair-software.eu)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/8844/badge)](https://www.bestpractices.dev/projects/8844)

[![Research Software Directory](https://img.shields.io/badge/rsd-haddock3-00a3e3.svg)](https://research-software-directory.org/software/haddock3)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10527751.svg)](https://doi.org/10.5281/zenodo.10527751)

* * *

# HADDOCK3

## 1. Installation

To install HADDOCK3 follow the instructions in the [INSTALL](docs/INSTALL.md)
file.

## 2. Documentation

HADDOCK3-beta documentation is hosted online at https://www.bonvinlab.org/haddock3/.
The documentation is rendered and update at every commit to the `main` branch. So,
you will always find the latest version in the link above.

If you want to compile the HADDOCK3 documentation pages locally (offline),
install HADDOCK3 and activate the `haddock3` python environment as explained in
the [installation instructions](docs/INSTALL.md). Then, in your terminal
window, run:

```bash
tox -e docs
```

*Ignore any warning messages.* After, use your favorite browser to open the
file `haddock3-docs/index.html`. This will open a local webpage with the
complete HADDOCK3 documentation. Exactly the same you will find online.
Navigate around, enjoy, and contribute.

## 3. Examples

### 3.1. Basic scoring of an ensemble of 5 structures:

In the `examples/` folder you find several examples for you to test and
learn HADDOCK3. Additional information is in the documentation pages.

```bash
cd examples/scoring/
haddock3 emscoring-test.cfg
```

## 4. Contribute

If you want to contribute to HADDOCK3's development, read the
[CONTRIBUTING](CONTRIBUTING.md) file for instructions.

## 5. Keep in contact and support us

HADDOCK3 is an academic project supported by various grants, including the EU
BioExcel Center of Excellence for Computational Biomolecular Research. HADDOCK3
is fully open-source and free to download. If you clone this repository and use
HADDOCK3 for your research, please support us by signing [this Google
form][googleform] if you have not yet done so. This will allow us contact you
when needed for HADDOCK3-related issues, and also provide us a mean to
demonstrate impact when reporting for grants.

[googleform]: https://docs.google.com/forms/d/e/1FAIpQLScDcd0rWtuzJ_4nftkDAHoLVwr1IAVwNJGhbaZdTYZ4vWu25w/viewform


# PureQML::QMLCore: QML to Javascript translator
QmlCore is a simple set of tools we (a small team of QML advocates) use since years to simplify the development of HTML5 UIs for both mobile and desktop devices.
It was designed with the original QML in mind, while it's not 100% compatible and improved in some aspects.
The main concepts are the same though, so if you're familiar with original QML, you could start right away.

## Usage
QmlCore provides a toolchain written in Python using python-future, allowing it to be run on top of both python versions.
Normally you don't need to use them directly. ```build``` provides a convenient wrapper around them all.

### Prerequisites
Any modern Python (2 or 3) will go well. Jinja2 is a different option for templating in case you want more than QML loader in your HTML file, while it's not required for small apps.

To install the requirements run:
```python
$ pip install -r requirements.txt
```

## Simple how-to
* Create project directory, ```cd <project-dir>```
* Clone QmlCore to it: ```git clone git@github.com:pureqml/qmlcore.git```
* Run ./qmlcore/build --boilerplate
* Look into ./src/app.qml
* Run ./qmlcore/build
Please find the resulting files in build.web/*

## How it works
The QML compiler scans source directories for QML files and parses each one. Filename starting with uppercase letter considered component, lowercase instantiated. Project-wide options are stored in the ```.manifest``` file. The result of the compilation is a single Javascript file with minimum dependencies (modernizr only), ready to use in mobile and desktop environment and accompanied by sample .html launcher.

## Manifest options
Manifest is a collection of project-wide hacks we used to botch various projects. Some of them may or may not be useful.
* ```templater``` - template engine to use, only 'simple' and 'jinja2' are supported at the moment
* ```web-prefix``` - see -p option below, specify css rules prefix
* ```minify``` - false/true or compiler name as string, only 'gcc' and 'uglify-js' are supported. google closure compiler requires java to run.
* ```platforms``` use additional platform/*/ files, default and only platform is 'web' for now
* ```path``` - additional directories to search sources for

## build tool command line options
* ```-m, --minify``` minify with default option ('uglify-js')
* ```-k, --keep``` keep original source after minification, useful for debugging minification warnings
* ```-d, --devel``` development mode, keep running and wait for changes, requires inotify module
* ```-p, --web-prefix``` web prefix, removed default CSS rules, adds 'qml-' prefix for them, allowing you to interchange HTML/QML.
* ```-u, --update-translation``` update translation files, specified in manifest.languages
* ```-j, --jobs``` run N jobs in parallel
* ```--boilerplate``` initialises bare minimum for quicker kick-off in current directory.

# Supported Platforms
We support many different platforms and video integration variants.

Most notable platforms are:
- HTML5 Web Browser (Obviously) with different addons:
	- Shaka
	- Dash.js
	- Video.js
Web Extension
- ElectronJS
- Android (Native engine and Cordova)
- IOS (Cordova)
- SmartTV Platorms:
	- Any Android TV via native Android support.
	- LG (Netcast and WebOS)
	- Samsung SmartTv (Orsay and Tizen)
	- OperaTV
- Many STB platforms (privately)
- Native C++ engine with minimal requirements (EGL or LinuxFB) (privately)

For the full list of supported platform see [here](https://github.com/pureqml/qmlcore/tree/master/platform) and [here](https://github.com/pureqml/qmlcore-tv/tree/master/platform). Or alternatively just ask us.

## Native Android support
Main repo for Pureqml native Android implementation is https://github.com/pureqml/qmlcore-android.

In order to build native android app you need to:
- Install Android SDK and set `ANDROID_HOME` environment variable
- Run `./qmlcore/platform/pure.femto/build-android-native.sh` script.

# Localisation
QmlCore uses Qt-approach to localisation. You write the code in your default language, then generate/update (build -u) .ts translation files, translate them with qt linguist and compile your project. QmlCore recognizes tr, qsTr, qsTranslate function, as well as QT_TR_NOOP/QT_TRANSLATE_NOOP macros.

# Controls library
QmlCore contains a bare minimum of platform controls: Images, texts, rectangles and model-view-delegate classes. Various controls that might be useful are in a separate repository.
Just clone it via ```git clone git@github.com:pureqml/controls.git``` in your project and that's it!

# QML differences
### Grammar
We require a semicolon after each statement. This may be changed in future.

### Focus
The biggest discrepancy with original QML is how focus is implemented. We're aiming to have "always-consistent" focus everywhere.
You have to mark every focus-able component with ```focus: true;``` property, and the rest should work without tweaking.
We provide several convenient properties to handle focus with ease:
- activeFocus — this item has current focus and got any user input first
- focused — this item has current focus in its parent, but not necessarily focused globally

## Random customization notes
### Adding modernizr features
Please use the following command to get the custom modernizr build page:
```head -n2 dist/modernizr-custom.js | tail -n1```
or just the second line of modernizr-custom.js file

## Thanks

### Modernizr project
Modernizr tells you what HTML, CSS and JavaScript features the user’s browser has to offer.
https://modernizr.com/

### SDL Game Controller Database
A community sourced database of game controller mappings
https://github.com/gabomdq/SDL_GameControllerDB

### Apache Cordova project
Apache Cordova is an open-source mobile development framework. It allows you to use standard web technologies - HTML5, CSS3, and JavaScript for cross-platform development.
https://cordova.apache.org


![](docs/resource/img/logo_zh.png)
---
[![license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://github.com/TencentBlueKing/bk-nodeman/blob/master/LICENSE)
[![Release](https://img.shields.io/badge/release-1.0.0-brightgreen.svg)](https://github.com/TencentBlueKing/bk-nodeman/releases)
[![Coverage Status](https://codecov.io/gh/TencentBlueKing/bk-nodeman/branch/V2.1.X/graph/badge.svg)](https://codecov.io/gh/TencentBlueKing/bk-nodeman)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/TencentBlueKing/bk-nodeman/pulls)
[![unittest](https://github.com/TencentBlueKing/bk-nodeman/actions/workflows/codecov.yml/badge.svg?event=schedule)](https://github.com/TencentBlueKing/bk-nodeman/actions/workflows/codecov.yml)

[English](readme_en.md) | 简体中文

用户的业务主机想要与蓝鲸服务体系建立联系，首先要确保每台业务主机安装 Agent 程序。蓝鲸推出节点管理 SaaS，用以解决各类网络环境下 Agent 的快速安装难题。

节点管理 是一款用来帮助运维人员实现 Agent 安装、插件管理等后台服务管理的应用。

和手动安装 Agent 的原理相同，但是节点管理省去了脚本安装命令的手动键入和繁琐执行。用户无需登录服务器，通过可视化的窗口即可直观简便的执行 Agent 安装和插件管理。支持多种环境下的 Agent 安装，支持主机的并发操作、CMDB 导入和批量导入等，支持插件的启动、停止、重启、重载、是否托管等管理操作，高效方便。

## Overview
蓝鲸节点管理

## Features
了解更多功能，请参考[蓝鲸节点管理白皮书](http://docs.bk.tencent.com/product_white_paper/bk_nodeman/)

## Getting started
- [开发环境后台部署](docs/install/dev_deploy.md)

## Version plan
- [版本日志](docs/release.md)

## Support
- [源码](https://github.com/TencentBlueKing/bk-nodeman/tree/master)
- [wiki](https://github.com/TencentBlueKing/bk-nodeman/wiki)
- [白皮书](http://docs.bk.tencent.com/product_white_paper/bk_nodeman/)
- [蓝鲸论坛](https://bk.tencent.com/s-mart/community)
- [蓝鲸 DevOps 在线视频教程](https://bk.tencent.com/s-mart/video/)
- 联系我们，技术交流QQ群：

  社区版QQ交流群：495299374

  蓝鲸运维开发交流群：878501914

  <img src="docs/resource/img/QR-Code.png" align=center style="zoom:67%;" />

## BlueKing Community
* [BK-CMDB](https://github.com/Tencent/bk-cmdb): 蓝鲸配置平台（蓝鲸 CMDB）是一个面向资产及应用的企业级配置管理平台。
- [BK-CI](https://github.com/Tencent/bk-ci): 蓝鲸持续集成平台是一个开源的持续集成和持续交付系统，可以轻松将你的研发流程呈现到你面前。
- [BK-BCS](https://github.com/Tencent/bk-bcs): 蓝鲸容器管理平台是以容器技术为基础，为微服务业务提供编排管理的基础服务平台。
- [BK-PaaS](https://github.com/Tencent/bk-PaaS): 蓝鲸 PaaS 平台是一个开放式的开发平台，让开发者可以方便快捷地创建、开发、部署和管理 SaaS 应用。
- [BK-SOPS](https://github.com/Tencent/bk-sops): 标准运维（SOPS）是通过可视化的图形界面进行任务流程编排和执行的系统，是蓝鲸体系中一款轻量级的调度编排类 SaaS 产品。
- [BK-BASE](https://github.com/Tencent/bk-base): 蓝鲸基础计算平台(BK-BASE)是一个专注于运维领域的的基础平台，打造一站式、低门槛的基础服务。通过简化运维数据的收集、获取，提升数据开发效率，辅助运维人员实时运维决策，助力企业运营体系数字化、智能化转型。

## Contributing
如果你有好的意见或建议，欢迎给我们提 Issues 或 Pull Requests，为蓝鲸开源社区贡献力量。关于 蓝鲸节点管理 分支管理、Issue 以及 PR 规范，
请阅读 [Contributing Guide](docs/CONTRIBUTING.md)。

[腾讯开源激励计划](https://opensource.tencent.com/contribution) 鼓励开发者的参与和贡献，期待你的加入。

## License
蓝鲸节点管理 是基于 MIT 协议， 详细请参考 [LICENSE](https://github.com/TencentBlueKing/bk-nodeman/blob/master/LICENSE)

我们承诺未来不会更改适用于交付给任何人的当前项目版本的开源许可证（MIT 协议）。


.. image:: doc/content/logo.svg
    :alt: Lona logo
    :height: 200px
    :width: 200px
.. image:: https://img.shields.io/pypi/l/lona.svg
    :alt: license MIT
    :target: https://pypi.org/project/lona
.. image:: https://img.shields.io/pypi/pyversions/lona.svg
    :alt: python 3
    :target: https://pypi.org/project/lona
.. image:: https://img.shields.io/pypi/v/lona.svg
    :alt: latest version
    :target: https://pypi.org/project/lona
.. image:: https://github.com/lona-web-org/lona/actions/workflows/ci.yml/badge.svg
    :alt: ci status
    :target: https://github.com/lona-web-org/lona/actions/workflows/ci.yml
.. image:: https://img.shields.io/codecov/c/github/lona-web-org/lona.svg
    :alt: code coverage
    :target: https://codecov.io/gh/lona-web-org/lona/

Lona is a web application framework, designed to write responsive web apps in
**full** Python.

**Demos:** `lona-web.org/demos <https://lona-web.org/demos/index.html>`_

**FAQ:** `lona-web.org/faq <http://lona-web.org/faq.html>`_

**Documentation:** `lona-web.org <http://lona-web.org>`_

**Changelog:** `lona-web.org/changelog <http://lona-web.org/changelog.html>`_

**Reddit:** `reddit.com/r/lona_web_org/ <https://www.reddit.com/r/lona_web_org/>`_

**Discord:** `discord.com/lona-web.org <https://discord.gg/WBf5PVACsj>`_

Web is a solved problem in Python since ages, but traditionally Python handles
only the server side. If you want to have client side interaction like
click events or you want update content live, you have to write an additional
Javascript application.

Lona handles the server side and the client side, and provides a simple,
pythonic API to write self contained views.

.. code-block:: text

    # pip install lona

.. code-block:: python

    from lona.html import HTML, Button, Div, H1
    from lona import LonaApp, LonaView

    app = LonaApp(__file__)

    @app.route('/')
    class MyView(LonaView):
        def handle_button_click(self, input_event):
            self.message.set_text('Button clicked')

        def handle_request(self, request):
            self.message = Div('Button not clicked')

            html = HTML(
                H1('Click the button!'),
                self.message,
                Button('Click me!', handle_click=self.handle_button_click),
            )

            return html

    if __name__ == '__main__':
        app.run(port=8080, live_reload=True)

**More information:**
`Getting Started <http://lona-web.org/tutorial/01-getting-started/index.html>`_

How does it work?
-----------------

Lona comes with a Javascript based browser library that speaks a specialized
protocol with the backend.
This protocol specifies messages like "hey frontend, please show $HTML" and
"hey backend, someone clicked on node XY".

**More information:**
`Basic Concept <https://lona-web.org/basic-concept.html>`_


 x2vec, Towhee is all you need!

  ENGLISH | 中文文档

[Towhee](https://towhee.io) is a cutting-edge framework designed to streamline the processing of unstructured data through the use of Large Language Model (LLM) based pipeline orchestration. It is uniquely positioned to extract invaluable insights from diverse unstructured data types, including lengthy text, images, audio and video files. Leveraging the capabilities of generative AI and the SOTA deep learning models, Towhee is capable of transforming this unprocessed data into specific formats such as text, image, or embeddings. These can then be efficiently loaded into an appropriate storage system like a vector database. Developers can initially build an intuitive data processing pipeline prototype with user friendly Pythonic API, then optimize it for production environments.

🎨 Multi Modalities: Towhee is capable of handling a wide range of data types. Whether it's image data, video clips, text, audio files, or even molecular structures, Towhee can process them all.

📃    LLM Pipeline orchestration:  Towhee offers flexibility to adapt to different Large Language Models (LLMs). Additionally, it allows for hosting open-source large models locally. Moreover, Towhee provides features like prompt management and knowledge retrieval, making the interaction with these LLMs more efficient and effective.

🎓 Rich Operators: Towhee provides a wide range of ready-to-use state-of-the-art models across five domains: CV, NLP, multimodal, audio, and medical. With over 140 models like BERT and CLIP and rich functionalities like video decoding, audio slicing, frame sampling,  and dimensionality reduction, it assists in efficiently building data processing pipelines.

🔌   Prebuilt ETL Pipelines: Towhee offers ready-to-use ETL (Extract, Transform, Load) pipelines for common tasks such as Retrieval-Augmented Generation, Text Image search, and Video copy detection. This means you don't need to be an AI expert to build applications using these features.
⚡  High performance backend: Leveraging the power of the Triton Inference Server, Towhee can speed up model serving on both CPU and GPU using platforms like TensorRT, Pytorch, and ONNX. Moreover, you can transform your Python pipeline into a high-performance docker container with just a few lines of code, enabling efficient deployment and scaling.

🐍 Pythonic API: Towhee includes a Pythonic method-chaining API for describing custom data processing pipelines. We also support schemas, which makes processing unstructured data as easy as handling tabular data.

## Getting started

Towhee requires Python 3.7+. You can install Towhee via `pip`:

```bash
pip install towhee towhee.models
```

### Pipeline

### Pre-defined Pipeline

Towhee provides some pre-defined pipelines to help users quickly implement some functions.
Currently implemented are:
- [Sentence Embedding](https://towhee.io/tasks/detail/pipeline/sentence-similarity)
- [Image Embedding](https://towhee.io/tasks/detail/pipeline/text-image-search)
- [Video deduplication](https://towhee.io/tasks/detail/pipeline/video-copy-detection)
- [Question Answer with Docs](https://towhee.io/tasks/detail/pipeline/retrieval-augmented-generation)

All pipelines can be found on Towhee Hub. Here is an example of using the sentence_embedding pipeline:

```python
from towhee import AutoPipes, AutoConfig
# get the built-in sentence_similarity pipeline
config = AutoConfig.load_config('sentence_embedding')
config.model = 'paraphrase-albert-small-v2'
config.device = 0
sentence_embedding = AutoPipes.pipeline('sentence_embedding', config=config)

# generate embedding for one sentence
embedding = sentence_embedding('how are you?').get()
# batch generate embeddings for multi-sentences
embeddings = sentence_embedding.batch(['how are you?', 'how old are you?'])
embeddings = [e.get() for e in embeddings]
```
### Custom pipelines

If you can't find the pipeline you want in towhee hub, you can also implement custom pipelines through the towhee Python API. In the following example, we will create a cross-modal retrieval pipeline based on CLIP.
```python

from towhee import ops, pipe, DataCollection
# create image embeddings and build index
p = (
    pipe.input('file_name')
    .map('file_name', 'img', ops.image_decode.cv2())
    .map('img', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch32', modality='image'))
    .map('vec', 'vec', ops.towhee.np_normalize())
    .map(('vec', 'file_name'), (), ops.ann_insert.faiss_index('./faiss', 512))
    .output()
)

for f_name in ['https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog1.png',
               'https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog2.png',
               'https://raw.githubusercontent.com/towhee-io/towhee/main/assets/dog3.png']:
    p(f_name)

# Flush faiss data into disk.
p.flush()
# search image by text
decode = ops.image_decode.cv2('rgb')
p = (
    pipe.input('text')
    .map('text', 'vec', ops.image_text_embedding.clip(model_name='clip_vit_base_patch32', modality='text'))
    .map('vec', 'vec', ops.towhee.np_normalize())
    # faiss op result format:  [[id, score, [file_name], ...]
    .map('vec', 'row', ops.ann_search.faiss_index('./faiss', 3))
    .map('row', 'images', lambda x: [decode(item[2][0]) for item in x])
    .output('text', 'images')
)

DataCollection(p('puppy Corgi')).show()

```

## Core Concepts

Towhee is composed of four main building blocks - `Operators`, `Pipelines`, `DataCollection API` and `Engine`.

- __Operators__: An operator is a single building block of a neural data processing pipeline. Different implementations of operators are categorized by tasks, with each task having a standard interface. An operator can be a deep learning model, a data processing method, or a Python function.

- __Pipelines__: A pipeline is composed of several operators interconnected in the form of a DAG (directed acyclic graph). This DAG can direct complex functionalities, such as embedding feature extraction, data tagging, and cross modal data analysis.

- __DataCollection API__: A Pythonic and method-chaining style API for building custom pipelines, providing multiple data conversion interfaces: map, filter, flat_map, concat, window, time_window, and window_all. Through these interfaces, complex data processing pipelines can be built quickly to process unstructured data such as video, audio, text, images, etc.

- __Engine__: The engine sits at Towhee's core. Given a pipeline, the engine will drive dataflow among individual operators, schedule tasks, and monitor compute resource usage (CPU/GPU/etc). We provide a basic engine within Towhee to run pipelines on a single-instance machine and a Triton-based engine for docker containers.

## Resource
- TowheeHub: https://towhee.io/
- docs: https://towhee.readthedocs.io/en/latest/
- examples: https://github.com/towhee-io/examples

## Contributing

Writing code is not the only way to contribute! Submitting issues, answering questions, and improving documentation are just some of the many ways you can help our growing community. Check out our [contributing page](https://github.com/towhee-io/towhee/blob/main/CONTRIBUTING.md) for more information.

Special thanks goes to these folks for contributing to Towhee, either on Github, our Towhee Hub, or elsewhere:

Looking for a database to store and index your embedding vectors? Check out [Milvus](https://github.com/milvus-io/milvus).


# BioThings SDK

[![Downloads](https://pepy.tech/badge/biothings)](https://pepy.tech/project/biothings)
[![biothings package](https://badge.fury.io/py/biothings.svg)](https://pypi.python.org/pypi/biothings)
[![biothings_version](https://img.shields.io/pypi/pyversions/biothings.svg)](https://pypi.python.org/pypi/biothings)
[![biothings_version](https://img.shields.io/pypi/format/biothings.svg)](https://pypi.python.org/pypi/biothings)
[![biothings_version](https://img.shields.io/pypi/status/biothings.svg)](https://pypi.python.org/pypi/biothings)
[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](CODE_OF_CONDUCT.md)
[![Build Status](https://github.com/biothings/biothings.api/actions/workflows/test-build.yml/badge.svg)](https://github.com/biothings/biothings.api/actions/workflows/test-build.yml)
[![Tests Status](https://github.com/biothings/biothings.api/actions/workflows/run-tests.yml/badge.svg)](https://github.com/biothings/biothings.api/actions/workflows/run-tests.yml)
[![Documentation Status](https://readthedocs.org/projects/biothingsapi/badge/?version=latest)](https://docs.biothings.io/en/latest/?badge=latest)

## Quick Summary

BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data source or multiple data sources. It has the particular focus on building data APIs for biomedical-related entities, a.k.a "BioThings" (such as genes, genetic variants, drugs, chemicals, diseases, etc.).

Documentation about BioThings SDK can be found at https://docs.biothings.io

## Introduction

### What's BioThings?

We use "**BioThings**" to refer to objects of any biomedical entity-type
represented in the biological knowledge space, such as genes, genetic
variants, drugs, chemicals, diseases, etc.

### BioThings SDK

SDK represents "Software Development Kit". BioThings SDK provides a
[Python-based](https://www.python.org/) toolkit to build
high-performance data APIs (or web services) from a single data source
or multiple data sources. It has the particular focus on building data
APIs for biomedical-related entities, a.k.a "*BioThings*", though it's
not necessarily limited to the biomedical scope. For any given
"*BioThings*" type, BioThings SDK helps developers to aggregate
annotations from multiple data sources, and expose them as a clean and
high-performance web API.

The BioThings SDK can be roughly divided into two main components: data
hub (or just "hub") component and web component. The hub component
allows developers to automate the process of monitoring, parsing and
uploading your data source to an
[Elasticsearch](https://www.elastic.co/products/elasticsearch) backend.
From here, the web component, built on the high-concurrency [Tornado Web
Server](http://www.tornadoweb.org/en/stable/) , allows you to easily
setup a live high-performance API. The API endpoints expose
simple-to-use yet powerful query features using [Elasticsearch's
full-text query capabilities and query
language](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-query-string-query.html#query-string-syntax).

### BioThings API

We also use "*BioThings API*" (or *BioThings APIs*) to refer to an API
(or a collection of APIs) built with BioThings SDK. For example, both
our popular [MyGene.Info](http://mygene.info/) and
[MyVariant.Info](http://myvariant.info/) APIs are built and maintained
using this BioThings SDK.

### BioThings Studio

*BioThings Studio* is a buildin, pre-configured environment used to build and
administer a BioThings API. At its core is the *Hub*, a backend service responsible for maintaining data up-to-date, producing data releases and
update API frontends.

## Installing BioThings SDK

You can install the latest stable BioThings SDK release with pip from
[PyPI](https://pypi.python.org/pypi), like:

    # default to install web requirements only for running an API
    pip install biothings
    # or include additional requirements useful for running an API on production
    # like msgpack, sentry-sdk pacakages
    pip install biothings[web_extra]

    # install hub requirements for running a hub (including CLI)
    pip install biothings[hub]

    # install CLI-only requirements if you only need CLI for develop a data plugin
    pip install biothings[cli]

    # need support for docker data plugin
    pip install biothings[hub,docker]
    # or if use ssh protocol to connect to a remote docker server
    pip install biothings[hub,docker_ssh]

    # just install everything for dev purpose
    pip install biothings[dev]

You can check more details for the optional dependecy packages directly in [setup.py](setup.py) file.

You can install the latest development version of BioThings SDK directly
from our github repository like:

    pip install git+https://github.com/biothings/biothings.api.git#egg=biothings

    # from a branch or commit
    pip install git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings

    # include optional dependecies
    pip install git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings[web_extra]

    # can be very useful to install in “editable” mode:
    pip install -e git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings[web_extra]

Alternatively, you can download the source code, or clone the [BioThings
SDK repository](https://github.com/biothings/biothings.api) and run:

    pip install .
    # or
    pip install .[web_extra]

## Get started to build a BioThings API

We recommend to follow [this tutorial](https://docs.biothings.io/en/latest/tutorial/studio.html) to develop your first BioThings API in our pre-configured **BioThings Studio** development environment.

## Documentation

The latest documentation is available at https://docs.biothings.io.

## How to contribute

Please check out this [Contribution Guidelines](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md) document.

## Active and past contributors

Please see [Contributors](Contributors.md)


Python Dialogflow CX Scripting API (SCRAPI)

    A high level scripting API for bot builders, developers, and maintainers.

  Table of Contents

      Introduction

What Can I Do with SCRAPI?

Built With

      Getting Started

Environment Setup

Authentication

Usage

      Library Composition

Core

Tools

Contributing

License

Contact

Acknowledgements

# Introduction

The Python Dialogflow CX Scripting API (DFCX SCRAPI) is a high level API that extends the official Google [Python Client for Dialogflow CX](https://github.com/googleapis/python-dialogflow-cx). SCRAPI makes using DFCX easier, more friendly, and more pythonic for bot builders, developers, and maintainers.

```
SCRAPI --> Python Dialogflow CX
as
Keras --> Tensorflow
```

## What Can I Do With DFCX SCRAPI?
With DFCX SCRAPI you can perform many bot building and maintenance actions at scale including, but not limited to:
- Create, Update, Delete, Get, and List for all CX resources types (i.e. Intents, Entity Types, Pages, Flows, etc.)
- Convert commonly accessed CX Resources to Pandas Dataframes
- Have fully automated conversations with a CX agent (powerful for regression testing!)
- Extract Validation information
- Extract Change History information
- Search across all Flows/Pages/Routes to find a specific parameter or utterance using Search Util functions
- Quickly move CX resources between agents using Copy Util functions!
- Build the fundamental protobuf objects that CX uses for each resource type using Builder methods
- ...and much, much more!

## Built With
* Python 3.8+

# Authentication
Authentication can vary depending on how and where you are interacting with SCRAPI.

## Google Colab
If you're using SCRAPI with a [Google Colab](https://colab.research.google.com/) notebook, you can add the following to the top of your notebook for easy authentication:
```py
project_id = '<YOUR_GCP_PROJECT_ID>'

# this will launch an interactive prompt that allows you to auth with GCP in a browser
!gcloud auth application-default login --no-launch-browser

# this will set your active project to the `project_id` above
!gcloud auth application-default set-quota-project $project_id
```

After running the above, Colab will pick up your credentials from the environment and pass them to SCRAPI directly. No need to use Service Account keys!
You can then use SCRAPI simply like this:
```py
from dfcx_scrapi.core.intents import Intents

agent_id = '<YOUR_AGENT_ID>'
i = Intents() # <-- Creds will be automatically picked up from the environment
intents_map = i.get_intents_map(agent_id)
```
---
## Cloud Functions / Cloud Run
If you're using SCRAPI with [Cloud Functions](https://cloud.google.com/functions) or [Cloud Run](https://cloud.google.com/run), SCRAPI can pick up on the default environment creds used by these services without any additional configuration!

1. Add `dfcx-scrapi` to your `requirements.txt` file
2. Ensure the Cloud Function / Cloud Run service account has the appropriate Dialogflow IAM Role

Once you are setup with the above, your function code can be used easily like this:
```py
from dfcx_scrapi.core.intents import Intents

agent_id = '<YOUR_AGENT_ID>'
i = Intents() # <-- Creds will be automatically picked up from the environment
intents_map = i.get_intents_map(agent_id)
```

---
## Local Python Environment
Similar to Cloud Functions / Cloud Run, SCRAPI can pick up on your local authentication creds _if you are using the gcloud CLI._

1. Install [gcloud CLI](https://cloud.google.com/sdk/docs/install).
2. Run `gcloud init`.
3. Run `gcloud auth login`
4. Run `gcloud auth list` to ensure your principal account is active.

This will authenticate your principal GCP account with the gcloud CLI, and SCRAPI can pick up the creds from here.

---
## Exceptions and Misc.
There are some classes in SCRAPI which still rely on Service Account Keys, notably the [DataframeFunctions](https://github.com/GoogleCloudPlatform/dfcx-scrapi/blob/main/src/dfcx_scrapi/tools/dataframe_functions.py) class due to how it authenticates with Google Sheets.

In order to use these functions, you will need a Service Account that has appropriate access to your GCP project.
For more information and to view the official documentation for service accounts go to [Creating and Managing GCP Service Accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts).

Once you've obtained a Service Account Key with appropriate permissions, you can use it as follows:
```py
from dfcx_scrapi.core.intents import Intents
from dfcx_scrapi.tools.dataframe_functions import DataframeFunctions

agent_id = '<YOUR_AGENT_ID>'
creds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'

i = Intents(creds_path=creds_path)
dffx = DataframeFunctions(creds_path=creds_path)

df = i.bulk_intent_to_df(agent_id)
dffx.dataframe_to_sheets('GOOGLE_SHEET_NAME', 'TAB_NAME', df)
```

# Getting Started
## Environment Setup
Set up Google Cloud Platform credentials and install dependencies.
```sh
gcloud auth login
gcloud auth application-default login
gcloud config set project <project name>
```
```sh
python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
```

## Usage
To run a simple bit of code you can do the following:
- Import a Class from `dfcx_scrapi.core`
- Assign your Service Account to a local variable

```python
from dfcx_scrapi.core.intents import Intents

creds_path = '<PATH_TO_YOUR_SERVICE_ACCOUNT_JSON_FILE>'
agent_path = '<FULL_DFCX_AGENT_ID_PATH>'

# DFCX Agent ID paths are in this format:
# 'projects/<project_id>/locations/<location_id>/agents/<agent_id>'

# Instantiate your class object and pass in your credentials
i = Intents(creds_path, agent_id=agent_path)

# Retrieve all Intents and Training Phrases from an Agent and push to a Pandas DataFrame
df = i.bulk_intent_to_df()
```

# Library Composition
Here is a brief overview of the SCRAPI library's structure and the motivation behind that structure.

## Core
The [Core](/src/dfcx_scrapi/core) folder is synonymous with the core Resource types in the DFCX Agents (agents, intents, flows, etc.)
* This folder contains the high level building blocks of SCRAPI
* These classes and methods can be used to build higher level methods or custom tools and applications

## Tools
The [Tools](/src/dfcx_scrapi/tools) folder contains various customized toolkits that allow you to do more complex bot management tasks, such as
- Manipulate Agent Resource types into various DataFrame structures
- Copy Agent Resources between Agents and GCP Projects on a resource by resource level
- Move data to and from DFCX and other GCP Services like BigQuery, Sheets, etc.
- Create customized search queries inside of your agent resources

## Builders
The [Builders](/src/dfcx_scrapi/builders) folder contains simple methods for constructing the underlying protos in Dialogflow CX
- Proto objects are the fundamental building blocks of Dialogflow CX
- Builder classes allow the user to construct Dialogflow CX resource _offline_ without any API calls
- Once the resource components are constructed, they can then be pushed to a live Dialogflow CX agent via API

# Contributing
We welcome any contributions or feature requests you would like to submit!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

# License
Distributed under the Apache 2.0 License. See [LICENSE](LICENSE.txt) for more information.

# Contact
Patrick Marlow - pmarlow@google.com  - [@kmaphoenix](https://github.com/kmaphoenix)
Milad Tabrizi - miladt@google.com - [@MRyderOC](https://github.com/MRyderOC)

Project Link: [https://github.com/GoogleCloudPlatform/dfcx-scrapi](https://github.com/GoogleCloudPlatform/dfcx-scrapi)

# Acknowledgements
[Dialogflow CX Python Client Library](https://github.com/googleapis/python-dialogflow-cx)
[Hugging Face - Pegasus Paraphrase](https://huggingface.co/tuner007/pegasus_paraphrase)


# Tanjun

A flexible command framework designed to extend Hikari.

# Installation

You can install Tanjun from PyPI using the following command in any Python 3.9 or above environment.

```
python -m pip install -U hikari-tanjun
```

# Quick Usage

For usage see the [documentation](https://tanjun.cursed.solutions/),
[usage guide](https://tanjun.cursed.solutions/usage/), and
[tutorials and articles](https://patchwork.systems/programming/hikari-discord-bot/index.html).

# Community Resources

* [Tan-chan](https://tanchan.cursed.solutions/) is a general utility library for
  Tanjun. It includes a command annotation parsing extension which parses docstrings to
  get the descriptions of slash commands and their options.
* Redis based implementations of the async cache dependency can be found in
  [hikari-sake](https://sake.cursed.solutions/) \>=v1.0.1a1 (exposed by
  [sake.redis.ResourceClient.add_to_tanjun][]).

# Support

[Hikari's support guild](https://discord.gg/hikari) provides for support for Tanjun.

# Contributing

Before contributing you should read through the
[contributing guidelines](https://github.com/FasterSpeeding/Tanjun/blob/master/CONTRIBUTING.md) and
the [code of conduct](https://github.com/FasterSpeeding/Tanjun/blob/master/CODE_OF_CONDUCT.md).


[![Release](https://github.com/opensearch-project/opensearch-py/actions/workflows/unified-release.yml/badge.svg)](https://github.com/opensearch-project/opensearch-py/actions/workflows/unified-release.yml)
[![CI](https://github.com/opensearch-project/opensearch-py/actions/workflows/ci.yml/badge.svg)](https://github.com/opensearch-project/opensearch-py/actions/workflows/ci.yml)
[![Integration](https://github.com/opensearch-project/opensearch-py/actions/workflows/integration.yml/badge.svg)](https://github.com/opensearch-project/opensearch-py/actions/workflows/integration.yml)
[![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/clients/)
![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success)

![OpenSearch logo](https://github.com/opensearch-project/opensearch-py/raw/main/OpenSearch.svg)

OpenSearch Python Client

- [Welcome!](https://github.com/opensearch-project/opensearch-py#welcome)
- [User Guide](https://github.com/opensearch-project/opensearch-py#user-guide)
- [API Doc](https://opensearch-project.github.io/opensearch-py/)
- [Compatibility with OpenSearch](https://github.com/opensearch-project/opensearch-py#compatibility-with-opensearch)
- [Project Resources](https://github.com/opensearch-project/opensearch-py#project-resources)
- [Code of Conduct](https://github.com/opensearch-project/opensearch-py#code-of-conduct)
- [License](https://github.com/opensearch-project/opensearch-py#license)
- [Copyright](https://github.com/opensearch-project/opensearch-py#copyright)

# Welcome!

**opensearch-py** is [a community-driven, open source fork](https://aws.amazon.com/blogs/opensource/introducing-opensearch/)
of elasticsearch-py licensed under the [Apache v2.0 License](https://github.com/opensearch-project/opensearch-py/blob/main/LICENSE.txt).
For more information, see [opensearch.org](https://opensearch.org/) and the [API Doc](https://opensearch-project.github.io/opensearch-py/).

## User Guide

To get started with the OpenSearch Python Client, see [User Guide](https://github.com/opensearch-project/opensearch-py/blob/main/USER_GUIDE.md). This repository also contains [working samples](https://github.com/opensearch-project/opensearch-py/tree/main/samples) and [benchmarks](https://github.com/opensearch-project/opensearch-py/tree/main/benchmarks).

## Compatibility with OpenSearch

See [Compatibility](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md).

## Project Resources

* [Project Website](https://opensearch.org/)
* [Downloads](https://opensearch.org/downloads.html)
* [Documentation](https://opensearch.org/docs/latest/clients/python/)
* Need help? Try [Forums](https://discuss.opendistrocommunity.dev/)
* [Project Principles](https://opensearch.org/#principles)
* [Contributing to OpenSearch](https://github.com/opensearch-project/opensearch-py/blob/main/CONTRIBUTING.md)
* [Maintainer Responsibilities](https://github.com/opensearch-project/opensearch-py/blob/main/MAINTAINERS.md)
* [Release Management](https://github.com/opensearch-project/opensearch-py/blob/main/RELEASING.md)
* [Admin Responsibilities](https://github.com/opensearch-project/opensearch-py/blob/main/ADMINS.md)
* [Security](https://github.com/opensearch-project/opensearch-py/blob/main/SECURITY.md)

## Code of Conduct

This project has adopted the
[Amazon Open Source Code of Conduct](https://github.com/opensearch-project/opensearch-py/blob/main/CODE_OF_CONDUCT.md).
For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact
[opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments.

## License

This project is licensed under the
[Apache v2.0 License](https://github.com/opensearch-project/opensearch-py/blob/main/LICENSE.txt).

## Copyright

Copyright OpenSearch Contributors. See
[NOTICE](https://github.com/opensearch-project/opensearch-py/blob/main/NOTICE.txt) for details.


# 蓝鲸用户管理
![](docs/images/logo.png)

---
[![license](https://img.shields.io/badge/license-mit-green.svg?style=flat)](https://github.com/TencentBlueKing/bk-user/blob/master/LICENSE)
[![Release Version](https://img.shields.io/badge/bk--user-2.3.4-green)](https://github.com/TencentBlueKing/bk-user/releases)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-green.svg)](https://github.com/TencentBlueKing/bk-user/pulls)

简体中文 | [English](readme_en.md)

蓝鲸用户管理是蓝鲸智云提供的企业组织架构和用户管理解决方案，为企业统一登录提供认证源服务。

## 总览

- [架构设计](docs/architecture.md)
- [代码目录](docs/develop_guide.md)

## 功能

- 支持多层级的组织架构管理
- 支持通过多种方式同步数据：OpenLDAP、Microsoft Active Directory(MAD)、Excel 表格等
- 支持用户密码周期管理、密码强度校验、用户登录试错限制、邮件发送随机密码等安全措施

详细介绍请参考[功能说明](https://bk.tencent.com/docs/document/6.0/146/6638)

## 快速开始

- [本地开发部署指引](/docs/develop_guide.md)

## 路线图

- [版本日志](https://github.com/TencentBlueKing/bk-user/releases)

## 支持

- [产品白皮书](https://bkdocs-1252002024.file.myqcloud.com/ZH/6.0/%E7%94%A8%E6%88%B7%E7%AE%A1%E7%90%86/%E7%94%A8%E6%88%B7%E7%AE%A1%E7%90%86.pdf)
- [蓝鲸论坛](https://bk.tencent.com/s-mart/community)

## 蓝鲸社区
- [BK-CI](https://github.com/Tencent/bk-ci)：蓝鲸持续集成平台是一个开源的持续集成和持续交付系统，可以轻松将你的研发流程呈现到你面前。
- [BK-BCS](https://github.com/Tencent/bk-bcs)：蓝鲸容器管理平台是以容器技术为基础，为微服务业务提供编排管理的基础服务平台。
- [BK-CMDB](https://github.com/Tencent/bk-cmdb)：蓝鲸配置平台（蓝鲸CMDB）是一个面向资产及应用的企业级配置管理平台。
- [BK-PaaS](https://github.com/Tencent/bk-PaaS)：蓝鲸PaaS平台是一个开放式的开发平台，让开发者可以方便快捷地创建、开发、部署和管理SaaS应用。
- [BK-SOPS](https://github.com/Tencent/bk-sops)：标准运维（SOPS）是通过可视化的图形界面进行任务流程编排和执行的系统，是蓝鲸体系中一款轻量级的调度编排类SaaS产品。

## 贡献
对于项目感兴趣，想一起贡献并完善项目请参阅 [Contributing Guide](docs/contributing.md)。

[腾讯开源激励计划](https://opensource.tencent.com/contribution) 鼓励开发者的参与和贡献，期待你的加入。

## 协议

基于 MIT 协议， 详细请参考[LICENSE](LICENSE)