[![Python Versions](https://img.shields.io/pypi/pyversions/pyqlib.svg?logo=python&logoColor=white)](https://pypi.org/project/pyqlib/#files)
[![Platform](https://img.shields.io/badge/platform-linux%20%7C%20windows%20%7C%20macos-lightgrey)](https://pypi.org/project/pyqlib/#files)
[![PypI Versions](https://img.shields.io/pypi/v/pyqlib)](https://pypi.org/project/pyqlib/#history)
[![Upload Python Package](https://github.com/microsoft/qlib/workflows/Upload%20Python%20Package/badge.svg)](https://pypi.org/project/pyqlib/)
[![Github Actions Test Status](https://github.com/microsoft/qlib/workflows/Test/badge.svg?branch=main)](https://github.com/microsoft/qlib/actions)
[![Documentation Status](https://readthedocs.org/projects/qlib/badge/?version=latest)](https://qlib.readthedocs.io/en/latest/?badge=latest)
[![License](https://img.shields.io/pypi/l/pyqlib)](LICENSE)
[![Join the chat at https://gitter.im/Microsoft/qlib](https://badges.gitter.im/Microsoft/qlib.svg)](https://gitter.im/Microsoft/qlib?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

## :newspaper: **What's NEW!**     :sparkling_heart:
Recent released features
| Feature | Status |
| --                      | ------    |
| KRNN and Sandwich models | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/1414/) on May 26, 2023 |
| Release Qlib v0.9.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.9.0) on Dec 9, 2022 |
| RL Learning Framework | :hammer: :chart_with_upwards_trend: Released on Nov 10, 2022. [#1332](https://github.com/microsoft/qlib/pull/1332), [#1322](https://github.com/microsoft/qlib/pull/1322), [#1316](https://github.com/microsoft/qlib/pull/1316),[#1299](https://github.com/microsoft/qlib/pull/1299),[#1263](https://github.com/microsoft/qlib/pull/1263), [#1244](https://github.com/microsoft/qlib/pull/1244), [#1169](https://github.com/microsoft/qlib/pull/1169), [#1125](https://github.com/microsoft/qlib/pull/1125), [#1076](https://github.com/microsoft/qlib/pull/1076)|
| HIST and IGMTF models | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/1040) on Apr 10, 2022 |
| Qlib [notebook tutorial](https://github.com/microsoft/qlib/tree/main/examples/tutorial) | 📖 [Released](https://github.com/microsoft/qlib/pull/1037) on Apr 7, 2022 |
| Ibovespa index data | :rice: [Released](https://github.com/microsoft/qlib/pull/990) on Apr 6, 2022 |
| Point-in-Time database | :hammer: [Released](https://github.com/microsoft/qlib/pull/343) on Mar 10, 2022 |
| Arctic Provider Backend & Orderbook data example | :hammer: [Released](https://github.com/microsoft/qlib/pull/744) on Jan 17, 2022 |
| Meta-Learning-based framework & DDG-DA  | :chart_with_upwards_trend:  :hammer: [Released](https://github.com/microsoft/qlib/pull/743) on Jan 10, 2022 |
| Planning-based portfolio optimization | :hammer: [Released](https://github.com/microsoft/qlib/pull/754) on Dec 28, 2021 |
| Release Qlib v0.8.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.8.0) on Dec 8, 2021 |
| ADD model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/704) on Nov 22, 2021 |
| ADARNN  model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/689) on Nov 14, 2021 |
| TCN  model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/668) on Nov 4, 2021 |
| Nested Decision Framework | :hammer: [Released](https://github.com/microsoft/qlib/pull/438) on Oct 1, 2021. [Example](https://github.com/microsoft/qlib/blob/main/examples/nested_decision_execution/workflow.py) and [Doc](https://qlib.readthedocs.io/en/latest/component/highfreq.html) |
| Temporal Routing Adaptor (TRA) | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/531) on July 30, 2021 |
| Transformer & Localformer | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/508) on July 22, 2021 |
| Release Qlib v0.7.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.7.0) on July 12, 2021 |
| TCTS Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/491) on July 1, 2021 |
| Online serving and automatic model rolling | :hammer:  [Released](https://github.com/microsoft/qlib/pull/290) on May 17, 2021 |
| DoubleEnsemble Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/286) on Mar 2, 2021 |
| High-frequency data processing example | :hammer: [Released](https://github.com/microsoft/qlib/pull/257) on Feb 5, 2021  |
| High-frequency trading example | :chart_with_upwards_trend: [Part of code released](https://github.com/microsoft/qlib/pull/227) on Jan 28, 2021  |
| High-frequency data(1min) | :rice: [Released](https://github.com/microsoft/qlib/pull/221) on Jan 27, 2021 |
| Tabnet Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/205) on Jan 22, 2021 |

Features released before 2021 are not listed here.

Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning.

An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies.

It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution.
For more details, please refer to our paper ["Qlib: An AI-oriented Quantitative Investment Platform"](https://arxiv.org/abs/2009.11189).

      Frameworks, Tutorial, Data & DevOps
      Main Challenges & Solutions in Quant Research

Plans

Framework of Qlib

Quick Start

Installation

Data Preparation

Auto Quant Research Workflow

Building Customized Quant Research Workflow by Code

Quant Dataset Zoo

Learning Framework

More About Qlib

Offline Mode and Online Mode

Performance of Qlib Data Server

Related Reports

Contact Us

Contributing

Main Challenges & Solutions in Quant Research

Forecasting: Finding Valuable Signals/Patterns

Quant Model (Paper) Zoo

Run a Single Model

Run Multiple Models

Adapting to Market Dynamics

Reinforcement Learning: modeling continuous decisions

# Plans
New features under development(order by estimated release time).
Your feedbacks about the features are very important.

# Framework of Qlib

The high-level framework of Qlib can be found above(users can find the [detailed framework](https://qlib.readthedocs.io/en/latest/introduction/introduction.html#framework) of Qlib's design when getting into nitty gritty).
The components are designed as loose-coupled modules, and each component could be used stand-alone.

Qlib provides a strong infrastructure to support Quant research. [Data](https://qlib.readthedocs.io/en/latest/component/data.html) is always an important part.
A strong learning framework is designed to support diverse learning paradigms (e.g. [reinforcement learning](https://qlib.readthedocs.io/en/latest/component/rl.html), [supervised learning](https://qlib.readthedocs.io/en/latest/component/workflow.html#model-section)) and patterns at different levels(e.g. [market dynamic modeling](https://qlib.readthedocs.io/en/latest/component/meta.html)).
By modeling the market, [trading strategies](https://qlib.readthedocs.io/en/latest/component/strategy.html) will generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can be [nested to be optimized and run together](https://qlib.readthedocs.io/en/latest/component/highfreq.html).
At last, a comprehensive [analysis](https://qlib.readthedocs.io/en/latest/component/report.html) will be provided and the model can be [served online](https://qlib.readthedocs.io/en/latest/component/online.html) in a low cost.

# Quick Start

This quick start guide tries to demonstrate
1. It's very easy to build a complete Quant research workflow and try your ideas with _Qlib_.
2. Though with *public data* and *simple models*, machine learning technologies **work very well** in practical Quant investment.

Here is a quick **[demo](https://terminalizer.com/view/3f24561a4470)** shows how to install ``Qlib``, and run LightGBM with ``qrun``. **But**, please make sure you have already prepared the data following the [instruction](#data-preparation).

## Installation

This table demonstrates the supported Python version of `Qlib`:
|               | install with pip           | install from source  | plot |
| ------------- |:---------------------:|:--------------------:|:----:|
| Python 3.7    | :heavy_check_mark:    | :heavy_check_mark:   | :heavy_check_mark: |
| Python 3.8    | :heavy_check_mark:    | :heavy_check_mark:   | :heavy_check_mark: |
| Python 3.9    | :x:                   | :heavy_check_mark:   | :x: |

**Note**:
1. **Conda** is suggested for managing your Python environment. In some cases, using Python outside of a `conda` environment may result in missing header files, causing the installation failure of certain packages.
1. Please pay attention that installing cython in Python 3.6 will raise some error when installing ``Qlib`` from source. If users use Python 3.6 on their machines, it is recommended to *upgrade* Python to version 3.7 or use `conda`'s Python to install ``Qlib`` from source.
1. For Python 3.9, `Qlib` supports running workflows such as training models, doing backtest and plot most of the related figures (those included in [notebook](examples/workflow_by_code.ipynb)). However, plotting for the *model performance* is not supported for now and we will fix this when the dependent packages are upgraded in the future.
1. `Qlib`Requires `tables` package, `hdf5` in tables does not support python3.9.

### Install with pip
Users can easily install ``Qlib`` by pip according to the following command.

```bash
  pip install pyqlib
```

**Note**: pip will install the latest stable qlib. However, the main branch of qlib is in active development. If you want to test the latest scripts or functions in the main branch. Please install qlib with the methods below.

### Install from source
Also, users can install the latest dev version ``Qlib`` by the source code according to the following steps:

* Before installing ``Qlib`` from source, users need to install some dependencies:

  ```bash
  pip install numpy
  pip install --upgrade  cython
  ```

* Clone the repository and install ``Qlib`` as follows.
    ```bash
    git clone https://github.com/microsoft/qlib.git && cd qlib
    pip install .  # `pip install -e .[dev]` is recommended for development. check details in docs/developer/code_standard_and_dev_guide.rst
    ```
  **Note**:  You can install Qlib with `python setup.py install` as well. But it is not the recommended approach. It will skip `pip` and cause obscure problems. For example, **only** the command ``pip install .`` **can** overwrite the stable version installed by ``pip install pyqlib``, while the command ``python setup.py install`` **can't**.

**Tips**: If you fail to install `Qlib` or run the examples in your environment,  comparing your steps and the [CI workflow](.github/workflows/test_qlib_from_source.yml) may help you find the problem.

**Tips for Mac**: If you are using Mac with M1, you might encounter issues in building the wheel for LightGBM, which is due to missing dependencies from OpenMP. To solve the problem, install openmp first with ``brew install libomp`` and then run ``pip install .`` to build it successfully.

## Data Preparation
❗ Due to more restrict data security policy. The offical dataset is disabled temporarily. You can try [this data source](https://github.com/chenditc/investment_data/releases) contributed by the community.
Here is an example to download the data updated on 20220720.
```bash
wget https://github.com/chenditc/investment_data/releases/download/20220720/qlib_bin.tar.gz
mkdir -p ~/.qlib/qlib_data/cn_data
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=2
rm -f qlib_bin.tar.gz
```

The official dataset below will resume in short future.

----

Load and prepare data by running the following code:

### Get with module
  ```bash
  # get 1d data
  python -m qlib.run.get_data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

  # get 1min data
  python -m qlib.run.get_data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

  ```

### Get from source

  ```bash
  # get 1d data
  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

  # get 1min data
  python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min

  ```

This dataset is created by public data collected by [crawler scripts](scripts/data_collector/), which have been released in
the same repository.
Users could create the same dataset with it. [Description of dataset](https://github.com/microsoft/qlib/tree/main/scripts/data_collector#description-of-dataset)

*Please pay **ATTENTION** that the data is collected from [Yahoo Finance](https://finance.yahoo.com/lookup), and the data might not be perfect.
We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to the [related document](https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-format)*.

### Automatic update of daily frequency data (from yahoo finance)
  > This step is *Optional* if users only want to try their models and strategies on history data.
  >
  > It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically.
  >
  > **NOTE**: Users can't incrementally  update data based on the offline data provided by Qlib(some fields are removed to reduce the data size). Users should use [yahoo collector](https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#automatic-update-of-daily-frequency-datafrom-yahoo-finance) to download Yahoo data from scratch and then incrementally update it.
  >
  > For more information, please refer to: [yahoo collector](https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#automatic-update-of-daily-frequency-datafrom-yahoo-finance)

  * Automatic update of data to the "qlib" directory each trading day(Linux)
      * use *crontab*: `crontab -e`
      * set up timed tasks:

        ```
        * * * * 1-5 python <script path> update_data_to_bin --qlib_data_1d_dir <user data dir>
        ```
        * **script path**: *scripts/data_collector/yahoo/collector.py*

  * Manual update of data
      ```
      python scripts/data_collector/yahoo/collector.py update_data_to_bin --qlib_data_1d_dir <user data dir> --trading_date <start date> --end_date <end date>
      ```
      * *trading_date*: start of trading day
      * *end_date*: end of trading day(not included)

## Auto Quant Research Workflow
Qlib provides a tool named `qrun` to run the whole workflow automatically (including building dataset, training models, backtest and evaluation). You can start an auto quant research workflow and have a graphical reports analysis according to the following steps:

1. Quant Research Workflow: Run  `qrun` with lightgbm workflow config ([workflow_config_lightgbm_Alpha158.yaml](examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml) as following.
    ```bash
      cd examples  # Avoid running program under the directory contains `qlib`
      qrun benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
    ```
    If users want to use `qrun` under debug mode, please use the following command:
    ```bash
    python -m pdb qlib/workflow/cli.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml
    ```
    The result of `qrun` is as follows, please refer to [Intraday Trading](https://qlib.readthedocs.io/en/latest/component/backtest.html) for more details about the result.

    ```bash

    'The following are analysis results of the excess return without cost.'
                           risk
    mean               0.000708
    std                0.005626
    annualized_return  0.178316
    information_ratio  1.996555
    max_drawdown      -0.081806
    'The following are analysis results of the excess return with cost.'
                           risk
    mean               0.000512
    std                0.005626
    annualized_return  0.128982
    information_ratio  1.444287
    max_drawdown      -0.091078
    ```
    Here are detailed documents for `qrun` and [workflow](https://qlib.readthedocs.io/en/latest/component/workflow.html).

2. Graphical Reports Analysis: Run `examples/workflow_by_code.ipynb` with `jupyter notebook` to get graphical reports
    - Forecasting signal (model prediction) analysis
      - Cumulative Return of groups
      ![Cumulative Return](http://fintech.msra.cn/images_v070/analysis/analysis_model_cumulative_return.png?v=0.1)
      - Return distribution
      ![long_short](http://fintech.msra.cn/images_v070/analysis/analysis_model_long_short.png?v=0.1)
      - Information Coefficient (IC)
      ![Information Coefficient](http://fintech.msra.cn/images_v070/analysis/analysis_model_IC.png?v=0.1)
      ![Monthly IC](http://fintech.msra.cn/images_v070/analysis/analysis_model_monthly_IC.png?v=0.1)
      ![IC](http://fintech.msra.cn/images_v070/analysis/analysis_model_NDQ.png?v=0.1)
      - Auto Correlation of forecasting signal (model prediction)
      ![Auto Correlation](http://fintech.msra.cn/images_v070/analysis/analysis_model_auto_correlation.png?v=0.1)

    - Portfolio analysis
      - Backtest return
      ![Report](http://fintech.msra.cn/images_v070/analysis/report.png?v=0.1)

   - [Explanation](https://qlib.readthedocs.io/en/latest/component/report.html) of above results

## Building Customized Quant Research Workflow by Code
The automatic workflow may not suit the research workflow of all Quant researchers. To support a flexible Quant research workflow, Qlib also provides a modularized interface to allow researchers to build their own workflow by code. [Here](examples/workflow_by_code.ipynb) is a demo for customized Quant research workflow by code.

# Main Challenges & Solutions in Quant Research
Quant investment is a very unique scenario with lots of key challenges to be solved.
Currently, Qlib provides some solutions for several of them.

## Forecasting: Finding Valuable Signals/Patterns
Accurate forecasting of the stock price trend is a very important part to construct profitable portfolios.
However, huge amount of data with various formats in the financial market which make it challenging to build forecasting models.

An increasing number of SOTA Quant research works/papers, which focus on building forecasting models to mine valuable signals/patterns in complex financial data, are released in `Qlib`

### [Quant Model (Paper) Zoo](examples/benchmarks)

Here is a list of models built on `Qlib`.
- [GBDT based on XGBoost (Tianqi Chen, et al. KDD 2016)](examples/benchmarks/XGBoost/)
- [GBDT based on LightGBM (Guolin Ke, et al. NIPS 2017)](examples/benchmarks/LightGBM/)
- [GBDT based on Catboost (Liudmila Prokhorenkova, et al. NIPS 2018)](examples/benchmarks/CatBoost/)
- [MLP based on pytorch](examples/benchmarks/MLP/)
- [LSTM based on pytorch (Sepp Hochreiter, et al. Neural computation 1997)](examples/benchmarks/LSTM/)
- [GRU based on pytorch (Kyunghyun Cho, et al. 2014)](examples/benchmarks/GRU/)
- [ALSTM based on pytorch (Yao Qin, et al. IJCAI 2017)](examples/benchmarks/ALSTM)
- [GATs based on pytorch (Petar Velickovic, et al. 2017)](examples/benchmarks/GATs/)
- [SFM based on pytorch (Liheng Zhang, et al. KDD 2017)](examples/benchmarks/SFM/)
- [TFT based on tensorflow (Bryan Lim, et al. International Journal of Forecasting 2019)](examples/benchmarks/TFT/)
- [TabNet based on pytorch (Sercan O. Arik, et al. AAAI 2019)](examples/benchmarks/TabNet/)
- [DoubleEnsemble based on LightGBM (Chuheng Zhang, et al. ICDM 2020)](examples/benchmarks/DoubleEnsemble/)
- [TCTS based on pytorch (Xueqing Wu, et al. ICML 2021)](examples/benchmarks/TCTS/)
- [Transformer based on pytorch (Ashish Vaswani, et al. NeurIPS 2017)](examples/benchmarks/Transformer/)
- [Localformer based on pytorch (Juyong Jiang, et al.)](examples/benchmarks/Localformer/)
- [TRA based on pytorch (Hengxu, Dong, et al. KDD 2021)](examples/benchmarks/TRA/)
- [TCN based on pytorch (Shaojie Bai, et al. 2018)](examples/benchmarks/TCN/)
- [ADARNN based on pytorch (YunTao Du, et al. 2021)](examples/benchmarks/ADARNN/)
- [ADD based on pytorch (Hongshun Tang, et al.2020)](examples/benchmarks/ADD/)
- [IGMTF based on pytorch (Wentao Xu, et al.2021)](examples/benchmarks/IGMTF/)
- [HIST based on pytorch (Wentao Xu, et al.2021)](examples/benchmarks/HIST/)
- [KRNN based on pytorch](examples/benchmarks/KRNN/)
- [Sandwich based on pytorch](examples/benchmarks/Sandwich/)

Your PR of new Quant models is highly welcomed.

The performance of each model on the `Alpha158` and `Alpha360` datasets can be found [here](examples/benchmarks/README.md).

### Run a single model
All the models listed above are runnable with ``Qlib``. Users can find the config files we provide and some details about the model through the [benchmarks](examples/benchmarks) folder. More information can be retrieved at the model files listed above.

`Qlib` provides three different ways to run a single model, users can pick the one that fits their cases best:
- Users can use the tool `qrun` mentioned above to run a model's workflow based from a config file.
- Users can create a `workflow_by_code` python script based on the [one](examples/workflow_by_code.py) listed in the `examples` folder.

- Users can use the script [`run_all_model.py`](examples/run_all_model.py) listed in the `examples` folder to run a model. Here is an example of the specific shell command to be used: `python run_all_model.py run --models=lightgbm`, where the `--models` arguments can take any number of models listed above(the available models can be found  in [benchmarks](examples/benchmarks/)). For more use cases, please refer to the file's [docstrings](examples/run_all_model.py).
    - **NOTE**: Each baseline has different environment dependencies, please make sure that your python version aligns with the requirements(e.g. TFT only supports Python 3.6~3.7 due to the limitation of `tensorflow==1.15.0`)

### Run multiple models
`Qlib` also provides a script [`run_all_model.py`](examples/run_all_model.py) which can run multiple models for several iterations. (**Note**: the script only support *Linux* for now. Other OS will be supported in the future. Besides, it doesn't support parallel running the same model for multiple times as well, and this will be fixed in the future development too.)

The script will create a unique virtual environment for each model, and delete the environments after training. Thus, only experiment results such as `IC` and `backtest` results will be generated and stored.

Here is an example of running all the models for 10 iterations:
```python
python run_all_model.py run 10
```

It also provides the API to run specific models at once. For more use cases, please refer to the file's [docstrings](examples/run_all_model.py).

## [Adapting to Market Dynamics](examples/benchmarks_dynamic)

Due to the non-stationary nature of the environment of the financial market, the data distribution may change in different periods, which makes the performance of models build on training data decays in the future test data.
So adapting the forecasting models/strategies to market dynamics is very important to the model/strategies' performance.

Here is a list of solutions built on `Qlib`.
- [Rolling Retraining](examples/benchmarks_dynamic/baseline/)
- [DDG-DA on pytorch (Wendi, et al. AAAI 2022)](examples/benchmarks_dynamic/DDG-DA/)

##  Reinforcement Learning: modeling continuous decisions
Qlib now supports reinforcement learning, a feature designed to model continuous investment decisions. This functionality assists investors in optimizing their trading strategies by learning from interactions with the environment to maximize some notion of cumulative reward.

Here is a list of solutions built on `Qlib` categorized by scenarios.

### [RL for order execution](examples/rl_order_execution)
[Here](https://qlib.readthedocs.io/en/latest/component/rl/overall.html#order-execution) is the introduction of this scenario.  All the methods below are compared [here](examples/rl_order_execution).
- [TWAP](examples/rl_order_execution/exp_configs/backtest_twap.yml)
- [PPO: "An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization", IJCAL 2020](examples/rl_order_execution/exp_configs/backtest_ppo.yml)
- [OPDS: "Universal Trading for Order Execution with Oracle Policy Distillation", AAAI 2021](examples/rl_order_execution/exp_configs/backtest_opds.yml)

# Quant Dataset Zoo
Dataset plays a very important role in Quant. Here is a list of the datasets built on `Qlib`:

| Dataset                                    | US Market | China Market |
| --                                         | --        | --           |
| [Alpha360](./qlib/contrib/data/handler.py) |  √        |  √           |
| [Alpha158](./qlib/contrib/data/handler.py) |  √        |  √           |

[Here](https://qlib.readthedocs.io/en/latest/advanced/alpha.html) is a tutorial to build dataset with `Qlib`.
Your PR to build new Quant dataset is highly welcomed.

# Learning Framework
Qlib is high customizable and a lot of its components are learnable.
The learnable components are instances of `Forecast Model` and `Trading Agent`. They are learned based on the `Learning Framework` layer and then applied to multiple scenarios in `Workflow` layer.
The learning framework leverages the `Workflow` layer as well(e.g. sharing `Information Extractor`, creating environments based on `Execution Env`).

Based on learning paradigms, they can be categorized into reinforcement learning and supervised learning.
- For supervised learning, the detailed docs can be found [here](https://qlib.readthedocs.io/en/latest/component/model.html).
- For reinforcement learning, the detailed docs can be found [here](https://qlib.readthedocs.io/en/latest/component/rl.html). Qlib's RL learning framework leverages `Execution Env` in `Workflow` layer to create environments.  It's worth noting that `NestedExecutor` is supported as well. This empowers users to optimize different level of strategies/models/agents together (e.g. optimizing an order execution strategy for a specific portfolio management strategy).

# More About Qlib
If you want to have a quick glance at the most frequently used components of qlib, you can try notebooks [here](examples/tutorial/).

The detailed documents are organized in [docs](docs/).
[Sphinx](http://www.sphinx-doc.org) and the readthedocs theme is required to build the documentation in html formats.
```bash
cd docs/
conda install sphinx sphinx_rtd_theme -y
# Otherwise, you can install them with pip
# pip install sphinx sphinx_rtd_theme
make html
```
You can also view the [latest document](http://qlib.readthedocs.io/) online directly.

Qlib is in active and continuing development. Our plan is in the roadmap, which is managed as a [github project](https://github.com/microsoft/qlib/projects/1).

# Offline Mode and Online Mode
The data server of Qlib can either deployed as `Offline` mode or `Online` mode. The default mode is offline mode.

Under `Offline` mode, the data will be deployed locally.

Under `Online` mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too. The documents of the online mode can be found in [Qlib-Server](https://qlib-server.readthedocs.io/). The online mode can be deployed automatically with [Azure CLI based scripts](https://qlib-server.readthedocs.io/en/latest/build.html#one-click-deployment-in-azure). The source code of online data server can be found in [Qlib-Server repository](https://github.com/microsoft/qlib-server).

## Performance of Qlib Data Server
The performance of data processing is important to data-driven methods like AI technologies. As an AI-oriented platform, Qlib provides a solution for data storage and data processing. To demonstrate the performance of Qlib data server, we
compare it with several other data storage solutions.

We evaluate the performance of several storage solutions by finishing the same task,
which creates a dataset (14 features/factors) from the basic OHLCV daily data of a stock market (800 stocks each day from 2007 to 2020). The task involves data queries and processing.

|                         | HDF5      | MySQL     | MongoDB   | InfluxDB  | Qlib -E -D  | Qlib +E -D   | Qlib +E +D  |
| --                      | ------    | ------    | --------  | --------- | ----------- | ------------ | ----------- |
| Total (1CPU) (seconds)  | 184.4±3.7 | 365.3±7.5 | 253.6±6.7 | 368.2±3.6 | 147.0±8.8   | 47.6±1.0     | **7.4±0.3** |
| Total (64CPU) (seconds) |           |           |           |           | 8.8±0.6     | **4.2±0.2**  |             |
* `+(-)E` indicates with (out) `ExpressionCache`
* `+(-)D` indicates with (out) `DatasetCache`

Most general-purpose databases take too much time to load data. After looking into the underlying implementation, we find that data go through too many layers of interfaces and unnecessary format transformations in general-purpose database solutions.
Such overheads greatly slow down the data loading process.
Qlib data are stored in a compact format, which is efficient to be combined into arrays for scientific computation.

# Related Reports
- [Guide To Qlib: Microsoft’s AI Investment Platform](https://analyticsindiamag.com/qlib/)
- [微软也搞AI量化平台？还是开源的！](https://mp.weixin.qq.com/s/47bP5YwxfTp2uTHjUBzJQQ)
- [微矿Qlib：业内首个AI量化投资开源平台](https://mp.weixin.qq.com/s/vsJv7lsgjEi-ALYUz4CvtQ)

# Contact Us
- If you have any issues, please create issue [here](https://github.com/microsoft/qlib/issues/new/choose) or send messages in [gitter](https://gitter.im/Microsoft/qlib).
- If you want to make contributions to `Qlib`, please [create pull requests](https://github.com/microsoft/qlib/compare).
- For other reasons, you are welcome to contact us by email([qlib@microsoft.com](mailto:qlib@microsoft.com)).
  - We are recruiting new members(both FTEs and interns), your resumes are welcome!

Join IM discussion groups:
|[Gitter](https://gitter.im/Microsoft/qlib)|
|----|
|![image](http://fintech.msra.cn/images_v070/qrcode/gitter_qr.png)|

# Contributing
We appreciate all contributions and thank all the contributors!

Before we released Qlib as an open-source project on Github in Sep 2020, Qlib is an internal project in our group. Unfortunately, the internal commit history is not kept. A lot of members in our group have also contributed a lot to Qlib, which includes Ruihua Wang, Yinda Zhang, Haisu Yu, Shuyu Wang, Bochen Pang, and [Dong Zhou](https://github.com/evanzd/evanzd). Especially thanks to [Dong Zhou](https://github.com/evanzd/evanzd) due to his initial version of Qlib.

## Guidance

This project welcomes contributions and suggestions.
**Here are some
[code standards and development guidance](docs/developer/code_standard_and_dev_guide.rst) for submiting a pull request.**

Making contributions is not a hard thing. Solving an issue(maybe just answering a question raised in [issues list](https://github.com/microsoft/qlib/issues) or [gitter](https://gitter.im/Microsoft/qlib)), fixing/issuing a bug, improving the documents and even fixing a typo are important contributions to Qlib.

For example, if you want to contribute to Qlib's document/code, you can follow the steps in the figure below.

If you don't know how to start to contribute, you can refer to the following examples.
| Type | Examples |
| -- | -- |
| Solving issues | [Answer a question](https://github.com/microsoft/qlib/issues/749);  [issuing](https://github.com/microsoft/qlib/issues/765) or [fixing](https://github.com/microsoft/qlib/pull/792) a bug |
| Docs | [Improve docs quality](https://github.com/microsoft/qlib/pull/797/files) ;  [Fix a typo](https://github.com/microsoft/qlib/pull/774) |
| Feature |  Implement a [requested feature](https://github.com/microsoft/qlib/projects) like [this](https://github.com/microsoft/qlib/pull/754); [Refactor interfaces](https://github.com/microsoft/qlib/pull/539/files) |
| Dataset | [Add a dataset](https://github.com/microsoft/qlib/pull/733) |
| Models |  [Implement a new model](https://github.com/microsoft/qlib/pull/689), [some instructions to contribute models](https://github.com/microsoft/qlib/tree/main/examples/benchmarks#contributing) |

[Good first issues](https://github.com/microsoft/qlib/labels/good%20first%20issue) are labelled to indicate that they are easy to start your contributions.

You can find some impefect implementation in Qlib by  `rg 'TODO|FIXME' qlib`

If you would like to become one of Qlib's maintainers to contribute more (e.g. help merge PR, triage issues), please contact us by email([qlib@microsoft.com](mailto:qlib@microsoft.com)).  We are glad to help to upgrade your permission.

## Licence
Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the right to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.


.. raw:: html

   <img align="right" width="auto" height="auto" src="https://www.elastic.co/static-res/images/elastic-logo-200.png">

Elasticsearch Python Client
===========================

.. image:: https://img.shields.io/pypi/v/elasticsearch
   :target: https://pypi.org/project/elasticsearch

.. image:: https://img.shields.io/conda/vn/conda-forge/elasticsearch?color=blue
   :target: https://anaconda.org/conda-forge/elasticsearch

.. image:: https://static.pepy.tech/badge/elasticsearch
   :target: https://pepy.tech/project/elasticsearch?versions=*

.. image:: https://badge.buildkite.com/68e22afcb2ea8f6dcc20834e3a5b5ab4431beee33d3bd751f3.svg
   :target: https://buildkite.com/elastic/elasticsearch-py-integration-tests

.. image:: https://readthedocs.org/projects/elasticsearch-py/badge/?version=latest&style=flat
   :target: https://elasticsearch-py.readthedocs.io

*The official Python client for Elasticsearch.*

Features
--------

* Translating basic Python data types to and from JSON
* Configurable automatic discovery of cluster nodes
* Persistent connections
* Load balancing (with pluggable selection strategy) across available nodes
* Failed connection penalization (time based - failed connections won't be
  retried until a timeout is reached)
* Support for TLS and HTTP authentication
* Thread safety across requests
* Pluggable architecture
* Helper functions for idiomatically using APIs together

Installation
------------

`Download the latest version of Elasticsearch <https://www.elastic.co/downloads/elasticsearch>`_
or
`sign-up <https://cloud.elastic.co/registration?elektra=en-ess-sign-up-page>`_
for a free trial of Elastic Cloud.

Refer to the `Installation section <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_installation>`_
of the getting started documentation.

Connecting
----------

Refer to the `Connecting section <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_connecting>`_
of the getting started documentation.

Usage
-----

* `Creating an index <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_creating_an_index>`_
* `Indexing a document <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_indexing_documents>`_
* `Getting documents <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_getting_documents>`_
* `Searching documents <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_searching_documents>`_
* `Updating documents <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_updating_documents>`_
* `Deleting documents <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_deleting_documents>`_
* `Deleting an index <https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_deleting_an_index>`_

Compatibility
-------------

Language clients are forward compatible; meaning that the clients support
communicating with greater or equal minor versions of Elasticsearch without
breaking. It does not mean that the clients automatically support new features
of newer Elasticsearch versions; it is only possible after a release of a new
client version. For example, a 8.12 client version won't automatically support
the new features of the 8.13 version of Elasticsearch, the 8.13 client version
is required for that. Elasticsearch language clients are only backwards
compatible with default distributions and without guarantees made.

+-----------------------+-------------------------+-----------+
| Elasticsearch version | elasticsearch-py branch | Supported |
+=======================+=========================+===========+
| main                  | main                    |           |
+-----------------------+-------------------------+-----------+
| 8.x                   | 8.x                     | 8.x       |
+-----------------------+-------------------------+-----------+
| 7.x                   | 7.x                     | 7.17      |
+-----------------------+-------------------------+-----------+

If you have a need to have multiple versions installed at the same time older
versions are also released as ``elasticsearch7`` and ``elasticsearch8``.

Documentation
-------------

Documentation for the client is `available on elastic.co`_ and `Read the Docs`_.

.. _available on elastic.co: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html
.. _Read the Docs: https://elasticsearch-py.readthedocs.io

Feedback 🗣
-----------

The engineering team here at Elastic is looking for developers to participate in
research and feedback sessions to learn more about how you use our Python client and
what improvements we can make to their design and your workflow. If you're interested in
sharing your insights into developer experience and language client design, please fill
out this `short form`_. Depending on the number of responses we get, we may either
contact you for a 1:1 conversation or a focus group with other developers who use the
same client. Thank you in advance - your feedback is crucial to improving the user
experience for all Elasticsearch developers!

.. _short form: https://forms.gle/bYZwDQXijfhfwshn9

License
-------

Copyright 2023 Elasticsearch B.V. Licensed under the Apache License, Version 2.0.


.. image:: docs/_static/aiortc.svg
   :width: 120px
   :alt: aiortc

.. image:: https://img.shields.io/pypi/l/aiortc.svg
   :target: https://pypi.python.org/pypi/aiortc
   :alt: License

.. image:: https://img.shields.io/pypi/v/aiortc.svg
   :target: https://pypi.python.org/pypi/aiortc
   :alt: Version

.. image:: https://img.shields.io/pypi/pyversions/aiortc.svg
   :target: https://pypi.python.org/pypi/aiortc
   :alt: Python versions

.. image:: https://github.com/aiortc/aiortc/workflows/tests/badge.svg
   :target: https://github.com/aiortc/aiortc/actions
   :alt: Tests

.. image:: https://img.shields.io/codecov/c/github/aiortc/aiortc.svg
   :target: https://codecov.io/gh/aiortc/aiortc
   :alt: Coverage

.. image:: https://readthedocs.org/projects/aiortc/badge/?version=latest
   :target: https://aiortc.readthedocs.io/
   :alt: Documentation

What is ``aiortc``?
-------------------

``aiortc`` is a library for `Web Real-Time Communication (WebRTC)`_ and
`Object Real-Time Communication (ORTC)`_ in Python. It is built on top of
``asyncio``, Python's standard asynchronous I/O framework.

The API closely follows its Javascript counterpart while using pythonic
constructs:

- promises are replaced by coroutines
- events are emitted using ``pyee.EventEmitter``

To learn more about ``aiortc`` please `read the documentation`_.

.. _Web Real-Time Communication (WebRTC): https://webrtc.org/
.. _Object Real-Time Communication (ORTC): https://ortc.org/
.. _read the documentation: https://aiortc.readthedocs.io/en/latest/

Why should I use ``aiortc``?
----------------------------

The main WebRTC and ORTC implementations are either built into web browsers,
or come in the form of native code. While they are extensively battle tested,
their internals are complex and they do not provide Python bindings.
Furthermore they are tightly coupled to a media stack, making it hard to plug
in audio or video processing algorithms.

In contrast, the ``aiortc`` implementation is fairly simple and readable. As
such it is a good starting point for programmers wishing to understand how
WebRTC works or tinker with its internals. It is also easy to create innovative
products by leveraging the extensive modules available in the Python ecosystem.
For instance you can build a full server handling both signaling and data
channels or apply computer vision algorithms to video frames using OpenCV.

Furthermore, a lot of effort has gone into writing an extensive test suite for
the ``aiortc`` code to ensure best-in-class code quality.

Implementation status
---------------------

``aiortc`` allows you to exchange audio, video and data channels and
interoperability is regularly tested against both Chrome and Firefox. Here are
some of its features:

- SDP generation / parsing
- Interactive Connectivity Establishment, with half-trickle and mDNS support
- DTLS key and certificate generation
- DTLS handshake, encryption / decryption (for SCTP)
- SRTP keying, encryption and decryption for RTP and RTCP
- Pure Python SCTP implementation
- Data Channels
- Sending and receiving audio (Opus / PCMU / PCMA)
- Sending and receiving video (VP8 / H.264)
- Bundling audio / video / data channels
- RTCP reports, including NACK / PLI to recover from packet loss

Installing
----------

The easiest way to install ``aiortc`` is to run:

.. code:: bash

    pip install aiortc

Building from source
--------------------

If there are no wheels for your system or if you wish to build aiortc from
source you will need a couple of libraries installed on your system:

- Opus for audio encoding / decoding
- LibVPX for video encoding / decoding

Linux
.....

On Debian/Ubuntu run:

.. code:: bash

    apt install libopus-dev libvpx-dev

OS X
....

On OS X run:

.. code:: bash

    brew install opus libvpx

License
-------

``aiortc`` is released under the `BSD license`_.

.. _BSD license: https://aiortc.readthedocs.io/en/latest/license.html


Interactive Tutorials
=====================

This is the open source repository for the free interactive tutorial websites:

* [learnpython.org](https://www.learnpython.org)
* [learnjavaonline.org](https://www.learnjavaonline.org)
* [learn-html.org](https://www.learn-html.org)
* [learn-c.org](https://www.learn-c.org)
* [learn-cpp.org](https://www.learn-cpp.org)
* [learn-js.org](https://www.learn-js.org)
* [learn-php.org](https://www.learn-php.org)
* [learnshell.org](https://www.learnshell.org)
* [learncs.org](https://www.learncs.org)
* [learn-perl.org](https://www.learn-perl.org)
* [learnrubyonline.org](https://www.learnrubyonline.org)
* [learn-golang.org](https://www.learn-golang.org)
* [learnrust.org](https://learnrust.org)

Please feel free to contribute your tutorials or exercises by sending a pull request and adding yourself on the list.

To run locally, first create a new file called `app.env` based on `app.env.example`.
(you don't need working credentials to run the website, just to execute code).

### Running with Docker

Execute:

    DEFAULT_DOMAIN=learnpython.org make run

This command will run learnpython.org website by default using Docker Compose.

To run a specific website, run with the DEFAULT_DOMAIN option set, as follows:

    DEFAULT_DOMAIN=learn-js.org make run

By default, the server process will run at http://localhost:5000.

The web server will locally compile and load all Markdown files into memory. The docker needs to be rebuilt upon any change in the Python code or the Markdown code.

### Running without Docker (locally)

To run the development server outside of a Docker, run the following command once:

    make build-local

And then run:

    DEFAULT_DOMAIN=learn-js.org make run-local

### IDEOne Credentials

To make the IDEOne execution API work locally, you must obtain a username and password,
and add the credentials to a file called app.env.
(TBD: add an option to use the remote version)

Contributors
============
- superreg
- JBoy Advance
- qwong95
- AddaxSoft
- derco0n
- ...<perhaps you>...

This project is supported by:


[![PyPI](https://img.shields.io/pypi/v/napalm.svg)](https://pypi.python.org/pypi/napalm)
[![PyPI versions](https://img.shields.io/pypi/pyversions/napalm.svg)](https://pypi.python.org/pypi/napalm)
[![Actions Build](https://github.com/napalm-automation/napalm/actions/workflows/commit.yaml/badge.svg?branch=develop)](https://github.com/napalm-automation/napalm/actions/workflows/commit.yaml)
[![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

NAPALM
======
NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different router vendor devices using a unified API.

![NAPALM logo](static/logo.png?raw=true "NAPALM logo")

NAPALM supports several methods to connect to the devices, to manipulate configurations or to retrieve data.

Supported Network Operating Systems
-----------------------------------

Please check the following [link](https://napalm.readthedocs.io/en/latest/support/index.html) to see which devices are supported. Make sure you understand the [caveats](https://napalm.readthedocs.io/en/latest/support/index.html#caveats).

Documentation
=============

Before using the library, please read the documentation at: [Read the Docs](https://napalm.readthedocs.io)

You can also watch a [live demo](https://youtu.be/93q-dHC0u0I) of NAPALM to see what it is and what it can do for you.

Install
=======

```
pip install napalm
```

*Note*: Beginning with release 5.0.0 and later, NAPALM offers support for
Python 3.8+ only.

*Note*: Beginning with release 4.0.0 and later, NAPALM offers support for
Python 3.7+ only.

*Note*: Beginning with release 3.0.0 and later, NAPALM offers support for
Python 3.6+ only.

Upgrading
=========

We plan to upgrade napalm as fast as possible. Adding new methods and bugfixes. To upgrade napalm it's a simple as repeating the steps you performed while installing but adding the `-U` flag. For example:

```
pip install napalm -U
```

We will be posting news on our slack channel and on Twitter.

Automation Frameworks
======================

Due to its flexibility, NAPALM can be integrated in widely used automation frameworks.

Ansible
-------

Please check [napalm-ansible](https://github.com/napalm-automation/napalm-ansible) for existing Ansible modules leveraging the NAPALM API. Make sure you read the documentation and you understand how it works before trying to use it.

SaltStack
---------

Beginning with release code named `Carbon` (2016.11), [NAPALM is fully integrated](https://mirceaulinic.net/2016-11-30-salt-carbon-released/) in SaltStack - no additional modules required. For setup recommendations, please see [napalm-salt](https://github.com/napalm-automation/napalm-salt). For documentation and usage examples, you can check the modules documentation, starting from the [release notes](https://docs.saltstack.com/en/develop/topics/releases/2016.11.0.html#network-automation-napalm) and [this blog post](https://mirceaulinic.net/2016-11-17-network-orchestration-with-salt-and-napalm/).

StackStorm
----------

NAPALM is usable from StackStorm using the [NAPALM integration pack](https://github.com/StackStorm-Exchange/stackstorm-napalm). See that repository for instructions on installing and configuring the pack to work with StackStorm. General StackStorm documentation can be found at [https://docs.stackstorm.com/](https://docs.stackstorm.com/), and StackStorm can be easily spun up for testing using [Vagrant](https://github.com/StackStorm/st2vagrant) or [Docker](https://github.com/StackStorm/st2-docker).

Contact
=======

Slack
-----

Slack is probably the easiest way to get help with NAPALM. You can find us in the channel `napalm` on the [network.toCode()](https://networktocode.herokuapp.com/) team.

FAQ
---

If you have any issues using NAPALM or encounter any errors, before submitting any questions (directly by email or on Slack), please go through the following checklist:

- Make sure you have the latest release installed. We release very often, so upgrading to the latest version might help in many cases.
- Double check you are able to access the device using the credentials provided.
- Does your device meet the minimum [requirements](http://napalm.readthedocs.io/en/latest/support/index.html#general-support-matrix)?
- Some operating systems have some specific [constraints](http://napalm.readthedocs.io/en/latest/support/index.html#caveats). (e.g. have you enabled the XML agent on IOS-XR, or the NXAPI feature on NXOS?)
- Are you able to connect to the device using NAPALM? Check using napalm CLI to get_facts:

```bash
$ napalm --vendor VENDOR --user USERNAME --password PASSWORD --optional_args OPTIONAL_ARGS HOSTNAME call get_facts
```

Where vendor, username, password and hostname are mandatory. [Optional arguments](http://napalm.readthedocs.io/en/latest/support/index.html#optional-arguments) are specified as comma separated values.

Example:

```bash
$ napalm --vendor junos --user napalm --password dbejmujz --optional_args 'port=12202, config_lock=False' edge01.bjm01 call get_facts
```

In case you have any errors, please review the steps above - this looks like a problem with your environment setup.

In order to get help faster, when submitting a bug/error make sure to include all the details requested.

News
====

Blog Posts
----------

* [NAPALM, Ansible, and Cisco IOS](https://pynet.twb-tech.com/blog/automation/napalm-ios.html) by Kirk Byers
* [Adding Cisco IOS support to NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support)](https://projectme10.wordpress.com/2015/12/07/adding-cisco-ios-support-to-napalm-network-automation-and-programmability-abstraction-layer-with-multivendor-support/) by Gabriele Gerbino
* [Network orchestration with Salt and NAPALM](https://mirceaulinic.net/2016-11-17-network-orchestration-with-salt-and-napalm/) by Mircea Ulinic
* [Network Configuration Consistency with StackStorm and NAPALM](https://stackstorm.com/2017/04/11/ensuring-network-configuration-consistency-stackstorm-napalm/) by Matt Oswalt

Presentations
-------------

* [NANOG 64 Presentation & Demo](https://youtu.be/93q-dHC0u0I) by David Barroso and Elisa Jasinska
* [Netnod Autumn Meeting 2015 Presentation](https://www.netnod.se/sites/default/files/NAPALM-david_barroso-Netnodautumnmeeting2015.pdf) by David Barroso
* [Automating IXP Device Configurations with Ansible at the Euro-IX Forum](https://www.euro-ix.net/m/uploads/2015/10/26/euroix-berlin-v2.pdf) by Elisa Jasinska
* [Network Automation with Salt and NAPALM at NANOG 68](https://www.nanog.org/sites/default/files/NANOG68%20Network%20Automation%20with%20Salt%20and%20NAPALM%20Mircea%20Ulinic%20Cloudflare%20(1).pdf); [video](https://www.youtube.com/watch?v=gV2918bH5_c); [recorded demo](https://www.youtube.com/watch?v=AqBk5fM7qZ0) by Mircea Ulinic

Podcasts
--------

* [NAPALM: Integrating Ansible with Network Devices on Software Gone Wild](http://blog.ipspace.net/2015/06/napalm-integrating-ansible-with-network.html) with David Barroso and Elisa Jasinska

Authors
=======
 * David Barroso ([dbarrosop@dravetech.com](mailto:dbarrosop@dravetech.com))
 * Mircea Ulinic ([ping@mirceaulinic.net](mailto:ping@mirceaulinic.net))
 * Kirk Byers ([ktbyers@twb-tech.com](mailto:ktbyers@twb-tech.com))
 * Elisa Jasinska ([elisa@bigwaveit.org](mailto:elisa@bigwaveit.org))
 * Many others, check the [contributors](https://github.com/napalm-automation/napalm/graphs/contributors) page for details.

Thanks
======

This project is maintained by David Barroso, Mircea Ulinic, and Kirk Byers and a set of other contributors.

Originally it was hosted by the [Spotify][spotify] organization but due to the many contributions received by third parties we agreed creating a dedicated organization for NAPALM and give a big thanks to [Spotify][spotify] for the support.

[spotify]: http://www.spotify.com
[bigwave]: http://bigwaveit.org/


|         |                                                                                |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| CI/CD   | [![Tests](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml) [![types - Mypy](https://img.shields.io/badge/types-Mypy-blue.svg)](https://github.com/python/mypy) [![Coverage Status](https://coveralls.io/repos/github/deepset-ai/haystack/badge.svg?branch=main)](https://coveralls.io/github/deepset-ai/haystack?branch=main) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) |
| Docs    | [![Website](https://img.shields.io/website?label=documentation&up_message=online&url=https%3A%2F%2Fdocs.haystack.deepset.ai)](https://docs.haystack.deepset.ai)                                                                                |
| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) |
| Meta    | [![Discord](https://img.shields.io/discord/993534733298450452?logo=discord)](https://discord.gg/haystack) [![Twitter Follow](https://img.shields.io/twitter/follow/haystack_ai)](https://twitter.com/haystack_ai)                                                                                |

[Haystack](https://haystack.deepset.ai/) is an end-to-end LLM framework that allows you to build applications powered by
LLMs, Transformer models, vector search and more. Whether you want to perform retrieval-augmented generation (RAG),
document search, question answering or answer generation, Haystack can orchestrate state-of-the-art embedding models
and LLMs into pipelines to build end-to-end NLP applications and solve your use case.

## Installation

The simplest way to get Haystack is via pip:

```sh
pip install haystack-ai
```

Haystack supports multiple installation methods including Docker images. For a comprehensive guide please refer
to the [documentation](https://docs.haystack.deepset.ai/v2.0/docs/installation).

## Documentation

If you're new to the project, check out ["What is Haystack?"](https://haystack.deepset.ai/overview/intro) then go
through the ["Get Started Guide"](https://haystack.deepset.ai/overview/quick-start) and build your first LLM application
in a matter of minutes. Keep learning with the [tutorials](https://haystack.deepset.ai/tutorials?v=2.0). For more advanced
use cases, or just to get some inspiration, you can browse our Haystack recipes in the
[Cookbook](https://github.com/deepset-ai/haystack-cookbook).

At any given point, hit the [documentation](https://docs.haystack.deepset.ai/v2.0/docs/intro) to learn more about Haystack, what can it do for you and the technology behind.

## Features

> [!IMPORTANT]
> **You are currently looking at the readme of Haystack 2.0**. We are still maintaining Haystack 1.x to give everyone
> enough time to migrate to 2.0. [Switch to Haystack 1.x here](https://github.com/deepset-ai/haystack/tree/v1.x).

- **Technology agnostic:** Allow users the flexibility to decide what vendor or technology they want and make it easy to switch out any component for another. Haystack allows you to use and compare models available from OpenAI, Cohere and Hugging Face, as well as your own local models or models hosted on Azure, Bedrock and SageMaker.
- **Explicit:** Make it transparent how different moving parts can “talk” to each other so it's easier to fit your tech stack and use case.
- **Flexible:** Haystack provides all tooling in one place: database access, file conversion, cleaning, splitting, training, eval, inference, and more. And whenever custom behavior is desirable, it's easy to create custom components.
- **Extensible:** Provide a uniform and easy way for the community and third parties to build their own components and foster an open ecosystem around Haystack.

Some examples of what you can do with Haystack:

-   Build **retrieval augmented generation (RAG)** by making use of one of the available vector databases and customizing your LLM interaction, the sky is the limit 🚀
-   Perform Question Answering **in natural language** to find granular answers in your documents.
-   Perform **semantic search** and retrieve documents according to meaning.
-   Build applications that can make complex decisions making to answer complex queries: such as systems that can resolve complex customer queries, do knowledge search on many disconnected resources and so on.
-   Scale to millions of docs using retrievers and production-scale components.
-   Use **off-the-shelf models** or **fine-tune** them to your data.
-   Use **user feedback** to evaluate, benchmark, and continuously improve your models.

> [!TIP]
>
>
> Are you looking for a managed solution that benefits from Haystack? [deepset Cloud](https://www.deepset.ai/deepset-cloud?utm_campaign=developer-relations&utm_source=haystack&utm_medium=readme) is our fully managed, end-to-end platform to integrate LLMs with your data, which uses Haystack for the LLM pipelines architecture.

## Telemetry

Haystack collects **anonymous** usage statistics of pipeline components. We receive an event every time these components are initialized. This way, we know which components are most relevant to our community.

Read more about telemetry in Haystack or how you can opt out in [Haystack docs](https://docs.haystack.deepset.ai/v2.0/docs/telemetry).

## 🖖 Community

If you have a feature request or a bug report, feel free to open an [issue in Github](https://github.com/deepset-ai/haystack/issues). We regularly check these and you can expect a quick response. If you'd like to discuss a topic, or get more general advice on how to make Haystack work for your project, you can start a thread in [Github Discussions](https://github.com/deepset-ai/haystack/discussions) or our [Discord channel](https://discord.gg/haystack). We also check [𝕏 (Twitter)](https://twitter.com/haystack_ai) and [Stack Overflow](https://stackoverflow.com/questions/tagged/haystack).

## Contributing to Haystack

We are very open to the community's contributions - be it a quick fix of a typo, or a completely new feature! You don't need to be a Haystack expert to provide meaningful improvements. To learn how to get started, check out our [Contributor Guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) first.

There are several ways you can contribute to Haystack:
- Contribute to the main Haystack project
- Contribute an integration on [haystack-core-integrations](https://github.com/deepset-ai/haystack-core-integrations)

> [!TIP]
>👉 **[Check out the full list of issues that are open to contributions](https://github.com/orgs/deepset-ai/projects/14)**

## Who Uses Haystack

Here's a list of projects and companies using Haystack. Want to add yours? Open a PR, add it to the list and let the
world know that you use Haystack!

-   [Airbus](https://www.airbus.com/en)
-   [Alcatel-Lucent](https://www.al-enterprise.com/)
-   [Apple](https://www.apple.com/)
-   [BetterUp](https://www.betterup.com/)
-   [Databricks](https://www.databricks.com/)
-   [Deepset](https://deepset.ai/)
-   [Etalab](https://www.deepset.ai/blog/improving-on-site-search-for-government-agencies-etalab)
-   [Infineon](https://www.infineon.com/)
-   [Intel](https://github.com/intel/open-domain-question-and-answer#readme)
-   [Intelijus](https://www.intelijus.ai/)
-   [Intel Labs](https://github.com/IntelLabs/fastRAG#readme)
-   [LEGO](https://github.com/larsbaunwall/bricky#readme)
-   [Netflix](https://netflix.com)
-   [NOS Portugal](https://www.nos.pt/en/welcome)
-   [Nvidia](https://developer.nvidia.com/blog/reducing-development-time-for-intelligent-virtual-assistants-in-contact-centers/)
-   [PostHog](https://github.com/PostHog/max-ai#readme)
-   [Rakuten](https://www.rakuten.com/)
-   [Sooth.ai](https://www.deepset.ai/blog/advanced-neural-search-with-sooth-ai)


SoCo
====

SoCo (Sonos Controller) is a Python library that allows you to
control `Sonos speakers`_ programmatically. It was originally created at `Music
Hack Day Sydney`_ by `Rahim Sonawalla`_ and is now developed by a `team of
people`_ at its `GitHub repository`_

For more background on the project, please see Rahim's `blog post
<http://www.hirahim.com/blog/2012/04/29/dissecting-the-sonos-controller/>`_.

Visit the `SoCo documentation`_ for a more detailed overview of the functionailty.

.. image:: https://badges.gitter.im/SoCo/SoCo.svg
   :alt: Join the chat at https://gitter.im/SoCo/SoCo
   :target: https://gitter.im/SoCo/SoCo?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge

.. image:: https://travis-ci.com/SoCo/SoCo.svg?branch=master
   :target: https://travis-ci.com/SoCo/SoCo
   :alt: Build Status

.. image:: https://img.shields.io/requires/github/SoCo/SoCo/master.svg?style=flat
   :target: https://requires.io/github/SoCo/SoCo/requirements/?branch=master
   :alt: Requirements Status

.. image:: https://img.shields.io/pypi/v/soco.svg?style=flat
    :target: https://pypi.python.org/pypi/soco/
    :alt: Latest PyPI version

WARNING
-------

Sonos has changed the way music service account information is available. This means that **currently a group of music service will give authentication issues and cannot be used at all**. Known members of this group are: Google Play Music, Apple Music, Amazon Music, Spotify and Napster.

Issue #557 is a meta issue for this problem and you can use that to track progress on solving the issues, but
*please refrain from posting "me too" comments* in there. Also, there is no need to open any more separate issue about this.
If you have another music service that should be on the list, comment in #557

**As of v0.26.0, nascent music service support has been reinstated, with some known issues. Testing and issue reporting
would be appreciated.**

Installation
------------

SoCo requires Python 3.6 or newer.

Use pip:

``pip install soco``

SoCo depends on a number of Python packages. If you use pip to install Soco,
the dependencies will be installed automatically for you. If not, you can inspect
the requirements in the `requirements.txt <https://github.com/SoCo/SoCo/blob/master/requirements.txt>`_
file.

Basic Usage
-----------

You can interact with a Sonos Zone Player through a SoCo object. If you know
the IP address of a Zone Player, you can create a SoCo object directly:

.. code:: python

    >>> from soco import SoCo
    >>> my_zone = SoCo('192.168.1.101')
    >>> my_zone.player_name
    Kitchen
    >>> my_zone.status_light = True
    >>> my_zone.volume = 6

But perhaps the easiest way is to use the module-level `discover` function.
This will find all the Zone Players on your network, and return a python
set containing them:

.. code:: python

    >>> from soco import discover
    >>> for zone in discover():
    ...        print(zone.player_name)
    Living Room
    Kitchen

If you prefer a list to a set:

.. code:: python

    >>> zone_list = list(discover())
    >>> zone_list
    [SoCo("192.168.1.101"), SoCo("192.168.1.102")]
    >>> zone_list[0].mute = True

Of course, you can also play music!

.. code:: python

    #!/usr/bin/env python
    from soco import SoCo

    if __name__ == '__main__':
        sonos = SoCo('192.168.1.102') # Pass in the IP of your Sonos speaker
        # You could use the discover function instead, if you don't know the IP

        # Pass in a URI to a media file to have it streamed through the Sonos
        # speaker
        sonos.play_uri(
            'http://ia801402.us.archive.org/20/items/TenD2005-07-16.flac16/TenD2005-07-16t10Wonderboy.mp3')

        track = sonos.get_current_track_info()

        print(track['title'])

        sonos.pause()

        # Play a stopped or paused track
        sonos.play()

Support
-------

If you need support for SoCo, feel free to post your question in the `SoCo Gitter Room <https://gitter.im/SoCo/SoCo>`_.

Example Applications
--------------------

To show off what can be made with SoCo, a simple web application is included in
the ``examples`` folder.

.. figure:: https://github.com/SoCo/SoCo/raw/master/examples/webapp/screenshot.png
   :alt: Screenshot of web app

   Screenshot of web app

Features
--------

SoCo supports the following controls amongst others:

-  Play, Pause, Stop
-  Next track, Previous track
-  Volume get and set
-  Mute (or unmute)
-  Get current transport information (if speaker is
   playing, paused or stopped)
-  Get information about the currently playing track

   -  Track title
   -  Artist
   -  Album
   -  Album Art (if available)
   -  Track length
   -  Duration played (for example, 30 seconds into a 3 minute song)
   -  Playlist position (for example, item 5 in the playlist)
   -  Track URI

-  Receive events when the player state changes
-  Search for and play music items:

   -  Local music library
   -  Webradio via TuneIn and music services (still unstable)
   -  Saved Sonos favorites, favorite radio stations and shows

-  Switch the speaker’s source to line-in or TV input (if the Zone Player
   supports it)
-  Manage the Sonos queue:

   -  Get the items in the queue
   -  Add items to the queue
   -  Clear the queue
   -  Play a specific song from the queue

-  Join or unjoin speakers from a group
-  Put all Sonos speakers in a network into “party mode”.

-  Get or set alarms
-  Get or set sleep timers

-  Enable or disable surround speakers or subwoofer
-  Get information regarding a home theater setup:

   - If surround speakers or a subwoofer are paired
   - Which audio channel a given speaker handles

-  Get or set the speaker’s bass and treble EQ
-  Toggle the speaker’s loudness compensation, night mode and dialog mode
-  Toggle the white status light on the unit
-  Get the speaker’s information

   -  Zone Name
   -  Zone Icon
   -  UID (usually something like RINCON\_XXXXXXXXXXXXXXXXX)
   -  Serial Number
   -  Software version
   -  Hardware version
   -  MAC Address

-  Set the speaker’s Zone Name
-  Start a music library update and determine if one is in progress

SoCo also supports lower level access from Python to all Sonos services
(e.g. ContentDirectory or RenderingControl).

Related Projects
----------------

**Socos** is a command line tool for controlling Sonos devices. It is developed
in conjunction with Soco, but in a `separate repository <https://github.com/SoCo/socos>`_.

**SoCo-CLI** (`soco-cli <https://github.com/avantrec/soco-cli>`_) is a powerful and
fully-featured command line tool suitable for use in scripts, scheduled tasks, etc. It
supports time-based and state-based actions, and repeated commands using loops. Audio
files on the local filesystem can be played back directly on Sonos from the command line.
Multi-household Sonos systems are supported.

Older Projects
^^^^^^^^^^^^^^

More of a Ruby fan? Not a problem, `Sam Soffes`_ is building out an
awesome `Ruby gem`_.

Looking for a GUI that’s more than just a sample project? `Joel
Björkman`_ is building a Sonos Controller GUI–great for folks on Linux
where there isn’t an official Sonos Controller application! Find, fork,
and contribute to it here: https://github.com/labero/SoCo-Tk.

SoCo Gitter Room
----------------

There is a `SoCo Gitter discussion room <https://gitter.im/SoCo/SoCo>`_.  Feel free to drop by for support, ideas or casual conversation related to SoCo.

License
-------

SoCo is released under the `MIT license`_.

.. _Sonos speakers: http://www.sonos.com/system/
.. _Music Hack Day Sydney: http://sydney.musichackday.org/2012/
.. _blog post: http://www.hirahim.com/blog/2012/04/29/dissecting-the-sonos-controller/
.. _Sam Soffes: https://github.com/soffes
.. _Ruby gem: https://github.com/soffes/sonos
.. _Joel Björkman: https://github.com/labero
.. _MIT license: http://www.opensource.org/licenses/mit-license.php
.. _Rahim Sonawalla: https://github.com/rahims/SoCo
.. _GitHub repository: https://github.com/SoCo/SoCo
.. _team of people: https://github.com/SoCo/SoCo/blob/master/AUTHORS.rst
.. _SoCo documentation: https://soco.readthedocs.org/en/latest/


[**English**](https://github.com/dbiir/UER-py) | [**中文**](https://github.com/dbiir/UER-py/blob/master/README_ZH.md)

[![Build Status](https://github.com/dbiir/UER-py/actions/workflows/github-actions.yml/badge.svg)](https://github.com/dbiir/UER-py/actions/workflows/github-actions.yml)
[![codebeat badge](https://codebeat.co/badges/f75fab90-6d00-44b4-bb42-d19067400243)](https://codebeat.co/projects/github-com-dbiir-uer-py-master)
![](https://img.shields.io/badge/license-MIT-000000.svg)
[![arXiv](https://img.shields.io/badge/arXiv-1909.05658-<color>.svg)](https://arxiv.org/abs/1909.05658)

Pre-training has become an essential part for NLP tasks. UER-py (Universal Encoder Representations) is a toolkit for pre-training on general-domain corpus and fine-tuning on downstream task. UER-py maintains model modularity and supports research extensibility. It facilitates the use of existing pre-training models, and provides interfaces for users to further extend upon. With UER-py, we build a model zoo which contains pre-trained models of different properties. **See the [UER-py project Wiki](https://github.com/dbiir/UER-py/wiki) for full documentation**.

**🚀** We have open-sourced the [TencentPretrain](https://github.com/Tencent/TencentPretrain), a refactored new version of UER-py. TencentPretrain supports multi-modal models and enables training of large models. If you are interested in text models of medium size (with parameter sizes of less than one billion), we recommend continuing to use the UER-py project.

Table of Contents
=================
  * [Features](#features)
  * [Requirements](#requirements)
  * [Quickstart](#quickstart)
  * [Pre-training data](#pre-training-data)
  * [Downstream datasets](#downstream-datasets)
  * [Modelzoo](#modelzoo)
  * [Instructions](#instructions)
  * [Competition solutions](#competition-solutions)
  * [Citation](#citation)
  * [Contact information](#contact-information)

## Features
UER-py has the following features:
- __Reproducibility__ UER-py has been tested on many datasets and should match the performances of the original pre-training model implementations such as BERT, GPT-2, ELMo, and T5.
- __Model modularity__ UER-py is divided into the following components: embedding, encoder, target embedding (optional), decoder (optional), and target. Ample modules are implemented in each component. Clear and robust interface allows users to combine modules to construct pre-training models with as few restrictions as possible.
- __Model training__ UER-py supports CPU mode, single GPU mode and distributed training mode.
- __Model zoo__ With the help of UER-py, we pre-train and release models of different properties. Proper selection of pre-trained models is important to the performances of downstream tasks.
- __SOTA results__ UER-py supports comprehensive downstream tasks (e.g. classification and machine reading comprehension) and provides winning solutions of many NLP competitions.
- __Abundant functions__ UER-py provides abundant functions related with pre-training, such as feature extractor and text generation.

## Requirements
* Python >= 3.6
* torch >= 1.1
* six >= 1.12.0
* argparse
* packaging
* regex
* For the pre-trained model conversion (related with TensorFlow) you will need TensorFlow
* For the tokenization with sentencepiece model you will need [SentencePiece](https://github.com/google/sentencepiece)
* For developing a stacking model you will need LightGBM and [BayesianOptimization](https://github.com/fmfn/BayesianOptimization)
* For the pre-training with whole word masking you will need word segmentation tool such as [jieba](https://github.com/fxsjy/jieba)
* For the use of CRF in sequence labeling downstream task you will need [pytorch-crf](https://github.com/kmkurn/pytorch-crf)

## Quickstart
This section uses several commonly-used examples to demonstrate how to use UER-py. More details are discussed in Instructions section. We firstly use BERT (a text pre-training model) on book review sentiment classification dataset. We pre-train model on book review corpus and then fine-tune it on book review sentiment classification dataset. There are three input files: book review corpus, book review sentiment classification dataset, and vocabulary. All files are encoded in UTF-8 and included in this project.

The format of the corpus for BERT is as follows (one sentence per line and documents are delimited by empty lines)：
```
doc1-sent1
doc1-sent2
doc1-sent3

doc2-sent1

doc3-sent1
doc3-sent2
```
The book review corpus is obtained from book review classification dataset. We remove labels and split a review into two parts from the middle to construct a document with two sentences (see *book_review_bert.txt* in *corpora* folder).

The format of the classification dataset is as follows:
```
label    text_a
1        instance1
0        instance2
1        instance3
```
Label and instance are separated by \t . The first row is a list of column names. The label ID should be an integer between (and including) 0 and n-1 for n-way classification.

We use Google's Chinese vocabulary file *models/google_zh_vocab.txt*, which contains 21128 Chinese characters.

We firstly pre-process the book review corpus. In the pre-processing stage, the corpus needs to be processed into the format required by the specified pre-training model (*--data_processor*):
```
python3 preprocess.py --corpus_path corpora/book_review_bert.txt --vocab_path models/google_zh_vocab.txt \
                      --dataset_path dataset.pt --processes_num 8 --data_processor bert
```
Notice that *six>=1.12.0* is required.

Pre-processing is time-consuming. Using multiple processes can largely accelerate the pre-processing speed (*--processes_num*). BERT tokenizer is used in default (*--tokenizer bert*). After pre-processing, the raw text is converted to *dataset.pt*, which is the input of *pretrain.py*. Then we download Google's pre-trained Chinese BERT model [*google_zh_model.bin*](https://share.weiyun.com/FR4rPxc4) (in UER format and the original model is from [here](https://github.com/google-research/bert)), and put it in *models* folder. We load the pre-trained Chinese BERT model and further pre-train it on book review corpus. Pre-training model is usually composed of embedding, encoder, and target layers. To build a pre-training model, we should provide related information. Configuration file (*--config_path*) specifies the modules and hyper-parameters used by pre-training models. More details can be found in *models/bert/base_config.json*. Suppose we have a machine with 8 GPUs:
```
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_zh_vocab.txt \
                    --pretrained_model_path models/google_zh_model.bin \
                    --config_path models/bert/base_config.json \
                    --output_model_path models/book_review_model.bin \
                    --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
                    --total_steps 5000 --save_checkpoint_steps 1000 --batch_size 32

mv models/book_review_model.bin-5000 models/book_review_model.bin
```
Notice that the model trained by *pretrain.py* is attacted with the suffix which records the training step (*--total_steps*). We could remove the suffix for ease of use.

Then we fine-tune the pre-trained model on downstream classification dataset. We use embedding and encoder layers of [*book_review_model.bin*](https://share.weiyun.com/PnxMrRwZ), which is the output of *pretrain.py*:
```
python3 finetune/run_classifier.py --pretrained_model_path models/book_review_model.bin \
                                   --vocab_path models/google_zh_vocab.txt \
                                   --config_path models/bert/base_config.json \
                                   --train_path datasets/book_review/train.tsv \
                                   --dev_path datasets/book_review/dev.tsv \
                                   --test_path datasets/book_review/test.tsv \
                                   --epochs_num 3 --batch_size 32
```
The default path of the fine-tuned classifier model is *models/finetuned_model.bin* . It is noticeable that the actual batch size of pre-training is *--batch_size* times *--world_size* ; The actual batch size of downstream task (e.g. classification) is *--batch_size* .
Then we do inference with the fine-tuned model.
```
python3 inference/run_classifier_infer.py --load_model_path models/finetuned_model.bin \
                                          --vocab_path models/google_zh_vocab.txt \
                                          --config_path models/bert/base_config.json \
                                          --test_path datasets/book_review/test_nolabel.tsv \
                                          --prediction_path datasets/book_review/prediction.tsv \
                                          --labels_num 2
```
*--test_path* specifies the path of the file to be predicted. The file should contain text_a column.
*--prediction_path* specifies the path of the file with prediction results.
We need to explicitly specify the number of labels by *--labels_num*. The above dataset is a two-way classification dataset.

The above content provides basic ways of using UER-py to pre-process, pre-train, fine-tune, and do inference. More use cases can be found in complete :arrow_right: [__quickstart__](https://github.com/dbiir/UER-py/wiki/Quickstart) :arrow_left: . The complete quickstart contains abundant use cases, covering most of the pre-training related application scenarios. It is recommended that users read the complete quickstart in order to use the project reasonably.

## Pre-training data
This section provides links to a range of :arrow_right: [__pre-training data__](https://github.com/dbiir/UER-py/wiki/Pretraining-data) :arrow_left: . UER can load these pre-training data directly.

## Downstream datasets
This section provides links to a range of :arrow_right: [__downstream datasets__](https://github.com/dbiir/UER-py/wiki/Downstream-datasets) :arrow_left: . UER can load these datasets directly.

## Modelzoo
With the help of UER, we pre-trained models of different properties (e.g. models based on different corpora, encoders, and targets). Detailed introduction of pre-trained models and their download links can be found in :arrow_right: [__modelzoo__](https://github.com/dbiir/UER-py/wiki/Modelzoo) :arrow_left: . All pre-trained models can be loaded by UER directly.

## Instructions
UER-py is organized as follows：
```
UER-py/
    |--uer/
    |    |--embeddings/ # contains modules of embedding component
    |    |--encoders/ # contains modules of encoder component such as RNN, CNN, Transformer
    |    |--decoders/ # contains modules of decoder component
    |    |--targets/ # contains modules of target component such as language modeling, masked language modeling
    |    |--layers/ # contains frequently-used NN layers
    |    |--models/ # contains model.py, which combines modules of different components
    |    |--utils/ # contains frequently-used utilities
    |    |--model_builder.py
    |    |--model_loader.py
    |    |--model_saver.py
    |    |--opts.py
    |    |--trainer.py
    |
    |--corpora/ # contains pre-training data
    |--datasets/ # contains downstream tasks
    |--models/ # contains pre-trained models, vocabularies, and configuration files
    |--scripts/ # contains useful scripts for pre-training models
    |--finetune/ # contains fine-tuning scripts for downstream tasks
    |--inference/ # contains inference scripts for downstream tasks
    |
    |--preprocess.py
    |--pretrain.py
    |--README.md
    |--README_ZH.md
    |--requirements.txt
    |--LICENSE

```

The code is organized based on components (e.g. embeddings, encoders). Users can use and extend upon it with little efforts.

Comprehensive examples of using UER can be found in :arrow_right: [__instructions__](https://github.com/dbiir/UER-py/wiki/Instructions) :arrow_left: , which help users quickly implement pre-training models such as BERT, GPT-2, ELMo, T5 and fine-tune pre-trained models on a range of downstream tasks.

## Competition solutions
UER-py has been used in winning solutions of many NLP competitions. In this section, we provide some examples of using UER-py to achieve SOTA results on NLP competitions, such as CLUE. See :arrow_right: [__competition solutions__](https://github.com/dbiir/UER-py/wiki/Competition-solutions) :arrow_left: for more detailed information.

## Citation
#### If you are using the work (e.g. pre-trained models) in UER-py for academic work, please cite the [system paper](https://arxiv.org/pdf/1909.05658.pdf) published in EMNLP 2019:
```
@article{zhao2019uer,
  title={UER: An Open-Source Toolkit for Pre-training Models},
  author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
  journal={EMNLP-IJCNLP 2019},
  pages={241},
  year={2019}
}
```

## Contact information
For communication related to this project, please contact Zhe Zhao (helloworld@alu.ruc.edu.cn; nlpzhezhao@tencent.com) or Yudong Li (liyudong123@hotmail.com) or Cheng Hou (chenghoubupt@bupt.edu.cn) or Wenhang Shi (wenhangshi@ruc.edu.cn).

This work is instructed by my enterprise mentors __Qi Ju__, __Xuefeng Yang__, __Haotang Deng__ and school mentors __Tao Liu__, __Xiaoyong Du__.

We also got a lot of help from Weijie Liu, Lusheng Zhang, Jianwei Cui, Xiayu Li, Weiquan Mao, Xin Zhao, Hui Chen, Jinbin Zhang, Zhiruo Wang, Peng Zhou, Haixiao Liu, and Weijian Wu.


 PaddleX -- 飞桨低代码开发工具，以低代码的形式支持开发者快速实现产业实际项目落地

## 近期动态
🔥 PaddleX3.0 升级中，6月正式发布，敬请期待，云端使用请前往飞桨 AI Studio 星河社区：https://aistudio.baidu.com/pipeline/mine ，点击「创建产线」开启使用。

## 产品介绍
PaddleX3.0 是飞桨精选模型的低代码开发工具，支持国内外多款主流硬件的训练和推理，覆盖工业、能源、金融、交通、教育等全行业，沉淀产业实际经验，并提供丰富的案例实践教程，全程助力开发者产业实践落地。

PaddleX3.0 分为本地端和云端，本地端提供统一任务API接口，支持图像分类、目标检测、图像分割、实例分割、OCR、时序相关等任务；云端提供[图形化开发界面](https://aistudio.baidu.com/pipeline/mine)，支持开发者使用零代码产线产出高质量模型和部署包。本项目面向本地端，开发者可以基于本项目快速完成模型训练、评估、推理。本项目提供了两种模型开发工具，即**单模型开发工具**和**模型产线开发工具**。

## 安装与快速开始
- [安装](./docs/tutorials/INSTALL.md)
- 快速开始
  - [单模型开发工具快速开始](./docs/tutorials/QUCK_STARTED.md)
  - 模型产线开发工具快速开始(comming soon)

## 单模型开发工具
本节介绍 PaddleX3.0 单模型的全流程开发流程，包括数据准备、模型训练/评估、模型推理的使用方法。PaddleX3.0 支持的模型可以参考 [PaddleX模型库](./docs/tutorials/models/support_model_list.md)。

### 1. 数据准备
- [数据准备流程](./docs/data/README.md)
- [数据标注](./docs/tutorials/data/annotation/README.md)
- [数据校验](./docs/tutorials/data/dataset_check.md)
### 2. 模型训练
- [模型训练/评估](./docs/tutorials/train/README.md)
- 模型优化(comming soon)

### 3. 模型推理
- 模型推理能力即将开源，敬请期待...

## 模型产线开发工具
本节将介绍 PaddleX3.0 模型产线的全流程开发流程，此部分内容即将开源，敬请期待...

## 多硬件支持
🔥 本项目支持在多种硬件上进行模型的开发，除了GPU外，当前支持的硬件还有**昆仑芯**、**昇腾芯**、**寒武纪芯**。只需添加一个配置设备的参数，即可在对应硬件上使用上述工具。详情参考[多硬件训练](./docs/tutorials/train/multi_device_train.md)。

- 昇腾芯支持的模型列表请参考 [PaddleX 昇腾芯模型列表](./docs/tutorials/models/support_npu_model_list.md)。
- 昆仑芯支持的模型列表请参考 [PaddleX 昆仑芯模型列表](./docs/tutorials/models/support_xpu_model_list.md)。
- 寒武纪芯支持的模型列表请参考 [PaddleX 寒武纪芯模型列表](./docs/tutorials/models/support_mlu_model_list.md)。

## 贡献代码

我们非常欢迎您为 PaddleX 贡献代码或者提供使用建议。如果您可以修复某个 issue 或者增加一个新功能，欢迎给我们提交 Pull Requests。


============
|logo_named|
============

Manipulate DNS records on various DNS providers in a standardized/agnostic way.

|build_status| |coverage_status| |docker_pulls| |pypy_version| |pypy_python_support| |github_license|

.. |logo_named| image:: https://raw.githubusercontent.com/AnalogJ/lexicon/master/docs/images/logo_named.svg
    :alt: Lexicon
.. |build_status| image:: https://img.shields.io/azure-devops/build/AnalogJ/8425f0f5-0178-4d8c-b1fd-11db7e11b6a7/1/master
    :target: https://dev.azure.com/AnalogJ/lexicon/_build/latest?definitionId=1&branchName=master
.. |coverage_status| image:: https://img.shields.io/coverallsCoverage/github/AnalogJ/lexicon
    :target: https://coveralls.io/github/AnalogJ/lexicon?branch=master
.. |docker_pulls| image:: https://img.shields.io/docker/pulls/analogj/lexicon
    :target: https://hub.docker.com/r/analogj/lexicon
.. |pypy_version| image:: https://img.shields.io/pypi/v/dns-lexicon
    :target: https://pypi.python.org/pypi/dns-lexicon
.. |pypy_python_support| image:: https://img.shields.io/pypi/pyversions/dns-lexicon
    :target: https://pypi.python.org/pypi/dns-lexicon
.. |github_license| image:: https://img.shields.io/github/license/AnalogJ/lexicon
    :target: https://github.com/AnalogJ/lexicon/blob/master/LICENSE

.. contents:: Table of Contents
   :local:

.. tag: intro-begin

Why using Lexicon?
==================

Lexicon provides a way to manipulate DNS records on multiple DNS providers in a standardized way.

Lexicon can be used as:

- a CLI tool:

.. code-block:: bash

    # Create a TXT entry in domain.net zone hosted by CloudFlare
    lexicon cloudflare create domain.net TXT --name foo --content bar

- or a Python library:

.. code-block:: python

    # Create a TXT entry in domain.net zone hosted by CloudFlare
    from lexicon.client import Client
    from lexicon.config import ConfigResolver

    config = ConfigResolver().with_env().with_dict({
        "provider_name" : "cloudflare",
        "domain": "domain.net",
    })

    with Client(config) as operations:
        operations.create_record("TXT", "foo", "bar")

Lexicon was designed to be used in automation, specifically letsencrypt.

* `Generating Intranet & Private Network SSL Certificates using Lets Encrypt & Lexicon <http://blog.thesparktree.com/post/138999997429/generating-intranet-and-private-network-ssl>`_

Supported providers
===================

Only DNS providers who have an API can be supported by `lexicon`.

The current supported providers are:

..
  This section is autogenerated and should not been modified directly.
  However you should add a reference to the provider API in the list below,
  using the following syntax: .. _provider: URL_API

.. tag: providers-table-begin

+-----------------+-----------------+-----------------+-----------------+-----------------+
| aliyun_         | aurora_         | azure_          | cloudflare_     | cloudns_        |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| cloudxns_       | conoha_         | constellix_     | ddns_           | digitalocean_   |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| dinahosting_    | directadmin_    | dnsimple_       | dnsmadeeasy_    | dnspark_        |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| dnspod_         | dnsservices_    | dreamhost_      | duckdns_        | dynu_           |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| easydns_        | easyname_       | euserv_         | exoscale_       | flexibleengine_ |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| gandi_          | gehirn_         | glesys_         | godaddy_        | googleclouddns_ |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| gransy_         | gratisdns_      | henet_          | hetzner_        | hostingde_      |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| hover_          | infoblox_       | infomaniak_     | internetbs_     | inwx_           |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| joker_          | linode_         | linode4_        | localzone_      | luadns_         |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| memset_         | misaka_         | mythicbeasts_   | namecheap_      | namecom_        |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| namesilo_       | netcup_         | nfsn_           | njalla_         | nsone_          |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| oci_            | onapp_          | online_         | ovh_            | plesk_          |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| pointhq_        | porkbun_        | powerdns_       | rackspace_      | rage4_          |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| rcodezero_      | route53_        | safedns_        | sakuracloud_    | softlayer_      |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| timeweb_        | transip_        | ultradns_       | valuedomain_    | vercel_         |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| vultr_          | webgo_          | wedos_          | yandex_         | yandexcloud_    |
+-----------------+-----------------+-----------------+-----------------+-----------------+
| zeit_           | zilore_         | zonomi_         |                 |                 |
+-----------------+-----------------+-----------------+-----------------+-----------------+

.. tag: providers-table-end

.. _aliyun: https://help.aliyun.com/document_detail/29739.html
.. _aurora: https://www.pcextreme.com/aurora/dns
.. _azure: https://docs.microsoft.com/en-us/rest/api/dns/
.. _cloudflare: https://api.cloudflare.com/#endpoints
.. _cloudns: https://www.cloudns.net/wiki/article/56/
.. _cloudxns: https://www.cloudxns.net/support/lists/cid/17.html
.. _conoha: https://www.conoha.jp/docs/
.. _constellix: https://api-docs.constellix.com/?version=latest
.. _ddns: https://www.rfc-editor.org/rfc/rfc2136
.. _digitalocean: https://developers.digitalocean.com/documentation/v2/#create-a-new-domain
.. _dinahosting: https://en.dinahosting.com/api
.. _directadmin: https://www.directadmin.com/features.php?id=504
.. _dnsimple: https://developer.dnsimple.com/v2/
.. _dnsmadeeasy: https://api-docs.dnsmadeeasy.com/?version=latest
.. _dnspark: https://dnspark.zendesk.com/entries/31210577-rest-api-dns-documentation
.. _dnspod: https://support.dnspod.cn/support/api
.. _dnsservices: https://dns.services/userapi
.. _dreamhost: https://help.dreamhost.com/hc/en-us/articles/217560167-api_overview
.. _duckdns: https://www.duckdns.org/spec.jsp
.. _dynu: https://www.dynu.com/support/api
.. _easydns: http://docs.sandbox.rest.easydns.net/
.. _easyname: https://www.easyname.com/en
.. _euserv: https://support.euserv.com/api-doc/
.. _exoscale: https://community.exoscale.com/documentation/dns/api/
.. _flexibleengine: https://registry.terraform.io/providers/FlexibleEngineCloud/flexibleengine/latest/docs/data-sources/dns_zone_v2
.. _gandi: http://doc.livedns.gandi.net/
.. _gehirn: https://support.gehirn.jp/apidocs/gis/dns/index.html
.. _glesys: https://github.com/glesys/api/wiki/
.. _godaddy: https://developer.godaddy.com/getstarted#access
.. _googleclouddns: https://cloud.google.com/dns/api/v1/
.. _gransy: https://subreg.cz/manual/
.. _gratisdns:
.. _henet: https://dns.he.net/
.. _hetzner: https://dns.hetzner.com/api-docs/
.. _hostingde:
.. _hover: https://www.hover.com/
.. _infoblox: https://docs.infoblox.com/display/ilp/infoblox+documentation+portal
.. _infomaniak: https://www.infomaniak.com
.. _internetbs: https://internetbs.net/resellerregistrardomainnameapi
.. _inwx: https://www.inwx.de/en/offer/api
.. _joker: https://joker.com/faq/index.php?action=show&cat=39
.. _linode: https://www.linode.com/api/dns
.. _linode4: https://developers.linode.com/api/docs/v4#tag/domains
.. _localzone:
.. _luadns: http://www.luadns.com/api.html
.. _memset: https://www.memset.com/apidocs/methods_dns.html
.. _misaka: https://misaka.io/dns/
.. _mythicbeasts: https://www.mythic-beasts.com/support/api/dnsv2
.. _namecheap: https://www.namecheap.com/support/api/methods.aspx
.. _namecom: https://www.name.com/api-docs
.. _namesilo: https://www.namesilo.com/api_reference.php
.. _netcup: https://ccp.netcup.net/run/webservice/servers/endpoint.php
.. _nfsn:
.. _njalla: https://njal.la/api/
.. _nsone: https://ns1.com/api/
.. _oci: https://docs.oracle.com/en-us/iaas/Content/DNS/home.htm
.. _onapp: https://docs.onapp.com/display/55api/onapp+5.5+api+guide
.. _online:
.. _ovh: https://api.ovh.com/
.. _plesk: https://docs.plesk.com/en-us/onyx/api-rpc/about-xml-api.28709/
.. _pointhq: https://pointhq.com/api/docs
.. _porkbun: https://kb.porkbun.com/article/190-getting-started-with-the-porkbun-dns-api
.. _powerdns: https://doc.powerdns.com/md/httpapi/api_spec/
.. _rackspace: https://developer.rackspace.com/docs/cloud-dns/v1/developer-guide/
.. _rage4: https://gbshouse.uservoice.com/knowledgebase/articles/109834-rage4-dns-developers-api
.. _rcodezero: https://my.rcodezero.at/api-doc
.. _route53: https://docs.aws.amazon.com/route53/latest/apireference/welcome.html
.. _safedns: https://developers.ukfast.io/documentation/safedns
.. _sakuracloud: https://developer.sakura.ad.jp/cloud/api/1.1/
.. _softlayer: https://sldn.softlayer.com/article/rest#http_request_types
.. _transip: https://api.transip.nl/rest/docs.html
.. _ultradns: https://ultra-portalstatic.ultradns.com/static/docs/rest-api_user_guide.pdf
.. _valuedomain: https://www.value-domain.com/service/api/
.. _vercel: https://vercel.com/docs/api#endpoints/dns
.. _vultr: https://www.vultr.com/api/#tag/dns
.. _webgo: https://www.webgo.de/
.. _wedos: https://www.wedos.com/cs/
.. _yandex: https://yandex.com/dev/domain/doc/reference/dns-add.html
.. _yandexcloud: https://cloud.yandex.com/en/docs/dns/api-ref/DnsZone/
.. _zeit:
.. _zilore: https://zilore.com/en/help/api
.. _zonomi: http://zonomi.com/app/dns/dyndns.jsp

.. tag: intro-end

Documentation
=============

Online documentation (user guide, configuration reference) is available in the `Lexicon documentation`_.

For a quick start, please have a look in particular at the `User guide`_.

.. _Lexicon documentation: https://dns-lexicon.readthedocs.io
.. _User guide: https://dns-lexicon.readthedocs.io/en/latest/user_guide.html

Contributing
============

If you want to help in the Lexicon development, you are welcome!

Please have a look at the `Developer guide`_ page to know how to start.

.. _Developer guide: https://dns-lexicon.readthedocs.io/en/latest/developer_guide.html

Licensing
=========

- MIT
- Logo_: transform by Mike Rowe from the Noun Project

.. _Logo: https://thenounproject.com/term/transform/397964