# Eclipse BaSyx Python SDK

(formerly known as PyI40AAS – Python Industry 4.0 Asset Administration Shell)

The Eclipse BaSyx Python project focuses on providing a Python implementation of the Asset Administration Shell (AAS)
for Industry 4.0 Systems.
These are the currently implemented specifications:

| Specification                         | Version                                                                                |
|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| Part 1: Metamodel                     | [v3.0 (01001-3-0)](https://industrialdigitaltwin.org/content-hub/aasspecifications/idta_01001-3-0_metamodel)                |
| Part 2: API                           | not implemented yet                                                                                |
| Part 3a: Data Specification IEC 61360 | [v3.0 (01003-a-3-0)](https://industrialdigitaltwin.org/content-hub/aasspecifications/idta_01003-a-3-0_data_specification)   |
| Part 5: Package File Format (AASX)    | [v3.0 (01005-3-0)](https://industrialdigitaltwin.org/content-hub/aasspecifications/idta-01005-3-0_package_file_format_aasx) |

## Features

* Modelling of AASs as Python objects
    * **except for**: *HasDataSpecification*
* Reading and writing of AASX package files
* (De-)serialization of AAS objects into/from JSON and XML
* Storing of AAS objects in CouchDB, Backend infrastructure for easy expansion
* Compliance checking of AAS XML and JSON files

### Project Structure

The BaSyx Python SDK project provides the `basax.aas` Python package with 6 submodules:

* `basyx.aas.model`: The AAS metamodel implemented in python
* `basyx.aas.adapter`: Adapters for various file formats
* `basyx.aas.backend`: Backend infrastructure for storing and retrieving AAS objects
* `basyx.aas.util`: Provides utilities
* `basyx.aas.examples`: Example data and tutorials

## License

The BaSyx Python SDK project is licensed under the terms of the MIT License.

SPDX-License-Identifier: MIT

For more information, especially considering the licenses of included third-party works, please consult the `NOTICE`
file.

## Release Schedule

The Eclipse BaSyx-Python SDK Team evaluates bi-monthly the newly added commits to the main branch towards the need
of releasing a new version.
If decided the changes warrant a release, it is initiated, using semantic versioning for the new release number.
If the changes do not warrant a release, the decision is postponed to the next meeting.

Additionally, security fixes may be released at any point.

## Dependencies

The BaSyx Python SDK requires the following Python packages to be installed for production usage. These dependencies are listed in
`setup.py` to be fetched automatically when installing with `pip`:
* `lxml` (BSD 3-clause License, using `libxml2` under MIT License)
* `python-dateutil` (BSD 3-clause License)
* `pyecma376-2` (Apache License v2.0)
* `urllib3` (MIT License)

Optional production usage dependencies:
* For using the Compliance Tool to validate JSON files against the JSON Schema: `jsonschema` and its
dependencies (MIT License, Apache License, PSF License)

Development/testing/documentation/example dependencies (see `requirements.txt`):
* `jsonschema` and its dependencies (MIT License, Apache License, PSF License)

Dependencies for building the documentation (see `docs/add-requirements.txt`):
* `Sphinx` and its dependencies (BSD 2-clause License, MIT License, Apache License)
* `sphinx-rtd-theme` and its dependencies (MIT License, PSF License)
* `sphinx-argparse` (MIT License)

## Getting Started

### Installation

Eclipse BaSyx Python SDK can be installed from PyPI, the Python Package Index, just as nearly every other Python package:
```bash
pip install basyx-python-sdk
```

For working with the current development version, you can also install the package directly from GitHub, using Pip's Git feature:
```bash
pip install git+https://github.com/eclipse-basyx/basyx-python-sdk.git@main
```

You may want to use a Python's `venv` or a similar tool to install BaSyx Python SDK and its dependencies only in a project-specific local environment.

### Example

The following code example shows how to create a `Submodel` with a `Property` serialize it into an XML file using the
Eclipse BaSyx Python SDK:

Create a `Submodel`:
```python
from basyx.aas import model  # Import all BaSyx Python SDK classes from the model package

identifier = 'https://acplt.org/Simple_Submodel'
submodel = model.Submodel(identifier)
```

Create a `Property` and add it to the `Submodel`:
```python
# create an external reference to a semantic description of the property
semantic_reference = model.ExternalReference(
    (model.Key(
        type_=model.KeyTypes.GLOBAL_REFERENCE,
        value='http://acplt.org/Properties/SimpleProperty'
    ),)
)
property = model.Property(
    id_short='ExampleProperty',  # Identifying string of the element within the submodel namespace
    value_type=model.datatypes.String,  # Data type of the value
    value='exampleValue',  # Value of the property
    semantic_id=semantic_reference  # set the semantic reference
)
submodel.submodel_element.add(property)
```

Serialize the `Submodel` to XML:
```python
from basyx.aas.adapter.xml import write_aas_xml_file

data: model.DictObjectStore[model.Identifiable] = model.DictObjectStore()
data.add(submodel)
write_aas_xml_file(file='Simple_Submodel.xml', data=data)
```

### Examples and Tutorials

For further examples and tutorials, check out the `basyx.aas.examples`-package. Here is a quick overview:

* [`tutorial_create_simple_aas`](./basyx/aas/examples/tutorial_create_simple_aas.py): Create an Asset Administration Shell, including an Asset object and a Submodel
* [`tutorial_storage`](./basyx/aas/examples/tutorial_storage.py): Manage a larger number of Asset Administration Shells in an ObjectStore and resolve references
* [`tutorial_serialization_deserialization`](./basyx/aas/examples/tutorial_serialization_deserialization.py): Use the JSON and XML serialization/deserialization for single objects or full standard-compliant files
* [`tutorial_aasx`](./basyx/aas/examples/tutorial_aasx.py): Export Asset Administration Shells with related objects and auxiliary files to AASX package files
* [`tutorial_backend_couchdb`](./basyx/aas/examples/tutorial_backend_couchdb.py): Use the *Backends* interface (`update()/commit()` methods) to manage and retrieve AAS objects in a CouchDB document database

### Documentation

A detailed, complete API documentation is available on Read the Docs: https://basyx-python-sdk.readthedocs.io

### Compliance Tool

The compliance tool functionality moved to [github.com/rwth-iat/aas-compliance-tool](https://github.com/rwth-iat/aas-compliance-tool).

## Contributing

For contributing with issues and code, please see our [Contribution Guideline](CONTRIBUTING.md).

### Eclipse Contributor Agreement

To contribute code to this project you need to sign the [Eclipse Contributor Agreement (ECA)](https://www.eclipse.org/legal/ECA.php).
This is done by creating an Eclipse account for your git e-mail address and then submitting the following form: https://accounts.eclipse.org/user/eca


[](https://flagopen.baai.ac.cn/)
![FlagAI](logo.png)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6052/badge)](https://bestpractices.coreinfrastructure.org/projects/6052)
[![Python application](https://github.com/FlagAI-Open/FlagAI/actions/workflows/python-app.yml/badge.svg)](https://github.com/FlagAI-Open/FlagAI/actions/workflows/python-app.yml)
![GitHub release (release name instead of tag name)](https://img.shields.io/github/v/release/FlagAI-Open/FlagAI?include_prereleases&style=social)

[简体中文](README_zh.md)

--------------------------------------------------------------------------------

FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model. Our goal is to support training, fine-tuning, and deployment of large-scale models on various downstream tasks with multi-modality.

## Why should I use FlagAI?

1. **Quickly Download Models via API**

    FlagAI provides an API that allows you to quickly download pre-trained models and fine-tune them on a wide range of datasets collected from [SuperGLUE](https://super.gluebenchmark.com/) and [CLUE](https://github.com/CLUEbenchmark/CLUE) benchmarks for both Chinese and English text.

    FlagAI now supports over 30 mainstream models, including Language Model [**Aquila**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila), multilingual text and image representation model [**AltCLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP), text-to-image generation model [**AltDiffusion**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltDiffusion) [![Huggingface space](https://img.shields.io/badge/🤗-Huggingface%20Space-cyan.svg)](https://huggingface.co/spaces/BAAI/bilingual_stable_diffusion), [**WuDao GLM**](/docs/GLM.md) (with a maximum of 10 billion parameters), [**EVA-CLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/EVA_CLIP), **OPT**, **BERT**, **RoBERTa**, **GPT2**, **T5**, **ALM**, and models from **Huggingface Transformers**, etc.

2. **Parallel train with fewer than 10 lines of code**

	Backed by the four most popular data/model parallel libraries -- [PyTorch](https://pytorch.org/), [Deepspeed](https://www.deepspeed.ai/), [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), [BMTrain](https://github.com/OpenBMB/BMTrain) -- FlagAI allows for seamless integration between them, enabling users to parallel their training/testing process with fewer than ten lines of code.

3. **Conveniently use the few-shot learning toolkits**

    FlagAI also provides [prompt-learning](/docs/TUTORIAL_7_PROMPT_LEARNING.md) toolkit for few-shot tasks.

4. **Particularly good at Chinese tasks**

    These models can be applied to (Chinese/English) Text, for tasks like text classification, information extraction, question answering, summarization, and text generation, with a particular focus on Chinese tasks.

## Toolkits and Pre-trained Models

> The code is partially based on [GLM](https://github.com/THUDM/GLM), [Transformers](https://github.com/huggingface/transformers)，[timm](https://github.com/rwightman/pytorch-image-models) and [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/Megatron-LM).

### Toolkits

| Name       | Description       | Examples            |
|:-------------- |:---------- |:------------------------------------------------------ |
| 	`GLM_custom_pvp` | Customizing PET templates   | [README.md](http:///examples/glm_custom_pvp/README.md) |
| `GLM_ptuning`    | p-tuning tool | ——                                                     |
| `BMInf-generate` | Accelerating generation | [README.md](http:///examples/bminf_generate/README.md) |

### Pre-trained Models

|   Model          |  Task    | Train | Finetune | Inference/Generate | Examples       |
| :---------------- | :------- | :-- |:-- | :-- | :--------------------------------------------- |
| Aquila      | Natural Language Processing  | ✅  | ✅  | ✅  | [README.md](examples/Aquila/README.md)
| ALM          | Arabic Text Generation  |  ✅  | ❌  | ✅  | [README.md](/examples/ALM/README.md)  |
| AltCLIP       | Image-Text Matching  | ✅  | ✅  | ✅  | [README.md](/examples/AltCLIP/README.md)   |
| AltCLIP-m18      | Image-Text Matching  | ✅  | ✅  | ✅  | [README.md](examples/AltCLIP-m18/README.md)   |
| AltDiffusion    | Text-to-Image Generation    | ❌  | ❌  | ✅  | [README.md](/examples/AltDiffusion/README.md)    |
| AltDiffusion-m18    | Text-to-Image Generation,supporting 18 languages    | ❌  | ❌  | ✅  |[README.md](/examples/AltDiffusion-m18/README.md)   |
| BERT-title-generation-english     | English Title Generation | ✅  | ❌  | ✅  | [README.md](/examples/bert_title_generation_english/README.md) |
| CLIP           | Image-Text Matching    | ✅  | ❌  | ✅  | ——   |
| CPM3-finetune       | Text Continuation   | ❌  | ✅  | ❌  | ——    |
| CPM3-generate    | Text Continuation  | ❌  | ❌  | ✅  | ——   |
| CPM3_pretrain    | Text Continuation  | ✅  | ❌  | ❌  | ——        |
| CPM_1     | Text Continuation   | ❌  | ❌  | ✅  | [README.md](/examples/cpm_1/README.md)      |
| EVA-CLIP                          | Image-Text Matching  | ✅  | ✅  | ✅  | [README.md](/examples/EVA_CLIP/README.md)                             |
| Galactica       | Text Continuation    | ❌  | ❌  | ✅  | ——      |
| GLM-large-ch-blank-filling        | Blank Filling     | ❌  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_11_GLM_BLANK_FILLING_QA.md)               |
| GLM-large-ch-poetry-generation    | Poetry Generation     | ✅  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_13_GLM_EXAMPLE_PEOTRY_GENERATION.md)       |
| GLM-large-ch-title-generation     | Title Generation   | ✅  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_12_GLM_EXAMPLE_TITLE_GENERATION.md)        |
| GLM-pretrain         | Pre-Train    | ✅  | ❌  | ❌  | ——   |
| GLM-seq2seq        | Generation    | ✅  | ❌  | ✅  | ——     |
| GLM-superglue      | Classification  | ✅  | ❌  | ❌  | ——     |
| GPT-2-text-writting      | Text Continuation   | ❌  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_18_GPT2_WRITING.md)        |
| GPT2-text-writting                | Text Continuation   | ❌  | ❌  | ✅  | —— |
| GPT2-title-generation             | Title Generation   | ❌  | ❌  | ✅  | ——  |
| OPT                               | Text Continuation   | ❌  | ❌  | ✅  | [README.md](/examples/opt/README.md) |
| RoBERTa-base-ch-ner               | Named Entity Recognition| ✅  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_17_BERT_EXAMPLE_NER.md)     |
| RoBERTa-base-ch-semantic-matching |Semantic Similarity Matching | ✅  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_16_BERT_EXAMPLE_SEMANTIC_MATCHING.md)      |
| RoBERTa-base-ch-title-generation  | Title Generation     | ✅  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_15_BERT_EXAMPLE_TITLE_GENERATION.md)       |
| RoBERTa-faq      |   Question-Answer   | ❌  | ❌  | ✅  | [README.md](/examples/roberta_faq/README.md) |
| Swinv1                            | Image Classification | ✅  | ❌  | ✅  | ——  |
| Swinv2                            | Image Classification   | ✅  | ❌  | ✅  | ——     |
| T5-huggingface-11b                | Train   | ✅  | ❌  | ❌  | [TUTORIAL](/docs/TUTORIAL_14_HUGGINGFACE_T5.md)                      |
| T5-title-generation               | Title Generation     | ❌  | ❌  | ✅  | [TUTORIAL](/docs/TUTORIAL_19_T5_EXAMPLE_TITLE_GENERATION.md)                |
| T5-flagai-11b                     | Pre-Train  | ✅  | ❌  | ❌  | ——    |
| ViT-cifar100                      |  Pre-Train  | ✅  | ❌  | ❌  | —— |

> * More excamples in  [./examples](https://github.com/FlagAI-Open/FlagAI/tree/master/examples)
> * More tutorials in [./docs](https://github.com/FlagAI-Open/FlagAI/tree/master/doc)

## Contributing

Thanks for your interest in contributing! There are many ways to get involved;
start with our [contributor guidelines](CONTRIBUTING.md) and then
check these [open issues](https://github.com/FlagAI-Open/FlagAI/issues) for specific tasks.

## Contact us

Welcome to raise your questions or feature requests on [GitHub Issues](https://github.com/FlagAI-Open/FlagAI/issues) , and share your experience on the  [Discussions](https://github.com/FlagAI-Open/FlagAI/discussions) board.

* Official email: open.platform@baai.ac.cn.
* Zhihu: [FlagAI](https://www.zhihu.com/people/95-22-20-18)
* Scan the qrcode to join the WeChat group for communication:

## Quick Start
We provide many models which are trained to perform different tasks. You can load these models by AutoLoader to make prediction. See more in `FlagAI/quickstart`.

### Requirements and Installation
* Python version >= 3.8
* PyTorch version >= 1.8.0
* [Optional] For training/testing models on GPUs, you'll also need to install CUDA and NCCL

- To install FlagAI with pip:
```shell
pip install -U flagai
```

- [Optional] To install FlagAI and develop locally:

```shell
git clone https://github.com/FlagAI-Open/FlagAI.git
python setup.py install
```

- [Optional] For faster training, install NVIDIA's [apex](https://github.com/NVIDIA/apex)
```
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```
- [Optional] For ZeRO optimizers, install [DEEPSPEED](https://github.com/microsoft/DeepSpeed) (>= 0.7.7)
```
git clone https://github.com/microsoft/DeepSpeed
cd DeepSpeed
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install -e .
ds_report # check the deespeed status
```
- [Optional] For BMTrain training, install [BMTrain](https://github.com/OpenBMB/BMTrain) (>= 0.2.2)
```
git clone https://github.com/OpenBMB/BMTrain
cd BMTrain
python setup.py install
```
- [Optional] For BMInf low-resource inference, install [BMInf](https://github.com/OpenBMB/BMInf)
```
pip install bminf

```
- [Optional] For Flash Attention, install [Flash-attention](https://github.com/HazyResearch/flash-attention) (>=1.0.2)
```
pip install flash-attn
```

- [Tips] For single-node docker environments, we need to set up ports for your ssh. e.g., root@127.0.0.1 with port 711
```
>>> vim ~/.ssh/config
Host 127.0.0.1
    Hostname 127.0.0.1
    Port 7110
    User root
```
- [Tips] For multi-node docker environments, generate ssh keys and copy the public key to all nodes (in `~/.ssh/`)
```
>>> ssh-keygen -t rsa -C "xxx@xxx.com"
```

### Load model and tokenizer
We provide the AutoLoad class to load the model and tokenizer quickly, for example:
```python
from flagai.auto_model.auto_loader import AutoLoader

auto_loader = AutoLoader(
    task_name="title-generation",
    model_name="BERT-base-en"
)
model = auto_loader.get_model()
tokenizer = auto_loader.get_tokenizer()
```
This example is for the `title_generation` task, and you can also model other tasks by modifying the `task_name`.
Then you can use the model and tokenizer to fine-tune or test.

### Examples

#### 1. Predictor

We provide the `Predictor` class to predict for different tasks, for example:

```python
from flagai.model.predictor.predictor import Predictor
predictor = Predictor(model, tokenizer)
test_data = [
    "Four minutes after the red card, Emerson Royal nodded a corner into the path of the unmarked Kane at the far post, who nudged the ball in for his 12th goal in 17 North London derby appearances. Arteta's misery was compounded two minutes after half-time when Kane held the ball up in front of goal and teed up Son to smash a shot beyond a crowd of defenders to make it 3-0.The goal moved the South Korea talisman a goal behind Premier League top scorer Mohamed Salah on 21 for the season, and he looked perturbed when he was hauled off with 18 minutes remaining, receiving words of consolation from Pierre-Emile Hojbjerg.Once his frustrations have eased, Son and Spurs will look ahead to two final games in which they only need a point more than Arsenal to finish fourth.",
]

for text in test_data:
    print(
        predictor.predict_generate_beamsearch(text,
                                              out_max_length=50,
                                              beam_size=3))
```
This example is for the `seq2seq` task, where we can get `beam-search` results by calling the `predict_generate_beamsearch` function. In addition, we also support prediction for tasks such as `NER` and `title generate`.

#### 2. NER

```python
from flagai.auto_model.auto_loader import AutoLoader
from flagai.model.predictor.predictor import Predictor

task_name = "ner"
model_name = "RoBERTa-base-ch"
target = ["O", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-PER", "I-PER"]
maxlen = 256

auto_loader = AutoLoader(task_name,
                         model_name=model_name,
                         load_pretrain_params=True,
                         class_num=len(target))

model = auto_loader.get_model()
tokenizer = auto_loader.get_tokenizer()

predictor = Predictor(model, tokenizer)

test_data = [
    "6月15日，河南省文物考古研究所曹操高陵文物队公开发表声明承认：“从来没有说过出土的珠子是墓主人的",
    "4月8日，北京冬奥会、冬残奥会总结表彰大会在人民大会堂隆重举行。习近平总书记出席大会并发表重要讲话。在讲话中，总书记充分肯定了北京冬奥会、冬残奥会取得的优异成绩，全面回顾了7年筹办备赛的不凡历程，深入总结了筹备举办北京冬奥会、冬残奥会的宝贵经验，深刻阐释了北京冬奥精神，对运用好冬奥遗产推动高质量发展提出明确要求。",
    "当地时间8日，欧盟委员会表示，欧盟各成员国政府现已冻结共计约300亿欧元与俄罗斯寡头及其他被制裁的俄方人员有关的资产。",
    "这一盘口状态下英国必发公司亚洲盘交易数据显示博洛尼亚热。而从欧赔投注看，也是主队热。巴勒莫两连败，",
]

for t in test_data:
    entities = predictor.predict_ner(t, target, maxlen=maxlen)
    result = {}
    for e in entities:
        if e[2] not in result:
            result[e[2]] = [t[e[0]:e[1] + 1]]
        else:
            result[e[2]].append(t[e[0]:e[1] + 1])
    print(f"result is {result}")
```

#### 3. Semantic Matching example

```python
from flagai.auto_model.auto_loader import AutoLoader
from flagai.model.predictor.predictor import Predictor

maxlen = 256

auto_loader = AutoLoader("semantic-matching",
                         model_name="RoBERTa-base-ch",
                         load_pretrain_params=True,
                         class_num=2)
model = auto_loader.get_model()
tokenizer = auto_loader.get_tokenizer()

predictor = Predictor(model, tokenizer)

test_data = [["后悔了吗", "你有没有后悔"], ["打开自动横屏", "开启移动数据"],
             ["我觉得你很聪明", "你聪明我是这么觉得"]]

for text_pair in test_data:
    print(predictor.predict_cls_classifier(text_pair))

```

## LICENSE

The majority of FlagAI is licensed under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0), however portions of the project are available under separate license terms:

* Megatron-LM is licensed under the [Megatron-LM license](https://github.com/NVIDIA/Megatron-LM/blob/main/LICENSE)
* GLM is licensed under the [MIT license](https://github.com/THUDM/GLM/blob/main/LICENSE)
* AltDiffusion is licensed under the [CreativeML Open RAIL-M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license)

## News
- [9 June 2023] release v1.7.0, Support Aquila [#324](https://github.com/FlagAI-Open/FlagAI/pull/324);
- [31 Mar 2023] release v1.6.3, Support AltCLIP-m18 [#303](https://github.com/FlagAI-Open/FlagAI/pull/303) and AltDiffusion-m18 [#302](https://github.com/FlagAI-Open/FlagAI/pull/302);
- [17 Mar 2023] release v1.6.2, Support application of new optimizers [#266](https://github.com/FlagAI-Open/FlagAI/pull/266), and added a new gpt model name 'GPT2-base-en' for English;
- [2 Mar 2023] release v1.6.1, Support Galactica model [#234](https://github.com/FlagAI-Open/FlagAI/pull/234); BMInf, a low-resource inference package [#238](https://github.com/FlagAI-Open/FlagAI/pull/238), and examples for p-tuning [#227](https://github.com/FlagAI-Open/FlagAI/pull/238)
- [12 Jan 2023] release v1.6.0, support a new parallel lib called [**BMTrain**](https://github.com/OpenBMB/BMTrain) and integate [**Flash Attention**](https://github.com/HazyResearch/flash-attention) to speedup training of BERT and ViT models, examples in [FlashAttentionBERT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/bert_title_generation_english/train_flash_atten.py) and [FlashAttentionViT](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/vit_cifar100/train_single_gpu_flash_atten.py). Also add the contrastive search based text generation method [**SimCTG**](https://github.com/yxuansu/SimCTG) and DreamBooth finetuning based on AltDiffusion, examples in [AltDiffusionNaruto](https://github.com/FlagAI-Open/FlagAI/blob/master/examples/AltDiffusion/dreambooth.py).
- [28 Nov 2022] release v1.5.0, support 1.1B [**EVA-CLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/EVA_CLIP) and [ALM: A large Arabic Language Model based on GLM], examples in [**ALM**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/ALM)
- [10 Nov 2022] release v1.4.0, support [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v1), examples in [**AltCLIP**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP) and [**AltDiffusion**](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltDiffusion)
- [29 Aug 2022] release v1.3.0, Added CLIP module and redesigned tokenizer APIs in [#81](https://github.com/FlagAI-Open/FlagAI/pull/81)
- [21 Jul 2022] release v1.2.0, ViTs are supported in [#71](https://github.com/FlagAI-Open/FlagAI/pull/71)
- [29 Jun 2022] release v1.1.0, support OPTs downloading and inference/fine-tuning [#63](https://github.com/FlagAI-Open/FlagAI/pull/63)
- [17 May 2022] made our first contribution in [#1](https://github.com/FlagAI-Open/FlagAI/pull/1)

## Platforms supported

## Misc

### ↳ Stargazers, thank you for your support!
[![Stargazers repo roster for @FlagAI-Open/FlagAI](https://reporoster.com/stars/FlagAI-Open/FlagAI)](https://github.com/FlagAI-Open/FlagAI/stargazers)

### ↳ Forkers, thank you for your support!
[![Forkers repo roster for @FlagAI-Open/FlagAI](https://reporoster.com/forks/FlagAI-Open/FlagAI)](https://github.com/FlagAI-Open/FlagAI/network/members)

### ↳ Star History

![Star History Chart](https://api.star-history.com/svg?repos=FlagAI-Open/FlagAI&type=Date)]


# pyQuARC

### Open Source Library for Earth Observation Metadata Quality Assessment

[![DOI](https://zenodo.org/badge/153786129.svg)](https://zenodo.org/doi/10.5281/zenodo.10724716)

## Introduction

The pyQuARC (*pronounced "pie-quark"*) library was designed to read and evaluate descriptive metadata used to catalog Earth observation data products and files. This type of metadata focuses and limits attention to important aspects of data, such as the spatial and temporal extent, in a structured manner that can be leveraged by data catalogs and other applications designed to connect users to data. Therefore, poor quality metadata (e.g. inaccurate, incomplete, improperly formatted, inconsistent) can yield subpar results when users search for data. Metadata that inaccurately represents the data it describes risks matching users with data that does not reflect their search criteria and, in the worst-case scenario, can make data impossible to find.

Given the importance of high quality metadata, it is necessary that metadata be regularly assessed and updated as needed. pyQuARC is a tool that can help streamline the process of assessing metadata quality by automating it as much as possible. In addition to basic validation checks (e.g. adherence to the metadata schema, controlled vocabularies, and link checking), pyQuARC flags opportunities to improve or add contextual metadata information to help the user connect to, access, and better understand the data product. pyQuARC also ensures that information common to both data product (i.e. collection) and the file-level (i.e. granule) metadata are consistent and compatible. As open source software, pyQuARC can be adapted and customized to allow for quality checks unique to different needs.

## pyQuARC Base Package

pyQuARC was specifically designed to assess metadata in NASA’s [Common Metadata Repository (CMR)](https://earthdata.nasa.gov/eosdis/science-system-description/eosdis-components), which is a centralized metadata repository for all of NASA’s Earth observation data products. In addition to NASA’s ~9,000 data products, the CMR also holds metadata for over 40,000 additional Earth observation data products submitted by external data partners. The CMR serves as the backend for NASA’s Earthdata Search (search.earthdata.nasa.gov) and is also the authoritative metadata source for NASA’s [Earth Observing System Data and Information System (EOSDIS).](https://earthdata.nasa.gov/eosdis)

pyQuARC was developed by a group called the [Analysis and Review of the CMR (ARC)](https://earthdata.nasa.gov/esds/impact/arc) team. The ARC team conducts quality assessments of NASA’s metadata records in the CMR, identifies opportunities for improvement in the metadata records, and collaborates with the data archive centers to resolve any identified issues. ARC has developed a [metadata quality assessment framework](http://doi.org/10.5334/dsj-2021-017) which specifies a common set of assessment criteria. These criteria focus on correctness, completeness, and consistency with the goal of making data more discoverable, accessible, and usable. The ARC metadata quality assessment framework is the basis for the metadata checks that have been incorporated into pyQuARC base package. Specific quality criteria for each CMR metadata element is documented in the following wiki:
[https://wiki.earthdata.nasa.gov/display/CMR/CMR+Metadata+Best+Practices%3A+Landing+Page](https://wiki.earthdata.nasa.gov/display/CMR/CMR+Metadata+Best+Practices%3A+Landing+Page)

There is an “ARC Metadata QA/QC” section on the wiki page for each metadata element that lists quality criteria categorized by level of [priority. Priority categories](https://wiki.earthdata.nasa.gov/display/CMR/ARC+Priority+Matrix) are designated as high (red), medium (yellow), or low (blue), and are intended to communicate the importance of meeting the specified criteria.

The CMR is designed around its own metadata standard called the [Unified Metadata Model (UMM).](https://earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/cmr/umm) In addition to being an extensible metadata model, the UMM also provides a cross-walk for mapping between the various CMR-supported metadata standards. CMR-supported metadata standards currently include:
* [DIF10](https://earthdata.nasa.gov/esdis/eso/standards-and-references/directory-interchange-format-dif-standard) (Collection/Data Product-level only)
* [ECHO10](https://earthdata.nasa.gov/esdis/eso/standards-and-references/echo-metadata-standard) (Collection/Data Product and Granule/File-level metadata)
* [ISO19115-1 and ISO19115-2](https://earthdata.nasa.gov/esdis/eso/standards-and-references/iso-19115) (Collection/Data Product and Granule/File-level metadata)
* [UMM-JSON](https://wiki.earthdata.nasa.gov/display/CMR/UMM+Documents) (UMM)
	* UMM-C (Collection/Data Product-level metadata)
	* UMM-G (Granule/File-level metadata)
	* UMM-S (Service metadata)
	* UMM-T (Tool metadata)

pyQuARC supports DIF10 (collection only), ECHO10 (collection and granule), UMM-C, and UMM-G standards. At this time, there are no plans to add ISO 19115 or UMM-S/T specific checks. **Note that pyQuARC development is still underway, so further enhancements and revisions are planned.**

**For inquiries, please email: jenny.wood@uah.edu**

## pyQuARC as a Service (QuARC)

QuARC is pyQuARC deployed as a service and can be found here: https://quarc.nasa-impact.net/docs/.

QuARC is still in beta but is regularly synced with the latest version of pyQuARC on GitHub. Fully cloud-native, the architecture diagram of QuARC is shown below:

![QuARC](https://user-images.githubusercontent.com/17416300/179866276-7c025699-01a1-4d3e-93cd-50e12c5a5ec2.png)

## Architecture

![pyQuARC Architecture](/images/architecture.png)

The Downloader is used to obtain a copy of a metadata record of interest from the CMR. This is accomplished using a [CMR API query,](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) where the metadata record of interest is identified by its unique identifier in the CMR (concept_id). CMR API documentation can be found here:
[https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html)

There is also the option to select and run pyQuARC on a metadata record already downloaded to your local desktop.

The `checks.json` file includes a comprehensive list of rules. Each rule is specified by its `rule_id,` associated function, and any dependencies on specific metadata elements.

The `rule_mapping.json` file specifies which metadata element(s) each rule applies to. The `rule_mapping.json` also references the `messages.json` file which includes messages that can be displayed when a check passes or fails.

Furthermore, the `rule_mapping.json` file specifies the level of severity associated with a failure. If a check fails, it will be assigned a severity category of “error”, “warning”, or "info.” These categories correspond to priority categorizations in [ARC’s priority matrix](https://wiki.earthdata.nasa.gov/display/CMR/ARC+Priority+Matrix) and communicate the importance of the failed check, with “error” being the most critical category, “warning” indicating a failure of medium priority, and “info” indicating a minor issue or inconsistency. Default severity values are assigned based on ARC’s metadata quality assessment framework, but can be customized to meet individual needs.

## Customization
pyQuARC is designed to be customizable. Output messages can be modified using the `messages_override.json` file - any messages added to `messages_override.json` will display over the default messages in the `message.json` file. Similarly, there is a `rule_mapping_override.json` file which can be used to override the default settings for which rules/checks are applied to which metadata elements.

There is also the opportunity for more sophisticated customization. New QA rules can be added and existing QA rules can be edited or removed. Support for new metadata standards can be added as well. Further details on how to customize pyQuARC will be provided in the technical user’s guide below.

While the pyQuARC base package is currently managed by the ARC team, the long term goal is for it to be owned and governed by the broader EOSDIS metadata community.

## Install/User’s Guide
### Running the program

*Note:* This program requires `Python 3.8` installed in your system.

**Clone the repo:** [https://github.com/NASA-IMPACT/pyQuARC/](https://github.com/NASA-IMPACT/pyQuARC/)

**Go to the project directory:** `cd pyQuARC`

**Create a python virtual environment:** `python -m venv env`

**Activate the environment:** `source env/bin/activate`

**Install the requirements:** `pip install -r requirements.txt`

**Run `main.py`:**

```plaintext
▶ python pyQuARC/main.py -h
usage: main.py [-h] [--query QUERY | --concept_ids CONCEPT_IDS [CONCEPT_IDS ...]] [--file FILE | --fake FAKE] [--format [FORMAT]] [--cmr_host [CMR_HOST]]
               [--version [VERSION]]

optional arguments:
  -h, --help                Show this help message and exit
  --query QUERY             CMR query URL.
  --concept_ids CONCEPT_IDS [CONCEPT_IDS ...]
                            List of concept IDs.
  --file FILE               Path to the test file, either absolute or relative to the root dir.
  --fake FAKE               Use a fake content for testing.
  --format [FORMAT]         The metadata format. Choices are: echo-c (echo10 collection), echo-g (echo10 granule), dif10 (dif10 collection), umm-c (umm-json collection),
                        umm-g (umm-json granules)
  --cmr_host [CMR_HOST]     The cmr host base url. Default is: https://cmr.earthdata.nasa.gov
  --version [VERSION]       The revision version of the collection. Default is the latest version.

```
To test a local file, use the `--file` argument. Give it either an absolute file path or a file path relative to the project root directory.

Example:
```
▶ python pyQuARC/main.py --file "tests/fixtures/test_cmr_metadata.echo10"
```
or
```
▶ python pyQuARC/main.py --file "/Users/batman/projects/pyQuARC/tests/fixtures/test_cmr_metadata.echo10"
```

### Adding a custom rule

To add a custom rule, follow the following steps:

**Add an entry to the `schemas/rule_mapping.json` file in the form:**

```json
"rule_id": "<An id for the rule in snake case>": {
    "rule_name": "<Name of the Rule>",
    "fields_to_apply": {
        "<metadata format (eg. echo-c)>": {
            "fields": [
                "<The primary field1 to apply to (full path separated by /)>",
                "<Related field 11>",
                "<Related field 12>",
                "<Related field ...>",
                "<Related field 1n>",
            ],
            "relation": "relation_between_the_fields_if_any",
            "dependencies": [
                [
                    "<any dependent check that needs to be run before this check (if any), for this specific metadata format>",
                    "<field to apply this dependent check to (if any)>"
                ]
            ]
        },
        "echo-g": {
            "fields": [
                "<The primary field2 to apply to (full path separated by /)>",
                "<Related field 21>",
                "<Related field 22>",
                "<Related field ...>",
                "<Related field 2n>",
            ],
            "relation": "relation_between_the_fields_if_any",
            "data": [ "<any external data that you want to send to the rule for this specific metadata format>" ]
        }
    },
    "data" : [ "<any external data that you want to send to the rule>" ],
    "check_id": "< one of the available checks, see CHECKS.md, or custom check if you are a developer>"
}
```

An example:

```json
"data_update_time_logic_check": {
    "rule_name": "Data Update Time Logic Check",
    "fields_to_apply": {
        "echo-c": [
            {
                "fields": [
                    "Collection/LastUpdate",
                    "Collection/InsertTime"
                ],
                "relation": "gte"
            }
        ],
        "echo-g": [
            {
                "fields": [
                    "Granule/LastUpdate",
                    "Granule/InsertTime"
                ],
                "relation": "gte"
            }
        ],
        "dif10": [
            {
                "fields": [
                    "DIF/Metadata_Dates/Data_Last_Revision",
                    "DIF/Metadata_Dates/Data_Creation"
                ],
                "relation": "gte",
                "dependencies": [
                    [
                        "date_or_datetime_format_check"
                    ]
                ]
            }
        ]
    },
    "severity": "info",
    "check_id": "datetime_compare"
},
```

`data` is any external data that you want to pass to the check. For example, for a `controlled_keywords_check`, it would be the controlled keywords list:

```json
"data": [ ["keyword1", "keyword2"] ]
```

`check_id` is the id of the corresponding check from `checks.json`. It'll usually be one of the available checks. An exhaustive list of all the available checks can be found in [CHECKS.md](./CHECKS.md).

**If you're writing your own custom check to `schemas/checks.json`:**

Add an entry in the format:

```json
"<a check id>": {
	"data_type": "<the data type of the value>",
	"check_function": "<the function that implements the check>",
	"dependencies": [
		"<any dependent check that needs to be run before this check>"
	],
	"description": "<description of the check>",
	"available": <check availability, either true or false>
},

```

The `data_type` can be `datetime`, `string`, `url` or `custom`.

The `check_function` should be either one of the available functions, or your own custom function.

An example:

```json
"date_compare": {
	"data_type": "datetime",
	"check_function": "compare",
	"dependencies": [
		"datetime_format_check"
	],
	"description": "Compares two datetimes based on the relation given.",
	"available": true
},
```

**If you’re writing your own check function:**

Locate the validator file based on the `data_type` of the check in `code/` directory. It is in the form: `<data_type>_validator.py`. Example: `string_validator.py`, `url_validator.py`, etc.

Write a `@staticmethod` member method in the class for that particular check. See examples in the file itself. The return value should be in the format:

```json
{
	"valid": <the_validity_based_on_the_check>,
	"value": <the_value_of_the_field_in_user_friendly_format>
}
```

You can re-use any functions that are already there to reduce redundancy.

**Adding output messages to checks**:

Add an entry to the `schemas/check_messages_override.json` file like this:

```json
{
	"check_id": "<The id of the check/rule>",
	"message": {
		"success": "<The message to show if the check succeeds>",
		"failure": "<The message to show if the check fails>",
		"warning": "<The warning message>"
	},
	"help": {
		"message": "<The help message if any.>",
		"url": "<The help url if any.>"
	},
	"remediation": "<The remediation step to make the check valid.>"
}
```

An example:

```json
{
	"check_id": "abstract_length_check",
	"message": {
		"success": "The length is correct.",
		"failure": "The length of the field should be less than 100. The current length is `{}`.",
		"warning": "Make sure length is 100."
	},
	"help": {
		"message": "The length of the field can only be less than 100 characters.",
		"url": "www.lengthcheckurl.com"
	},
	"remediation": "A remedy."
}
```
**Note:** See the `{}` in the failure message above? It is a placeholder for any value you want to show in the output message. To fill this placeholder with a particular value, you have to return that value from the check function that you write. You can have as many placeholders as you like, you just have to return that many values from your check function.

An example:
Suppose you have a check function:

```python
@staticfunction
def is_true(value1, value2):
	return {
		"valid": value1 and value2,
		"value": [value1, value2]
	}
```

And a message:

```json
...
	"failure": "The values `{}` and `{}` do not amount to a true value",
...
```
Then, if the check function receives input `value1=0` and `value2=1`, the output message will be:

```plaintext
The values 0 and 1 do not amount to a true value
```

### Using as a package
*Note:* This program requires `Python 3.8` installed in your system.

**Clone the repo:** [https://github.com/NASA-IMPACT/pyQuARC/](https://github.com/NASA-IMPACT/pyQuARC/)

**Go to the project directory:** `cd pyQuARC`

**Install package:** `python setup.py install`

**To check if the package was installed correctly:**

```python
▶ python
>>> from pyQuARC import ARC
>>> validator = ARC(fake=True)
>>> validator.validate()
>>> ...
```

**To provide locally installed file:**

```python
▶ python
>>> from pyQuARC import ARC
>>> validator = ARC(file_path="<path to metadata file>")
>>> validator.validate()
>>> ...
```

**To provide rules for new fields or override:**

```python
▶ cat rule_override.json
{
    "data_update_time_logic_check": {
        "rule_name": "Data Update Time Logic Check",
        "fields_to_apply": [
            {
                "fields": [
                    "Collection/LastUpdate",
                    "Collection/InsertTime"
                ],
                "relation": "lte"
            }
        ],
        "severity": "info",
        "check_id": "date_compare"
    },
    "new_field": {
        "rule_name": "Check for new field",
        "fields_to_apply": [
            {
                "fields": [
                    "<new field name>",
                    "<other new field name>",
                ],
                "relation": "lte"
            }
        ],
        "severity": "info",
        "check_id": "<check_id>"
    }
}
▶ python
>>> from pyQuARC import ARC
>>> validator = ARC(checks_override="<path to rule_override.json>")
>>> validator.validate()
>>> ...
```

**To provide custom messages for new or old fields:**

```python
▶ cat messages_override.json
{
    "data_update_time_logic_check": {
        "failure": "The UpdateTime `{}` comes after the provided InsertTime `{}`.",
        "help": {
            "message": "",
            "url": "https://wiki.earthdata.nasa.gov/display/CMR/Data+Dates"
        },
        "remediation": "Everything is alright!"
    },
    "new_check": {
        "failure": "Custom check for `{}` and `{}.",
        "help": {
            "message": "",
            "url": "https://wiki.earthdata.nasa.gov/display/CMR/Data+Dates"
        },
        "remediation": "<remediation steps>"
    }
}
▶ python
>>> from pyQuARC import ARC
>>> validator = ARC(checks_override="<path to rule_override.json>", messages_override=<path to messages_override.json>)
>>> validator.validate()
>>> ...
```


# Autonomous Driving Data Framework(ADDF)

<img src="https://github.com/awslabs/autonomous-driving-data-framework/blob/main/docs/images/logo.png?raw=true" width="300" alt="ADDF logo">

![Static Checking](https://github.com/awslabs/autonomous-driving-data-framework/workflows/Module%20Checks/badge.svg?branch=main)
![Integration Tests](https://codebuild.us-west-2.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoicVlmcUxrUjVSZ3dyT1FtUWNPUkxwK3lWZjU2RHZJT0xGYU5jbjdQcUV2QTZaSnlKSi9TeGZ6Smttck9rdTBVYWdEblB6VG9KMTloVWp6N3VhRFIzTzhzPSIsIml2UGFyYW1ldGVyU3BlYyI6IkUzUzBKcDhxZ0szM1JzRkYiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main)

ADDF is a collection of modules for Scene Detection, Simulation (mock), Visualization, Compute, Storage, Centralized logging etc, deployed using [SeedFarmer](https://github.com/awslabs/seed-farmer) orchestration tool. ADDF allows you to build distinct, stand alone Infrastructure as code (IAAC) modules and exchange information about dependencies using metadata which can be exported from one module and imported into another. Each module can be found under the `modules` directory of this repository.

## Deployment Instructions

You can refer to the SeedFarmer [guide](https://seed-farmer.readthedocs.io/en/latest) to understand how SeedFarmer CLI can be used to bootstrap and deploy ADDF.

You can follow instructions in the Deployment Guide [Readme](docs/deployment_guide.md). You can also follow the [blogpost](https://aws.amazon.com/blogs/industries/develop-and-deploy-a-customized-workflow-using-autonomous-driving-data-framework-addf-on-aws/) for understanding ADDF in detail.

Please see the [ADDF Security and Operations Guide](https://docs.aws.amazon.com/prescriptive-guidance/latest/addf-security-and-operations/welcome.html) for in-depth recommendations on assessing, deploying, customizing, and operating ADDF.

## Different types of modules supported by ADDF

### Use-case specific Modules

| Type | Description |
| --- | --- |
|  [Rosbag Image Pipeline Module](modules/analysis/rosbag-image-pipeline/README.md)  |  Deploys a Rosbag Image pipeline for use in ADDF  |
|  [Rosbag WebViz Module](modules/demo-only/rosbag-webviz/README.md) |  Deploys and Visualizes Rosbag Data on AWS using Webviz for use in ADDF  |

### Optional Modules

| Type | Description |
| --- | --- |
|  [DataLake Buckets Module](modules/optionals/datalake-buckets/README.md) |  Deploys shared datalake buckets such as input, intermediate, output, logging, artifact buckets for use in ADDF  |

### Integration Modules

| Type | Description |
| --- | --- |
|  [DDB to Opensearch Module](modules/integration/ddb-to-opensearch/README.md)  |  This module integrates DynamoDB table with Opensearch cluster  |
|  [EKS to Opensearch Module](modules/integration/eks-to-opensearch/README.md) |  This module integrates EKS Cluster with Opensearch cluster  |
|  [EMR to Opensearch Module](modules/integration/emr-to-opensearch/README.md)  |  This module integrates EMR Cluster with Opensearch cluster  |
|  [Opensearch Proxy Module](modules/demo-only/opensearch-proxy/README.md)  |  This module deploys a Proxy server to access Opensearch cluster   |

### Simulation Modules

| Type | Description |
| --- | --- |
|  [K8s-Managed Module](modules/simulations/k8s-managed/README.md)  |  This module helps running simulations on AWS EKS, when triggered by KubernetesJob Operator from airflow environment   |
|  [AWS Batch Module](modules/simulations/batch-managed/README.md) |  This module helps running simulations on AWS Batch, when triggered by Batch Operator from airflow environment  |

### IDE Modules

| Type | Description |
| --- | --- |
|  [Self Managed JupyterHub Module](modules/demo-only/jupyter-hub/README.md)  |  This module deploys self managed JupyterHub environment on AWS EKS  |
|  [Self Managed VSCode Module](modules/demo-only/vscode-on-eks/README.md) |  This module deploys self managed VSCode environment on AWS EKS  |
|  [AWS Managed EMR Studio Module](modules/beta/emrstudio-on-eks/README.md)  |  This module deploys AWS managed EMR Studio with EMR on EKS  |

### Example Modules

| Type | Description |
| --- | --- |
|  [Example DAG Module](modules/examples/example-dags/README.md)  |  This module deploys a pattern to integrate a target DAG module to work with shared MWAA Cluster  |

## Reporting Issues

If you notice a defect, feel free to create an [Issue](https://github.com/awslabs/autonomous-driving-data-framework/issues)

### Deployment FAQ

If you need to debug a deployment in ADDF, here are few things you can checkout [Readme](docs/faq.md)


## PyPose: A Library for Robot Learning with Physics-based Optimization

![robot](https://user-images.githubusercontent.com/8695500/193484553-2da66824-4461-4aca-ad8c-b17c05bef067.png)

-----

Deep learning has had remarkable success in robotic perception, but its data-centric nature suffers when it comes to generalizing to ever-changing environments. By contrast, physics-based optimization generalizes better, but it does not perform as well in complicated tasks due to the lack of high-level semantic information and the reliance on manual parametric tuning. To take advantage of these two complementary worlds, we present PyPose: a **robotics-oriented**, **PyTorch-based** library that combines **deep perceptual models** with **physics-based optimization techniques**. Our design goal for PyPose is to make it **user-friendly**, **efficient**, and **interpretable** with a tidy and well-organized architecture. Using an **imperative style interface**, it can be easily integrated into **real-world robotic applications**.

-----

### Current Features

##### [LieTensor](https://pypose.org/docs/main/modules/)

- Lie group: [`SO3`](https://pypose.org/docs/main/generated/pypose.SO3/), [`SE3`](https://pypose.org/docs/main/generated/pypose.SE3/), [`Sim3`](https://pypose.org/docs/main/generated/pypose.Sim3/), [`RxSO3`](https://pypose.org/docs/main/generated/pypose.RxSO3/)
- Lie algebra: [`so3`](https://pypose.org/docs/main/generated/pypose.so3/), [`se3`](https://pypose.org/docs/main/generated/pypose.se3/), [`sim3`](https://pypose.org/docs/main/generated/pypose.sim3/), [`rxso3`](https://pypose.org/docs/main/generated/pypose.rxso3/)

##### [Modules](https://pypose.org/docs/main/modules/)

- System: [`LTI`](https://pypose.org/docs/main/generated/pypose.module.LTI), [`LTV`](https://pypose.org/docs/main/generated/pypose.module.LTV), [`NLS`](https://pypose.org/docs/main/generated/pypose.module.NLS)
- Filter: [`EKF`](https://pypose.org/docs/main/generated/pypose.module.EKF/), [`UKF`](https://pypose.org/docs/main/generated/pypose.module.UKF/), [`PF`](https://pypose.org/docs/main/generated/pypose.module.PF/)
- PnP Solver: [`EPnP`](https://pypose.org/docs/main/generated/pypose.module.EPnP/)
- Linear Quadratic Regulator: [`LQR`](https://pypose.org/docs/main/generated/pypose.module.LQR/)
- IMU Preintegration: [`IMUPreintegrator`](https://pypose.org/docs/main/generated/pypose.module.IMUPreintegrator/)
- ......

##### [Second-order Optimizers](https://pypose.org/docs/main/optim/)

- [`GaussNewton`](https://pypose.org/docs/main/generated/pypose.optim.GaussNewton)
- [`LevenbergMarquardt`](https://pypose.org/docs/main/generated/pypose.optim.LevenbergMarquardt/)
- ......

Want more features? [Create an issue here](https://github.com/pypose/pypose/issues) to request new features.

##### PyPose is highly efficient and supports parallel computing for Jacobian of Lie group and Lie algebra. See following comparison.

<img width="1167" alt="image" src="https://user-images.githubusercontent.com/8695500/203210668-1a90224a-ae08-4d31-b9d1-e293be75ef3e.png">

Efficiency and memory comparison of batched Lie group operations (we take Theseus performance as 1×).

More information about efficiency comparison goes to [our paper for PyPose](https://arxiv.org/abs/2209.15428).

## Getting Started

### Installation

#### Install from **pypi**
```bash
pip install pypose
```

#### Install from source

1. Requirement:

On Ubuntu, macOS, or Windows, install [PyTorch](https://pytorch.org/), then run:

```bash
pip install -r requirements/runtime.txt
```

2. Install locally:

```bash
git clone  https://github.com/pypose/pypose.git
cd pypose && python setup.py develop
```

3. Run tests

```bash
pytest
```

####  For contributors

1. Make sure the above installation is correct.

2. Go to [CONTRIBUTING.md](CONTRIBUTING.md)

#### Examples

1. The following code sample shows how to rotate random points and compute the gradient of batched rotation.

```python
>>> import torch, pypose as pp

>>> # A random so(3) LieTensor
>>> r = pp.randn_so3(2, requires_grad=True)
    so3Type LieTensor:
    tensor([[ 0.1606,  0.0232, -1.5516],
            [-0.0807, -0.7184, -0.1102]], requires_grad=True)

>>> R = r.Exp() # Equivalent to: R = pp.Exp(r)
    SO3Type LieTensor:
    tensor([[ 0.0724,  0.0104, -0.6995,  0.7109],
            [-0.0395, -0.3513, -0.0539,  0.9339]], grad_fn=<AliasBackward0>)

>>> p = R @ torch.randn(3) # Rotate random point
    tensor([[ 0.8045, -0.8555,  0.5260],
            [ 0.3502,  0.8337,  0.9154]], grad_fn=<ViewBackward0>)

>>> p.sum().backward()     # Compute gradient
>>> r.grad                 # Print gradient
    tensor([[-0.7920, -0.9510,  1.7110],
            [-0.2659,  0.5709, -0.3855]])
```

2. This example shows how to estimate batched inverse of transform by a second-order optimizer. Two usage options for a `scheduler` are provided, each of which can work independently.

```python
>>> from torch import nn
>>> import torch, pypose as pp
>>> from pypose.optim import LM
>>> from pypose.optim.strategy import Constant
>>> from pypose.optim.scheduler import StopOnPlateau

>>> class InvNet(nn.Module):

        def __init__(self, *dim):
            super().__init__()
            init = pp.randn_SE3(*dim)
            self.pose = pp.Parameter(init)

        def forward(self, input):
            error = (self.pose @ input).Log()
            return error.tensor()

>>> device = torch.device("cuda")
>>> input = pp.randn_SE3(2, 2, device=device)
>>> invnet = InvNet(2, 2).to(device)
>>> strategy = Constant(damping=1e-4)
>>> optimizer = LM(invnet, strategy=strategy)
>>> scheduler = StopOnPlateau(optimizer, steps=10, patience=3, decreasing=1e-3, verbose=True)

>>> # 1st option, full optimization
>>> scheduler.optimize(input=input)

>>> # 2nd option, step optimization
>>> while scheduler.continual():
        loss = optimizer.step(input)
        scheduler.step(loss)

>>> # Note: remove one of the above options for usage!
```

For more usage, see [Documentation](https://pypose.org/docs). For more applications, see [Examples](https://github.com/pypose/pypose/tree/main/examples).

## Citing PyPose

If you use PyPose, please cite the paper below. You may also [download it here](https://arxiv.org/abs/2209.15428).

```bibtex
@inproceedings{wang2023pypose,
  title = {{PyPose}: A Library for Robot Learning with Physics-based Optimization},
  author = {Wang, Chen and Gao, Dasong and Xu, Kuan and Geng, Junyi and Hu, Yaoyu and Qiu, Yuheng and Li, Bowen and Yang, Fan and Moon, Brady and Pandey, Abhinav and Aryan and Xu, Jiahe and Wu, Tianhao and He, Haonan and Huang, Daning and Ren, Zhongqiang and Zhao, Shibo and Fu, Taimeng and Reddy, Pranay and Lin, Xiao and Wang, Wenshan and Shi, Jingnan and Talak, Rajat and Cao, Kun and Du, Yi and Wang, Han and Yu, Huai and Wang, Shanzhao and Chen, Siyu and Kashyap, Ananth  and Bandaru, Rohan and Dantu, Karthik and Wu, Jiajun and Xie, Lihua and Carlone, Luca and Hutter, Marco and Scherer, Sebastian},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023}
}
```

More papers describing PyPose:

```bibtex
@inproceedings{zhan2023pypose,
  title = {{PyPose} v0.6: The Imperative Programming Interface for Robotics},
  author = {Zitong Zhan and Xiangfu Li and Qihang Li and Haonan He and Abhinav Pandey and Haitao Xiao and Yangmengfei Xu and Xiangyu Chen and Kuan Xu and Kun Cao and Zhipeng Zhao and Zihan Wang and Huan Xu and Zihang Fang and Yutian Chen and Wentao Wang and Xu Fang and Yi Du and Tianhao Wu and Xiao Lin and Yuheng Qiu and Fan Yang and Jingnan Shi and Shaoshu Su and Yiren Lu and Taimeng Fu and Karthik Dantu and Jiajun Wu and Lihua Xie and Marco Hutter and Luca Carlone and Sebastian Scherer and Daning Huang and Yaoyu Hu and Junyi Geng and Chen Wang},
  year = {2023},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop},
}
```


[![License](http://img.shields.io/:license-apache%202.0-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)
[![Sink Connector(Kafka version) tests](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-kafka-tests.yml/badge.svg)](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-kafka-tests.yml)
[![Sink Connector(Light-weight) Tests](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-lightweight-tests.yml/badge.svg)](https://github.com/Altinity/clickhouse-sink-connector/actions/workflows/sink-connector-lightweight-tests.yml)

### Latest Releases
https://github.com/Altinity/clickhouse-sink-connector/releases                                                                                |

# Altinity Sink Connector for ClickHouse

The Altinity Sink Connector moves data automatically from
transactional database tables in MySQL and PostgreSQL to ClickHouse
for analysis.

## Features

* Initial data dump and load
* Change data capture of new transactions using Debezium
* Automatic loading into ClickHouse
* Sources: Support for MySQL, PostgreSQL (other databases experimental)
* Target: Support for ClickHouse ReplacingMergeTree
* Able to recover/restart from failures on source or target
* Handle upstream schema changes automatically
* Checksum-based table comparisons
* Scalable to 1000s of tables
* Multiple deployment models
  * Lightweight: single process that transfers from source to target.
  * Kafka: separate source and target processes using Kafka as transport.
* Distribution as [Docker](https://hub.docker.com/layers/altinityinfra/clickhouse-sink-connector/408-97b1d3d83ef93c1b76a2b1c4d9c544dc67fbbec3-lt/images/sha256-d134bc05e50df7f63025e776ab6e3216c6622cd159eb0f2d459ea2ce8975f396?context=explore)
 container

## Getting Started

[QuickStart Guide: Lightweight (MySQL)](doc/quickstart.md)\
[QuickStart Guide: Lightweight (PostgreSQL)](doc/quickstart_postgres.md)\
[QuickStart Guide: Kafka](doc/quickstart_kafka.md)

## Blog Articles

First two are good tutorials on MySQL and PostgreSQL respectively.

- [Altinity Sink Connector (MySQL)](https://altinity.com/blog/fast-mysql-to-clickhouse-replication-announcing-the-altinity-sink-connector-for-clickhouse)
- [Altinity Sink Connector (PostgreSQL)](https://altinity.com/blog/replicating-data-from-postgresql-to-clickhouse-with-the-altinity-sink-connector)
- [ClickHouse as an analytic extension for MySQL](https://altinity.com/blog/using-clickhouse-as-an-analytic-extension-for-mysql?utm_campaign=Brand&utm_content=224583767&utm_medium=social&utm_source=linkedin&hss_channel=lcp-10955938)

## Reference Documentation

### General

* [Architecture Overview](doc/architecture.md)
* [Lightweight Sink Connect CLI](doc/sink_connector_cli.md)
* [Mutable Data Handling](doc/mutable_data.md)
* [ClickHouse Table Engine Types](doc/clickhouse_engines.md)
* [Troubleshooting](doc/Troubleshooting.md)

### Operations

* [Monitoring](doc/Monitoring.md)
* [Load Testing with Sysbench](doc/Performance.md)

### Development

* [Development](doc/development.md)
* [Testing](doc/TESTING.md)

## Roadmap

[2024 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)

## Help

File an issue or contact us on the Altinity public Slack workspace. Use
the link on the Slack badge at the top of this page.

## Contributing

Contributions to the project are welcome in any form.

* Submit issues documenting feature requests and bugs
* Submit PRs to make changes
* Talk about the project, write blog articles, or give presentations

We recommend that you file an issue before implementing feature additions
or major fixes. We are happy to provide guidance and encouragement!

## Commercial Support

Altinity is the primary maintainer of the Sink Connector. It is used
together with Altinity.Cloud as well as self-managed ClickHouse
installations.  Altinity.Cloud and is also used in self-managed
installations. Altinity offers a range of software and services related
to ClickHouse and analytic applications built on ClickHouse.

- [Official website](https://altinity.com/) - Get a high level overview of Altinity and our offerings.
- [Altinity.Cloud](https://altinity.com/cloud-database/) - Run ClickHouse in our cloud or yours.
- [Altinity Support](https://altinity.com/support/) - Get Enterprise-class support for ClickHouse and Sink Connector.
- [Slack](https://altinitydbworkspace.slack.com/join/shared_invite/zt-1togw9b4g-N0ZOXQyEyPCBh_7IEHUjdw#/shared-invite/email) - Talk directly with ClickHouse users and Altinity devs.
- [Contact us](https://hubs.la/Q020sH3Z0) - Contact Altinity with your questions or issues.
- [Free consultation](https://hubs.la/Q020sHkv0) - Get a free consultation with a ClickHouse expert today.


   Quick Start
     |
   Documentation
     |
   Join Slack

[![version](https://img.shields.io/pypi/v/label-sleuth)](https://pypi.org/project/label-sleuth/)  ![license](https://img.shields.io/github/license/label-sleuth/label-sleuth)  ![python](https://img.shields.io/badge/python-3.9%20--%203.11-blue)  ![python test](https://img.shields.io/github/actions/workflow/status/label-sleuth/label-sleuth/test_python.yml?branch=main&label=python%20tests)  ![react test](https://img.shields.io/github/actions/workflow/status/label-sleuth/label-sleuth/verify_react.yml?branch=main&label=react%20tests)  [![Slack](https://img.shields.io/badge/Slack-darkblue?logo=slack&logoColor=white)](https://join.slack.com/t/labelsleuth/shared_invite/zt-1j5tpz1jl-W~UaNEKmK0RtzK~lI3Wkxg)

[Label Sleuth](https://ibm.biz/label-sleuth) is an open source no-code system for text annotation and building text classifers. With Label Sleuth, domain experts (e.g., physicians, lawyers, psychologists) can quickly create custom NLP models by themselves, with no dependency on NLP experts.

Creating real-world NLP models typically requires a combination of two expertise - deep knowledge of the target domain, provided by domain experts, and machine learning knowledge, provided by NLP experts. Thus, domain experts are dependent on NLP experts. Label Sleuth comes to eliminate this dependency. With an intuitive UX, it escorts domain experts in the process of labeling the data and building NLP models which are tailored to their specific needs. As domain experts label examples within the system, machine learning models are being automatically trained in the background, make predictions on new examples, and provide suggestions for the users on the examples they should label next.

Label Sleuth is a no-code system, no knowledge in machine learning is need, and - it is fast to obtain a model – from task definition to a working model in just a few hours!

**Table of contents**

[Installation for end users](#installation-for-end-users-non-developers)

[Setting up a development environment](#setting-up-a-development-environment)

[Project structure](#project-structure)

[Using the system](#using-the-system)

[Customizing the system](#customizing-the-system)
* [System configuration](#system-configuration)
* [Implementing new components](#implementing-new-components)

[Reference](#reference)

## Installation for end users (non-developers)
Follow [the instructions on our website](https://www.label-sleuth.org/docs/installation.html).

## Setting up a development environment
The system requires Python 3.8 or 3.9 (other versions are currently not supported and may cause issues).
1. Clone the repository:

   `git clone git@github.com:label-sleuth/label-sleuth.git`
2. cd to the cloned directory: `cd label-sleuth`
3. Install the project dependencies using `conda` (recommended) or `pip`:
Installing with <tt>conda</tt>

- Install Anaconda https://docs.anaconda.com/anaconda/install/index.html

- Restart your console

- Use the following commands to create a new anaconda environment and install the requirements:
```bash
# Create and activate a virtual environment:
conda create --yes -n label-sleuth python=3.9
conda activate label-sleuth
# Install requirements
pip install -r requirements.txt
```

Installing with <tt>pip</tt>

Assuming python 3.8/3.9 is already installed.

- Install pip https://pip.pypa.io/en/stable/installation/

- Restart your console

- Install requirements:
```bash
pip install -r requirements.txt
```

4. Start the Label Sleuth server: run `python -m label_sleuth.start_label_sleuth`.

   By default all project files are written to `<home_directory>/label-sleuth`, to change the directory add `--output_path <your_output_path>`.

   You can add `--load_sample_corpus wiki_animals_2000_pages` to load a sample corpus into the system at startup. This fetches a collection of Wikipedia documents from the [data-examples repository](https://github.com/label-sleuth/data-examples).

   By default, the host will be `localhost` to expose the server only on the host machine. If you wish to expose the server to external communication, add `--host <IP>` for example, `--host 0.0.0.0` to listen to all IPs.

   Default port is 8000, to change the port add `--port <port_number>` to the command.

   The system can then be accessed by browsing to http://localhost:8000 (or http://localhost:<port_number>)

## Project Structure
The repository consists of a backend library, written in Python, and a frontend that uses React. A compiled version of the frontend can be found under `label_sleuth/build`.

## Using the system

See our website for a [simple tutorial](https://www.label-sleuth.org/docs/tutorial.html) that illustrates how to use the system with a sample dataset of Wikipedia pages. Before starting the tutorial, make sure you pre-load the sample dataset by running:

`python -m label_sleuth.start_label_sleuth --load_sample_corpus wiki_animals_2000_pages`.

## Customizing the system

### System configuration
The configurable parameters of the system are specified in a json file. The default configuration file is [label_sleuth/config.json](label_sleuth/config.json).

A custom configuration can be applied by passing the `--config_path` parameter to the "start_label_sleuth" command, e.g., `python -m label_sleuth.start_label_sleuth --config_path <path_to_my_configuration_json>`

Alternatively, it is possible to override specific configuration parameters at startup by adding them to the run command, e.g., `python -m label_sleuth.start_label_sleuth --changed_element_threshold 100`

**Configurable parameters:**

| Parameter                         | Description                                                                                |
|-----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `first_model_positive_threshold`  | Number of elements that must be assigned a positive label for the category in order to trigger the training of a classification model.

 _See also:_ The [training invocation](https://www.label-sleuth.org/docs/dev/model_training.html#training-invocation) documentation.                                                                                |
| `first_model_negative_threshold`  | Number of elements that must be assigned a negative label for the category in order to trigger the training of a classification model.

 _See also:_ The [training invocation](https://www.label-sleuth.org/docs/dev/model_training.html#training-invocation) documentation.                                                                                |
| `changed_element_threshold`       | Number of changes in user labels for the category -- relative to the last trained model -- that are required to trigger the training of a new model. A change can be a assigning a label (positive or negative) to an element, or changing an existing label. Note that `first_model_positive_threshold` must also be met for the training to be triggered.

 _See also:_ The [training invocation](https://www.label-sleuth.org/docs/dev/model_training.html#training-invocation) documentation.                                                                                |
| `training_set_selection_strategy` | Strategy to be used from [TrainingSetSelectionStrategy](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/training_set_selector/train_set_selector_api.py#L24). A TrainingSetSelectionStrategy determines which examples will be sent in practice to the classification models at training time - these will not necessarily be identical to the set of elements labeled by the user. For currently supported implementations see [get_training_set_selector()](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/training_set_selector/training_set_selector_factory.py).

 _See also:_ The [training set selection](https://www.label-sleuth.org/docs/dev/model_training.html#training-set-selection) documentation.                                                                     |
| `model_policy`                    | Policy to be used from [ModelPolicies](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/models/core/model_policies.py). A [ModelPolicy](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/models/policy/model_policy.py#L21) determines which type of classification model(s) will be used, and _when_ (e.g. always / only after a specific number of iterations / etc.).

 _See also:_ The [model selection](https://www.label-sleuth.org/docs/dev/model_training.html#model-selection) documentation.                                                                                |
| `active_learning_strategy`        | Strategy to be used from [ActiveLearningCatalog](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/active_learning/core/catalog.py#L22). An [ActiveLearner](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/active_learning/core/active_learning_api.py#L26) module implements the strategy for recommending the next elements to be labeled by the user, aiming to increase the efficiency of the annotation process. For currently supported implementations see the [ActiveLearningCatalog](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/active_learning/core/catalog.py#L22).

 _See also:_ The [active learning](https://www.label-sleuth.org/docs/dev/active_learning.html) documentation. |
| `precision_evaluation_size`       | Sample size to be used for estimating the precision of the current model. To be used in future versions of the system, which will provide built-in evaluation capabilities.                                                                                |
| `apply_labels_to_duplicate_texts` | Specifies how to treat elements with identical texts. If `true`, assigning a label to an element will also assign the same label to other elements which share the exact same text; if `false`, the label will only be assigned to the specific element labeled by the user.                                                                                |
| `language`                        | Specifies the chosen system-wide language. This determines some language-specific resources that will be used by models and helper functions (e.g., stop words). The list of supported languages can be found in [Languages](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/models/core/languages.py). We welcome contributions of additional languages.                                                                                |
| `login_required`                  | Specifies whether or not using the system will require user authentication. If `true`, the configuration file must also include a `users` parameter.                                                                                |
| `users`                           | Only relevant if `login_required` is `true`. Specifies the pre-defined login information in the following format:
"users":[
 {
   "username": "<predefined_username1>",
   "token":"<randomly_generated_token1>",
   "password":"<predefined_user1_password>"
 }
]
 * The list of usernames is static and currently all users have access to all the workspaces in the system.                                                                                |

### Implementing new components
Label Sleuth is a modular system. We welcome the contribution of additional implementations for the various modules, aiming to support a wider range of user needs and to harness efficient and innovative machine learning algorithms.

Below are instructions for implementing new models and active learning strategies:

Implementing a new machine learning model

   These are the steps for integrating a new classification model:
   1. Implement a new `ModelAPI`

   Machine learning models are integrated by adding a new implementation of the ModelAPI.

   The main functions are *_train()*, *load_model()* and *infer()*:

   ```python
   def _train(self, model_id: str, train_data: Sequence[Mapping], model_params: Mapping):
   ```
   - model_id
   - train_data - a list of dictionaries with at least the "text" and "label" fields. Additional fields can be passed e.g.
   *[{'text': 'text1', 'label': 1, 'additional_field': 'value1'}, {'text': 'text2', 'label': 0, 'additional_field': 'value2'}]*
   - model_params - dictionary for additional model parameters (can be None)

   ```python
   def load_model(self, model_path: str):
   ```
   - model_path: path to a directory containing all model components

   Returns an object that contains all the components that are necessary to perform inference (e.g., the trained model itself, the language recognized by the model, a trained vectorizer/tokenizer etc.).

   ```python
   def infer(self, model_components, items_to_infer) -> Sequence[Prediction]:
   ```
   - model_components: the return value of `load_model()`, i.e. an object containing all the components that are necessary to perform inference
   - items_to_infer: a list of dictionaries with at least the "text" field. Additional fields can be passed,
   e.g. *[{'text': 'text1', 'additional_field': 'value1'}, {'text': 'text2', 'additional_field': 'value2'}]*

   Returns a list of [Prediction](https://github.com/label-sleuth/label-sleuth/blob/main/label_sleuth/models/core/prediction.py#L20) objects - one for each item in *items_to_infer* - where
    Prediction.label is a boolean and Prediction.score is a float in the range [0-1].
    Additional outputs can be passed by inheriting from the base Prediction class and overriding the get_predictions_class() method.

   2. Add the newly implemented ModelAPI to `ModelsCatalog`

   3. Add one or more policies that use the new model to `ModelPolicies`

   Implementing a new active learning strategy</tt>

These are the steps for integrating a new active learning approach:

   1. Implement a new `ActiveLearner`

   Active learning modules are integrated by adding a new implementation of the ActiveLearner API.
   The function to implement is *get_per_element_score*:
   ```python
    def get_per_element_score(self, candidate_text_elements: Sequence[TextElement],
                              candidate_text_element_predictions: Sequence[Prediction], workspace_id: str,
                              dataset_name: str, category_name: str) -> Sequence[float]:
   ```
   Given sequences of text elements and the model predictions for these elements, this function returns an active learning score for each element.
   The elements with the highest scores will be recommended for the user to label next.

   2. Add the newly implemented ActiveLearner to the `ActiveLearningCatalog`

## Reference
Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang, Lucy Yip, Liat Ein-Dor, Lena Dankin, Ilya Shnayderman, Ranit Aharonov, Yunyao Li, Naftali Liberman, Philip Levin Slesarev, Gwilym Newton, Shila Ofek-Koifman, Noam Slonim and Yoav Katz (EMNLP 2022). [Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours](https://aclanthology.org/2022.emnlp-demos.16).

Please cite:
```
@inproceedings{shnarch2022labelsleuth,
	title={{L}abel {S}leuth: From Unlabeled Text to a Classifier in a Few Hours},
	author={Shnarch, Eyal and Halfon, Alon and Gera, Ariel and Danilevsky, Marina and Katsis, Yannis and Choshen, Leshem and Cooper, Martin Santillan and Epelboim, Dina and Zhang, Zheng and Wang, Dakuo and Yip, Lucy and Ein-Dor, Liat and Dankin, Lena and Shnayderman, Ilya and Aharonov, Ranit and Li, Yunyao and Liberman, Naftali and Slesarev, Philip Levin and Newton, Gwilym and Ofek-Koifman, Shila and Slonim, Noam and Katz, Yoav},
	booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing ({EMNLP}): System Demonstrations},
    	month={dec},
    	year={2022},
    	address={Abu Dhabi, UAE},
	publisher={Association for Computational Linguistics},
	url={https://aclanthology.org/2022.emnlp-demos.16},
    	pages={159--168}
}
```


# LowTechFilter（原 FutaFilter）

一款專為臺灣人設計的廣告阻擋規則。

## 為何叫做 LowTechFilter？

這是致敬[美秀集團](https://streetvoice.com/bisiugroup/)。

## 討論群組

嘿，有使用上的問題嗎？
歡迎到 Telegram 群組上討拍喔。
https://t.me/adblock_tw

## 訂閱網址

### DOM 清單（支援：AdBlock, AdGuard）

> LowTechFilter 專門處理網頁上的 DOM 元素處理。例：側邊廣告欄

- <https://filter.futa.gg/filter.txt>

> 隱藏 Google 搜尋結果裡的
> [農場文](https://content-farm-terminator.blogspot.com/2018/12/about-content-farm-terminator.html)
> 專用清單

- <https://filter.futa.gg/hide_farm_from_search.txt>

> experimental 實驗性清單，公開測試封鎖對策是否對現行服務有影響，無偵錯能力使用者不建議使用。

- <https://filter.futa.gg/experimental.txt>

> PureView 實驗性清單，將網頁不必要的元素去除，只保留最純粹的瀏覽體驗。

- <https://filter.futa.gg/PureView/news.txt>
- （行動裝置版）<https://filter.futa.gg/PureView/news_mobile.txt>

### DNS 清單（支援：AdGuard Home, AdGuard 的 DNS 過濾, Pi-hole, AdAway）

- <https://filter.futa.gg/hosts_abp.txt>
- <https://filter.futa.gg/nofarm_abp.txt>

> 由台灣 165 反詐騙提供

- <https://filter.futa.gg/TW165_abp.txt>

> 由台灣 165 反詐騙提供，並帶有覆寫功能（打開符合規則的網址後會提示已被 165 封鎖）。

- <https://filter.futa.gg/TW165-redirect.txt>

> 由台灣 165 反詐騙提供，純 domain 此為特定用途。

- <https://filter.futa.gg/TW165_domains.txt>

> 從網路上羅列的 NRD 清單，經過整理的版本，NRD (Newly Registered Domain
) 意思是近期註冊的網域，通常新註冊的網域有較高的風險，經常為詐騙集團所用，此清單提供過去 1 天致 30 天的清單
- <https://filter.futa.gg/ndr/past-01day_hosts.txt> （過去 1 天，hosts 格式）
- <https://filter.futa.gg/ndr/past-07day_abp.txt> （過去 7 天，adblock 格式）

| hosts 清單一覽 | LowTechHost                                          | TW165 台灣反詐騙                                     | TW RPZ 阻止解析清單                                      | NoFarm 農場文清單                                     | NRD 清單(過去1天)                                             |
| -------------- | ---------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------- |
| Adblock 語法            | [訂閱連結](https://filter.futa.gg/hosts_abp.txt)     | [訂閱連結](https://filter.futa.gg/TW165_abp.txt)     | [訂閱連結](https://filter.futa.gg/TWNIC-RPZ_abp.txt)     | [訂閱連結](https://filter.futa.gg/nofarm_abp.txt)     | [訂閱連結](https://filter.futa.gg/nrd/past-01day_abp.txt) |
| hosts          | [訂閱連結](https://filter.futa.gg/hosts.txt)   | [訂閱連結](https://filter.futa.gg/TW165_hosts.txt)   | [訂閱連結](https://filter.futa.gg/TWNIC-RPZ_hosts.txt)   | [訂閱連結](https://filter.futa.gg/nofarm_hosts.txt)   | [訂閱連結](https://filter.futa.gg/nrd/past-01day_hosts.txt) |
| 純網域        | [訂閱連結](https://filter.futa.gg/hosts_domains.txt) | [訂閱連結](https://filter.futa.gg/TW165_domains.txt) | [訂閱連結](https://filter.futa.gg/TWNIC-RPZ_domains.txt) | [訂閱連結](https://filter.futa.gg/nofarm_domains.txt) | [訂閱連結](https://filter.futa.gg/nrd/past-01day_domains.txt) |

### Surge 語法

> Surge 專用的語法。

- <https://filter.futa.gg/Surge/filters.txt>

## 如何安裝

請參考 [wiki](https://github.com/FutaGuard/FutaFilter/wiki)

## Q&A

Q: 我用了這個規則之後發現我的噗浪連結打不開了欸！！！！\
A: 請服用這個 [userscript](https://greasyfork.org/en/scripts/40884-plurk-no-redirector)

## 貢獻
本專案歡迎任何人貢獻，但請注意，提交貢獻前請先[閱讀](https://github.com/FutaGuard/LowTechFilter/wiki/%E5%A6%82%E4%BD%95%E8%B2%A2%E7%8D%BB%E6%88%96%E8%AB%8B%E6%B1%82%E6%96%B0%E5%A2%9E%E5%B0%81%E9%8E%96%E8%A6%8F%E5%89%87%EF%BC%9F)，若有任何疑慮，請先開 issue 討論。

## 贊助

歡迎贊助我們，所有贊助金額將完全用做 FutaGuard DNS 運作以及維護清單費用支出。
<https://core.newebpay.com/EPG/futaguard/FMaZ4E>


# KH2Randomizer

[Website](https://tommadness.github.io/KH2Randomizer/)

![Screenshot of the seed generator user interface](docs/seed-generator-screenshot.png)

## Acknowledgements

Icons and font by Televo

Special thanks to Sonicshadowsilver2 for the creation of the Garden of Assemblage mod, 1234567890num for helping bring
it to PC, and Bizkit047, Valaxor, and Desa3579 for the original Randomizer

Special thanks to Xeeynamo and the OpenKH team for their reverse engineering effort and the OpenKH Mods Manager, both of
which makes this all possible.


fakeredis: A fake version of a redis-py
=======================================

[![badge](https://img.shields.io/pypi/v/fakeredis)](https://pypi.org/project/fakeredis/)
[![CI](https://github.com/cunla/fakeredis-py/actions/workflows/test.yml/badge.svg)](https://github.com/cunla/fakeredis-py/actions/workflows/test.yml)
[![badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/cunla/b756396efb895f0e34558c980f1ca0c7/raw/fakeredis-py.json)](https://github.com/cunla/fakeredis-py/actions/workflows/test.yml)
[![badge](https://img.shields.io/pypi/dm/fakeredis)](https://pypi.org/project/fakeredis/)
[![badge](https://img.shields.io/pypi/l/fakeredis)](./LICENSE)
[![Open Source Helpers](https://www.codetriage.com/cunla/fakeredis-py/badges/users.svg)](https://www.codetriage.com/cunla/fakeredis-py)
--------------------

Documentation is hosted in https://fakeredis.readthedocs.io/

# Intro

FakeRedis is a pure-Python implementation of the Redis key-value store.

It enables running tests requiring redis server without an actual server.

It provides enhanced versions of the redis-py Python bindings for Redis. That provide the following added functionality:
A built-in Redis server that is automatically installed, configured and managed when the Redis bindings are used. A
single server shared by multiple programs or multiple independent servers. All the servers provided by
FakeRedis support all Redis functionality including advanced features such as RedisJson, GeoCommands.

See [official documentation](https://fakeredis.readthedocs.io/) for list of supported commands.

# Sponsor

fakeredis-py is developed for free.

You can support this project by becoming a sponsor using [this link](https://github.com/sponsors/cunla).


# ros2_tracing

[![GitHub CI](https://github.com/ros2/ros2_tracing/actions/workflows/test.yml/badge.svg?branch=rolling)](https://github.com/ros2/ros2_tracing/actions/workflows/test.yml)
[![codecov](https://codecov.io/gh/ros2/ros2_tracing/branch/rolling/graph/badge.svg)](https://codecov.io/gh/ros2/ros2_tracing)

Tracing tools for ROS 2.

## Overview

`ros2_tracing` provides [tracing instrumentation](#tracetools) for the core ROS 2 packages.
It also provides [tools to configure tracing](#tracing) through [a launch action](#launch-file-trace-action) and a [`ros2` CLI command](#trace-command).

`ros2_tracing` currently only supports the [LTTng](https://lttng.org/) tracer.
Consequently, it currently only supports Linux.

**Note**: make sure to use the right branch, depending on the ROS 2 distro: [use `rolling` for Rolling, `galactic` for Galactic, etc.](https://docs.ros.org/en/rolling/The-ROS2-Project/Contributing/Developer-Guide.html#branches)

## Publications & presentations

[Read the `ros2_tracing` paper!](https://arxiv.org/abs/2201.00393)
If you use or refer to `ros2_tracing`, please cite:
* C. Bédard, I. Lütkebohle, and M. Dagenais, ["ros2_tracing: Multipurpose Low-Overhead Framework for Real-Time Tracing of ROS 2,"](https://arxiv.org/abs/2201.00393) *IEEE Robotics and Automation Letters*, vol. 7, no. 3, pp. 6511–6518, 2022.

    BibTeX

    ```bibtex
    @article{bedard2022ros2tracing,
      title={ros2\_tracing: Multipurpose Low-Overhead Framework for Real-Time Tracing of ROS 2},
      author={B{\'e}dard, Christophe and L{\"u}tkebohle, Ingo and Dagenais, Michel},
      journal={IEEE Robotics and Automation Letters},
      year={2022},
      volume={7},
      number={3},
      pages={6511--6518},
      doi={10.1109/LRA.2022.3174346}
    }
    ```

This other paper leverages `ros2_tracing` to analyze and visualize the flow of messages across distributed ROS 2 systems:
* C. Bédard, P.-Y. Lajoie, G. Beltrame, and M. Dagenais, ["Message Flow Analysis with Complex Causal Links for Distributed ROS 2 Systems,"](https://arxiv.org/abs/2204.10208) *Robotics and Autonomous Systems*, vol. 161, p. 104361, 2023.

    BibTeX

    ```bibtex
    @article{bedard2023messageflow,
      title={Message flow analysis with complex causal links for distributed {ROS} 2 systems},
      author={B{\'e}dard, Christophe and Lajoie, Pierre-Yves and Beltrame, Giovanni and Dagenais, Michel},
      journal={Robotics and Autonomous Systems},
      year={2023},
      volume={161},
      pages={104361},
      doi={10.1016/j.robot.2022.104361}
    }
    ```

Finally, check out the following presentations:

* ROSCon 2023: "Improving Your Application's Algorithms and Optimizing Performance Using Trace Data" ([video](https://vimeo.com/879001159), [slides](https://roscon.ros.org/2023/talks/Improving_Your_Applications_Algorithms_and_Optimizing_Performance_Using_Trace_Data.pdf))
* ROS World 2021: "Tracing ROS 2 with ros2_tracing" ([video](https://vimeo.com/652633418), [slides](https://github.com/ros2/ros2_tracing/blob/rolling/doc/2021-10-20_ROS_World_2021_-_Tracing_ROS_2_with_ros2_tracing.pdf))

## Tutorials & demos

* ROS 2 documentation:
    * [Building ROS 2 with tracing](https://docs.ros.org/en/rolling/How-To-Guides/Building-ROS-2-with-Tracing.html)
    * [How to use `ros2_tracing` to trace and analyze an application](https://docs.ros.org/en/rolling/Tutorials/Advanced/ROS2-Tracing-Trace-and-Analyze.html)
* ROS World 2021 demo: [github.com/christophebedard/ros-world-2021-demo](https://github.com/christophebedard/ros-world-2021-demo)

## Building

As of Iron, the LTTng tracer is a ROS 2 dependency.
Therefore, ROS 2 can be traced out-of-the-box on Linux; this package does not need to be re-built.

To make sure that the instrumentation and tracepoints are available:

```
$ source /opt/ros/rolling/setup.bash  # With a binary install
$ source ./install/setup.bash  # When building from source
$ ros2 run tracetools status
Tracing enabled
```

A ROS 2 installation only includes the LTTng userspace tracer (LTTng-UST), which is all that is needed to trace ROS 2.
To trace the Linux kernel, the [LTTng kernel tracer](https://lttng.org/docs/v2.13/#doc-tracing-the-linux-kernel) must be installed separately:

```
$ sudo apt-get update
$ sudo apt-get install lttng-modules-dkms
```

For more information about LTTng, [refer to its documentation](https://lttng.org/docs/v2.13/).

### Removing the instrumentation

To build and remove all instrumentation, use `TRACETOOLS_DISABLED`:

```
$ colcon build --cmake-args -DTRACETOOLS_DISABLED=ON
```

This will remove all instrumentation from the core ROS 2 packages, and thus they will not depend on or link against the shared library provided by the [`tracetools` package](#tracetools).
This also means that LTTng is not required at build-time or at runtime.

### Excluding tracepoints

Alternatively, to only exclude the actual tracepoints, use `TRACETOOLS_TRACEPOINTS_EXCLUDED`:

```
$ colcon build --packages-select tracetools --cmake-clean-cache --cmake-args -DTRACETOOLS_TRACEPOINTS_EXCLUDED=ON
```

This will keep the instrumentation but remove all tracepoints.
This also means that LTTng is not required at build-time or at runtime.
This option can be useful, since tracepoints can be added back in or removed by simply replacing/re-building the shared library provided by the [`tracetools` package](#tracetools).

## Tracing

By default, trace data will not be generated, and thus these packages will have virtually no impact on execution.
LTTng has to be configured for tracing.
The packages in this repo provide two options: a [command](#trace-command) and a [launch file action](#launch-file-trace-action).

**Note**: tracing must be started before the application is launched.
Metadata is recorded during the initialization phase of the application.
This metadata is needed to understand the rest of the trace data, so if tracing is started after the application started executing, then the trace data might be unusable.
For more information, refer to the [design document](./doc/design_ros_2.md#general-guidelines).
The [launch file action](#launch-file-trace-action) is designed to automatically start tracing before the application launches.

The tracing directory can be configured using command/launch action parameters, or through environment variables with the following logic:

* Use `$ROS_TRACE_DIR` if `ROS_TRACE_DIR` is set and not empty.
* Otherwise, use `$ROS_HOME/tracing`, using `~/.ros` for `ROS_HOME` if not set or if empty.

Additionally, **if you're using kernel tracing with a non-root user, make sure that the [`tracing` group exists and that your user is added to it](https://lttng.org/docs/v2.13/#doc-tracing-group)**.

```
# Create group if it doesn't exist
$ sudo groupadd -r tracing
# Add user to the group
$ sudo usermod -aG tracing $USER
```

### Trace command

The first option is to use the `ros2 trace` command.

```
$ ros2 trace
```

By default, it will enable all ROS 2 tracepoints.
The trace will be written to `~/.ros/tracing/session-YYYYMMDDHHMMSS`.
Run the command with `-h` for more information.

The `ros2 trace` command requires user interaction to start and then stop tracing.
To trace without user interaction (e.g., in scripts), or for finer-grained tracing control, the following sub-commands can be used:

```
$ ros2 trace start session_name   # Configure tracing session and start tracing
$ ros2 trace pause session_name   # Pause tracing after starting
$ ros2 trace resume session_name  # Resume tracing after pausing
$ ros2 trace stop session_name    # Stop tracing after starting or resuming
```

Run each command with `-h` for more information.

You must [install the kernel tracer](#building) if you want to enable kernel events (using the `-k`/`--kernel-events` option).
If you have installed the kernel tracer, use kernel tracing, and still encounter an error here, make sure to [add your user to the `tracing` group](#tracing).

### Launch file trace action

Another option is to use the `Trace` action in a Python, XML, or YAML launch file along with your `Node` action(s).
This way, tracing automatically starts when launching the launch file and ends when it exits or when terminated.

```
$ ros2 launch tracetools_launch example.launch.py
```

The `Trace` action will also set the `LD_PRELOAD` environment to preload [LTTng's userspace tracing helper(s)](https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers) if the corresponding event(s) are enabled.
For more information, see [this example launch file](./tracetools_launch/launch/example.launch.py) and the [`Trace` action](./tracetools_launch/tracetools_launch/action.py).

You must [install the kernel tracer](#building) if you want to enable kernel events (`events_kernel` in Python, `events-kernel` in XML or YAML).
If you have installed the kernel tracer, use kernel tracing, and still encounter an error here, make sure to [add your user to the `tracing` group](#tracing).

## Design

See the [design document](./doc/design_ros_2.md).

## Real-time

LTTng-UST, the current default userspace tracer used for [tracing ROS 2](#overview), was designed for real-time production applications.
It is a low-overhead tracer with many important real-time compatible features:

* userspace tracer completely implemented in userspace, independent from the kernel
* reentrant, thread-safe, signal-safe, non-blocking
* no system calls in the fast path
* no copies of the trace data

However, some settings need to be tuned for it to be fully real-time safe and for performance to be optimal for your use-case:

* timers[^rt-1]: use [read timer](https://lttng.org/docs/v2.13/#doc-channel-read-timer) to avoid a write(2) call
* sub-buffer[^rt-1] count and size:
    * see [documentation](https://lttng.org/docs/v2.13/#doc-channel-subbuf-size-vs-subbuf-count) for sub-buffer count and size tuning tips based on your use-case
    * minimize sub-buffer count to minimize sub-buffer switching overhead
* one-time memory allocation/lock/syscall per thread:
    * usually done the first time a tracepoint is executed within a thread for URCU thread registration, but registration can be manually performed to force it to be done during your application's initialization
    * see [this LTTng mailing list message](https://lists.lttng.org/pipermail/lttng-dev/2019-November/029409.html)

[^rt-1]: this setting cannot currently be set through the [`Trace` launch file action](#launch-file-trace-action) or the [`ros2 trace` command](#trace-command), see [#20](https://github.com/ros2/ros2_tracing/issues/20)

For further reading:

* [LTTng documentation](https://lttng.org/docs/v2.13/)
* [Combined Tracing of the Kernel and Applications with LTTng](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.641.1965&rep=rep1&type=pdf#page=87): LTTng-UST architecture and design goals (section 3)
* [Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead](https://dl.acm.org/doi/abs/10.1145/3158644): LTTng-UST overhead and design compared to other kernel and userspace tracers (table 6: average latency overhead per tracepoint of 158 ns)

The LTTng kernel tracer has a similar implementation, but is separate from the userspace tracer.

## Packages

### lttngpy

Package containing `liblttng-ctl` Python bindings.

### ros2trace

Package containing a `ros2cli` extension to enable tracing.

### tracetools

Library to support instrumenting ROS packages, including core packages.

This package claims to be in the **Quality Level 1** category, see the [Quality Declaration](./tracetools/QUALITY_DECLARATION.md) for more details.

See the [API documentation](https://docs.ros.org/en/rolling/p/tracetools/).

### tracetools_launch

Package containing tools to enable tracing through launch files.

### tracetools_read

Package containing tools to read traces.

### tracetools_test

Package containing tools for tracing-related tests.

### tracetools_trace

Package containing tools to enable tracing.

### test_ros2trace

Package containing system tests for `ros2trace`.

### test_tracetools

Package containing unit and system tests for `tracetools`.

### test_tracetools_launch

Package containing system tests for `tracetools_launch`.

## Analysis

See [`tracetools_analysis`](https://github.com/ros-tracing/tracetools_analysis).