<div align='center'>
<h1> MiV-OS: Spike Analysis and Computing Framework </h1>

[![License][badge-LICENSE]][link-LICENSE]
[![Release pypi][badge-pypi]][link-pypi]
[![Build Status][badge-CI]][link-CI]
[![Documentation Status][badge-docs-status]][link-docs-status]
[![Downloads][badge-pepy-download-count]][link-pepy-download-count]
[![codecov][badge-codecov]][link-codecov]

</div>

---

Python analysis and computing framework developed for [Mind-in-Vitro(MiV)][link-project-website] project.

## Installation
[![PyPI version][badge-pypi]][link-pypi]

MiV-OS is compatible with Python 3.8+. The easiest way to install is using python installation package (PIP)

~~~bash
$ pip install MiV-OS
~~~

## Documentation
[![Documentation Status][badge-docs-status]][link-docs-status]

Documentation of the package is available [here][link-docs-status]

## Contribution

If you would like to participate, please read our [contribution guideline](CONTRIBUTING.md)

The development of MiV-OS is lead by the [Gazzola Lab][link-lab-website] at the University of Illinois at Urbana-Champaign.

## List of publications and submissions

## Citation

```
@misc{MiV-OS,
  author = {Gazzola Lab},
  title = {MiV-OS: Analysis and Computation Framework on MiV System and Simulator},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/GazzolaLab/MiV-OS}},
}
```

We ask that any publications which use MiV-OS package to cite the following papers:

```
```

## Developers ✨
_Names arranged alphabetically_
- Arman Tekinalp
- Andrew Dou
- [Frithjof Gressmann](https://github.com/frthjf)
- Gaurav Upadhyay
- [Seung Hyun Kim](https://github.com/skim0119)

[//]: # (Collection of URLs.)

[link-lab-website]: http://mattia-lab.com/
[link-project-website]: https://mindinvitro.illinois.edu/
[link-docs-status]: https://miv-os.readthedocs.io/en/latest/?badge=latest
[link-CI]: https://github.com/GazzolaLab/MiV-OS/actions
[link-LICENSE]: https://opensource.org/licenses/MIT
[link-pypi]: https://badge.fury.io/py/MiV-OS
[link-pepy-download-count]: https://pepy.tech/project/MiV-OS
[link-codecov]: https://codecov.io/gh/GazzolaLab/MiV-OS

[//]: # (Collection of Badges)

[badge-docs-status]: https://readthedocs.org/projects/miv-os/badge/?version=latest
[badge-CI]: https://github.com/GazzolaLab/MiV-OS/workflows/CI/badge.svg
[badge-LICENSE]: https://img.shields.io/badge/License-MIT-yellow.svg
[badge-pypi]: https://badge.fury.io/py/MiV-OS.svg
[badge-pepy-download-count]: https://static.pepy.tech/badge/MiV-OS
[badge-codecov]: https://codecov.io/gh/GazzolaLab/MiV-OS/branch/main/graph/badge.svg?token=OM5LYWF5KP


    data load tool (dlt) — the open-source Python library for data loading

Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or a GPT-4 assisted development playground—dlt can be dropped in anywhere.

🚀 Join our thriving community of likeminded developers and build the future together!

## Installation

dlt supports Python 3.8+.

**pip:**
```sh
pip install dlt
```

**pixi:**
```sh
pixi add dlt
```

**conda:**
```sh
conda install -c conda-forge dlt
```

## Quick Start

Load chess game data from chess.com API and save it in DuckDB:

```python
import dlt
from dlt.sources.helpers import requests

# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
    pipeline_name='chess_pipeline',
    destination='duckdb',
    dataset_name='player_data'
)

# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
    response = requests.get(f'https://api.chess.com/pub/player/{player}')
    response.raise_for_status()
    data.append(response.json())

# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
```

Try it out in our **[Colab Demo](https://colab.research.google.com/drive/1NfSB1DpwbbHX9_t5vlalBTf13utwpMGx?usp=sharing)**

## Features

- **Automatic Schema:** Data structure inspection and schema creation for the destination.
- **Data Normalization:** Consistent and verified data before loading.
- **Seamless Integration:** Colab, AWS Lambda, Airflow, and local environments.
- **Scalable:** Adapts to growing data needs in production.
- **Easy Maintenance:** Clear data pipeline structure for updates.
- **Rapid Exploration:** Quickly explore and gain insights from new data sources.
- **Versatile Usage:** Suitable for ad-hoc exploration to advanced loading infrastructures.
- **Start in Seconds with CLI:** Powerful CLI for managing, deploying and inspecting local pipelines.
- **Incremental Loading:** Load only new or changed data and avoid loading old records again.
- **Open Source:** Free and Apache 2.0 Licensed.

## Ready to use Sources and Destinations

Explore ready to use sources (e.g. Google Sheets) in the [Verified Sources docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and supported destinations (e.g. DuckDB) in the [Destinations docs](https://dlthub.com/docs/dlt-ecosystem/destinations).

## Documentation

For detailed usage and configuration, please refer to the [official documentation](https://dlthub.com/docs).

## Examples

You can find examples for various use cases in the [examples](docs/examples) folder.

## Adding as dependency

`dlt` follows the semantic versioning with the [`MAJOR.MINOR.PATCH`](https://peps.python.org/pep-0440/#semantic-versioning) pattern. Currently, we are using **pre-release versioning** with the major version being 0.

- `minor` version change means breaking changes
- `patch` version change means new features that should be backward compatible
- any suffix change, e.g., `post10` -> `post11`, is considered a patch

We suggest that you allow only `patch` level updates automatically:
* Using the [Compatible Release Specifier](https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release). For example **dlt~=0.3.10** allows only versions **>=0.3.10** and less than **<0.4**
* Poetry [caret requirements](https://python-poetry.org/docs/dependency-specification/). For example **^0.3.10** allows only versions **>=0.3.10** to **<0.4**
## Get Involved

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

- **Connect with the Community**: Join other dlt users and contributors on our [Slack](https://dlthub.com/community)
- **Report issues and suggest features**: Please use the [GitHub Issues](https://github.com/dlt-hub/dlt/issues) to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
- **Track progress of our work and our plans**: Please check out our [public Github project](https://github.com/orgs/dlt-hub/projects/9)
- **Contribute Verified Sources**: Contribute your custom sources to the [dlt-hub/verified-sources](https://github.com/dlt-hub/verified-sources) to help other folks in handling their data tasks.
- **Contribute code**: Check out our [contributing guidelines](CONTRIBUTING.md) for information on how to make a pull request.
- **Improve documentation**: Help us enhance the dlt documentation.

## License

`dlt` is released under the [Apache 2.0 License](LICENSE.txt).


===
Neo
===

Neo is a Python package for working with electrophysiology data in Python, together
with support for reading a wide range of neurophysiology file formats, including
Spike2, NeuroExplorer, AlphaOmega, Axon, Blackrock, Plexon, Tdt, and support for
writing to a subset of these formats plus non-proprietary formats including HDF5.

The goal of Neo is to improve interoperability between Python tools for
analyzing, visualizing and generating electrophysiology data by providing a common,
shared object model. In order to be as lightweight a dependency as possible,
Neo is deliberately limited to represention of data, with no functions for data
analysis or visualization.

Neo is used by a number of other software tools, including
SpykeViewer_ (data analysis and visualization), Elephant_ (data analysis),
the G-node_ suite (databasing), PyNN_ (simulations), tridesclous_ (spike sorting)
and ephyviewer_ (data visualization).
OpenElectrophy_ (data analysis and visualization) uses an older version of neo.

Neo implements a hierarchical data model well adapted to intracellular and
extracellular electrophysiology and EEG data with support for multi-electrodes
(for example tetrodes). Neo's data objects build on the quantities package,
which in turn builds on NumPy by adding support for physical dimensions. Thus
Neo objects behave just like normal NumPy arrays, but with additional metadata,
checks for dimensional consistency and automatic unit conversion.

A project with similar aims but for neuroimaging file formats is `NiBabel`_.

Code status
-----------

.. image:: https://github.com/NeuralEnsemble/python-neo/actions/workflows/core-test.yml/badge.svg?event=push&branch=master
   :target: https://github.com/NeuralEnsemble/python-neo/actions?query=event%3Apush+branch%3Amaster
   :alt: Core Test Status (Github Actions)
.. image:: https://github.com/NeuralEnsemble/python-neo/actions/workflows/io-test.yml/badge.svg?event=push&branch=master
   :target: https://github.com/NeuralEnsemble/python-neo/actions?query=event%3Apush+branch%3Amaster
   :alt: IO Test Status (Github Actions)
.. image:: https://coveralls.io/repos/NeuralEnsemble/python-neo/badge.png
   :target: https://coveralls.io/r/NeuralEnsemble/python-neo
   :alt: Unit Test Coverage

More information
----------------

- Home page: http://neuralensemble.org/neo
- Mailing list: http://neuralensemble.org/community
- Documentation: http://neo.readthedocs.io/
- Bug reports: https://github.com/NeuralEnsemble/python-neo/issues

For installation instructions, see doc/source/install.rst

To cite Neo in publications, see CITATION.txt

:copyright: Copyright 2010-2024 by the Neo team, see doc/source/authors.rst.
:license: 3-Clause Revised BSD License, see LICENSE.txt for details.

Funding
-------

Development of Neo has been partially funded by the European Union Sixth Framework Program (FP6) under
grant agreement FETPI-015879 (FACETS), by the European Union Seventh Framework Program (FP7/2007-2013)
under grant agreements no. 269921 (BrainScaleS) and no. 604102 (HBP),
and by the European Union’s Horizon 2020 Framework Programme for
Research and Innovation under the Specific Grant Agreements No. 720270 (Human Brain Project SGA1),
No. 785907 (Human Brain Project SGA2) and No. 945539 (Human Brain Project SGA3).

.. _OpenElectrophy: https://github.com/OpenElectrophy/OpenElectrophy
.. _Elephant: http://neuralensemble.org/elephant
.. _G-node: http://www.g-node.org/
.. _Neuroshare: http://neuroshare.org/
.. _SpykeViewer: https://spyke-viewer.readthedocs.org/en/latest/
.. _NiBabel: http://nipy.sourceforge.net/nibabel/
.. _PyNN: http://neuralensemble.org/PyNN
.. _quantities: http://pypi.python.org/pypi/quantities
.. _`NeuralEnsemble mailing list`: http://neuralensemble.org/community
.. _`issue tracker`: https://github.c
.. _tridesclous: https://github.com/tridesclous/tridesclous
.. _ephyviewer: https://github.com/NeuralEnsemble/ephyviewer


# cellxgene curation tools

[![codecov](https://codecov.io/gh/chanzuckerberg/single-cell-curation/branch/main/graph/badge.svg?token=J8OT7OXKHJ)](https://codecov.io/gh/chanzuckerberg/single-cell-curation)

This repository contains documents and code used by cellxgene's curation team. Issues/suggestions pertaining to datasets and how they interact with cellxgene should be created here.

For information/issues about cellxgene and its portal please refer to:

- [single-cell-explorer](https://github.com/chanzuckerberg/single-cell-explorer)
- [single-cell-data-portal](https://github.com/chanzuckerberg/single-cell-data-portal)

## Installation

The primary curation tool is the `cellxgene-schema` CLI. It enables curators to perform [schema](./schema/3.0.0/schema.md) validation for datasets to be hosted on the [cellxgene Data Portal](https://cellxgene.cziscience.com/).

It requires Python >= 3.8. It is available through pip:

```
pip install cellxgene-schema
```

It can also be installed from the source by cloning this repository and running:

```
make install
```

And you can run the tests with:

```
make unit-test
```

## Usage

The CLI validates an [AnnData file](https://anndata.readthedocs.io/en/latest/) (\*.h5ad) to ensure that it addresses the schema requirements.

Datasets can be validated using the following command line:

```
cellxgene-schema validate input.h5ad
```

If the validation succeeds, the command returns a zero exit code; otherwise, it returns a non-zero exit code and prints validation failure messages.

---

The data portal runs the following in the backend:

```
cellxgene-schema validate --add-labels output.h5ad input.h5ad
```

This execution validates the dataset as above AND adds the human-readable labels for the ontology and gene IDs as defined in the schema. If the validation is successful, a new AnnData file (output.h5ad) is written to disk with the labels appended.

This option SHOULD NOT be used by data contributors.

## Contributing

Please read our contributing [guidelines](CONTRIBUTING.md) and make sure adhere to the Contributor Covenant [code of conduct](https://github.com/chanzuckerberg/.github/blob/master/CODE_OF_CONDUCT.md).

## Reporting Security Issues

Please read our [security reporting policy](SECURITY.md)


# Hedera

Hedera is an application for viewing texts in different languages such that:
- pedagogical aids such as glosses can assist the reader;
- the reader can track personal vocabulary lists;
- the reader can refer to shared vocabulary lists;
- texts can be organized into classes;
- and a reader can see how much of a text should already be familiar.

## Getting Started

### With Docker Compose
You will need to have docker installed, use the following link to [install docker](https://docs.docker.com/get-docker/) into your respective OS

To run the app:

```
docker compose up
```

When are you are finished, simply hit `ctrl-c` and the containers will shutdown.

On initial running, you might want do run the following in another terminal:

```
docker compose run --rm django python manage.py create_cms
```
#### Importing Data
To import data into the database:

```
docker compose run --rm django python manage.py shell -c "import loaders.lat.load_latin_lemma"
docker compose run --rm django python manage.py shell -c "import loaders.lat.load_latin_form_to_lemma"
docker compose run --rm django python manage.py shell -c "import loaders.lat.load_latin_gloss"
docker compose run --rm django python manage.py shell -c "import loaders.lat.load_latin_vocab"
```

<!-- TODO UPDATE -->
(Optional) Chinese lattice data:

**Caution**: _This can take more than one hour._

```sh
docker compose run --rm django python manage.py shell -c "import loaders.zho.load_chinese_lattice"
```

#### **Good to know's and Gotcha's**

Steps to add a new python package:
Prequisite - Install pip-compile into your virtual env via this command `python -m pip install pip-tools`. [Documentation here](https://github.com/jazzband/pip-tools)
1. Add package to `hedera/requirements/base.in`
2. Note that the django container must be running before executing the next command.
3. Run:
    ```
    pip-compile hedera/requirements/base.in --output-file=- > hedera/requirements/base.txt
    docker compose exec django pip install <package>
    docker compose exec worker pip install <package>
    ```
    Note: Alternatively we could also rebuild the images for the `django` and `worker` services after running `pip-compile hedera/requirements/base.in --output-file=- > hedera/requirements/base.txt` via these commands:
    ```sh
    docker compose build django
    docker compose build worker
    ```

If you'd like to remove all existing images and start fresh:

```sh
docker compose down --rmi all
docker compose build
docker compose up
```

Note: This will not remove the data in the database, to keep from having to re-load the language data. If you want to remove the database, you can remove `./.docker-data` in this directory. This is specified in the `docker-compose.yml`, s if that doesn't work you can check if the configuration has changed there.

If you've run a lot of containers and you'd like to clean them up so that your `docker compose` command output isn't endless:

```
docker ps --filter status=exited --filter name=hedera_django_run -q | xargs docker rm
```

##### Font Awesome Kit
Notes: base.html is using a js file provided by Font Awesome Kit `https://kit.fontawesome.com/2db98e34e3.js`. This means that icons like the familarity and edit buttons will stop working if the link expires. Link is created from [Font Awesome Kit Setup](https://fontawesome.com/docs/web/setup/use-kit). We could download the js file and run it locally as well, but will hold off for now.

## PDF Service

To enable the PDF export of a learner's text with gloss you need to run the PDF
service that is packaged up as a docker container.

For local development:

```
docker run --publish 9000:9000 --env X_API_KEY=test-secret eldarioninc/pdf-service
```

## Contributing Guidelines

### Tests

#### Frontend / Javascript

There are some tests for the Vue frontend code:

```
npm run test:unit
```

#### Backend / Python

Unit tests depend on:
* a connection to a postgres DB; this is due to limitations in the `django-lti-provider` package; see `test_settings.py`), and
* a connection to redis (`django-rq` requires a redis connection, with no simply way to mock it for tests).

Here's one way to run them, using the containerized app, which is already configured to look for the required support services on the docker container network:

```
docker compose up -d postgres redis
# make sure postgres and redis are available, then run:
docker compose run --rm --no-deps --service-ports -e DJANGO_SETTINGS_MODULE=hedera.test_settings django python manage.py test
```

You can also run them locally (i.e. not in a container) if you have the appropriate python environment configured. For instance:

```
docker compose up -d postgres redis
# make sure postgres and redis are available, then run:
DJANGO_SETTINGS_MODULE=hedera.test_settings python manage.py test
```
##### Note:
    To log out entire diff in a test set self.maxDiff = None
    https://docs.python.org/3/library/unittest.html#unittest.TestCase.maxDiff

### Linting

This project uses `isort` for import sorting, `flake8` for Python linting, and various `eslint plugins` for Javascript linting.

### Passing Builds via Github

The GitHub Actions workflow .github/workflows/ci.yaml runs the unit tests and various code quality checks.

#### Pre-commit

To make development more streamlined we have implemented a pre-commit package manager to manage pre-commit hooks for python and javascript.
- Install [Pre-commit](https://pre-commit.com/)
Note: the installation guide covers different install methods
- you will then need to run the following command in the root of the project
```
pre-commit install
```
You're all set! Now when you commit code upstream the commit will trigger the different linters and flag the required changes
Notes: you can quickly fix javascript eslint errors with `npm run lint:fix`

## Resetting the Database

Every so often, it may be requested that the dev DB be "reset." Meaning, tables from the lemma, lemmatized_text, and vocab_list be truncated, and the data for lemmas will be reloaded. User account information should be preserved. You can do that with the following set of commands. Note that these commands are for executing against a running docker container. If you are running the project in another way, adjust the incantations accordingly.

```sh
# Run the custom management command to delete all of the relevant data.
# The --no_input flag answers 'yes' to all prompts asking if you'd like to delete data.
# Run without --no_input if you'd like to step through the process interatctively.
$ docker compose exec django python manage.py reset_db --no_input
```
Run commands from the [importing data step](#importing-data) to reload the data

## Adding a new language

In order to add a new language, you will need to add an entry to `SUPPORTED_LANGUAGES` in `hedera.supported_languages.py`. Every language requires a "code", "verbose_name", "service", and "tokenizer." Language specific services are defined in `lemmatization.services`. Please, review the existing services and tokenizers, and add your language's module.

## Debugging Programmatically

To debug from within vscode you can enter this line into the file/endpoint you want to debug. [Reference to debugger](https://code.visualstudio.com/docs/editor/debugging) and [debugpy vscode reference](https://code.visualstudio.com/docs/python/debugging)
```python
        import debugpy

        try:
            debugpy.listen(("0.0.0.0", 5678))
            print("Waiting for debugger attach")
            debugpy.wait_for_client()
            debugpy.breakpoint()
            print('break on this line')
        except:
            pass

```


# plateau

_flat files, flat land_

[![Build Status](https://github.com/data-engineering-collective/plateau/workflows/CI/badge.svg)](https://github.com/data-engineering-collective/plateau/actions?query=branch%3Amain)
[![Documentation Status](https://readthedocs.org/projects/plateau/badge/?version=stable)](https://plateau.readthedocs.io/en/stable/?badge=stable)
[![codecov.io](https://codecov.io/github/data-engineering-collective/plateau/coverage.svg?branch=master)](https://codecov.io/github/data-engineering-collective/plateau)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/data-engineering-collective/plateau/blob/master/LICENSE.txt)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/plateau/badges/installer/conda.svg)](https://conda.anaconda.org/conda-forge)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/plateau/badges/downloads.svg)](https://anaconda.org/conda-forge/plateau)

`plateau` is a Python library to manage (create, read, update, delete) large
amounts of tabular data in a blob store. It stores data as datasets, which
it presents as pandas DataFrames to the user. Datasets are a collection of
files with the same schema that reside in a blob store. plateau uses a metadata
definition to handle these datasets efficiently. For distributed access and
manipulation of datasets plateau offers a [Dask](https://dask.org) interface.

Storing data distributed over multiple files in a blob store (S3, ABS, GCS,
etc.) allows for a fast, cost-efficient and highly scalable data infrastructure.
A downside of storing data solely in an object store is that the storages
themselves give little to no guarantees beyond the consistency of a single file.
In particular, they cannot guarantee the consistency of your dataset. If we
demand a consistent state of our dataset at all times, we need to track the
state of the dataset. plateau frees us from having to do this manually.

The `plateau.io` module provides building blocks to create and modify these
datasets in data pipelines. plateau handles I/O, tracks dataset partitions
and selects subsets of data transparently.

## Installation

Installers for the latest released version are availabe at the [Python
package index](https://pypi.org/project/plateau) and on conda-forge.

```sh
# Install with pip
pip install plateau
```

```sh
# Install with conda/micromamba, optionally add conda-forge as a source
# conda config --add channels conda-forge
conda install plateau
micromamba install plateau
```


[![PyPI](https://img.shields.io/pypi/v/modelscope)](https://pypi.org/project/modelscope/)

[![license](https://img.shields.io/github/license/modelscope/modelscope.svg)](https://github.com/modelscope/modelscope/blob/master/LICENSE)
[![open issues](https://isitmaintained.com/badge/open/modelscope/modelscope.svg)](https://github.com/modelscope/modelscope/issues)
[![GitHub pull-requests](https://img.shields.io/github/issues-pr/modelscope/modelscope.svg)](https://GitHub.com/modelscope/modelscope/pull/)
[![GitHub latest commit](https://badgen.net/github/last-commit/modelscope/modelscope)](https://GitHub.com/modelscope/modelscope/commit/)
[![Leaderboard](https://img.shields.io/badge/ModelScope-Check%20Your%20Contribution-orange)](https://opensource.alibaba.com/contribution_leaderboard/details?projectValue=modelscope)

        English |
        中文 |
        日本語

# Introduction

[ModelScope]( https://www.modelscope.cn) is built upon the notion of “Model-as-a-Service” (MaaS). It seeks to bring together most advanced machine learning models from the AI community, and streamlines the process of leveraging AI models in real-world applications. The core ModelScope library open-sourced in this repository provides the interfaces and implementations that allow developers to perform  model inference, training and evaluation.

In particular, with rich layers of API-abstraction, the ModelScope library offers unified experience to explore state-of-the-art models spanning across domains such as CV, NLP, Speech, Multi-Modality, and Scientific-computation. Model contributors of different areas can integrate models into the ModelScope ecosystem through the layered-APIs, allowing easy and unified access to their models. Once integrated, model inference, fine-tuning, and evaluations can be done with only a few lines of codes. In the meantime, flexibilities are also provided so that different components in the model applications can be customized wherever necessary.

Apart from harboring implementations of a wide range of different models, ModelScope library also enables the necessary interactions with ModelScope backend services, particularly with the Model-Hub and Dataset-Hub. Such interactions facilitate management of  various entities (models and datasets) to be performed seamlessly under-the-hood, including entity lookup, version control, cache management, and many others.

# Models and Online Accessibility

Hundreds of models are made publicly available on [ModelScope]( https://www.modelscope.cn)  (700+ and counting), covering the latest development in areas such as NLP, CV, Audio, Multi-modality, and AI for Science, etc. Many of these models represent the SOTA in their specific fields, and made their open-sourced debut on ModelScope. Users can visit ModelScope([modelscope.cn](http://www.modelscope.cn)) and experience first-hand how these models perform via online experience, with just a few clicks. Immediate developer-experience is also possible through the ModelScope Notebook, which is backed by ready-to-use CPU/GPU development environment in the cloud - only one click away on [ModelScope](https://www.modelscope.cn).

Some representative examples include:

LLM:

* [Yi-1.5-34B-Chat](https://modelscope.cn/models/01ai/Yi-1.5-34B-Chat/summary)

* [Qwen1.5-110B-Chat](https://modelscope.cn/models/qwen/Qwen1.5-110B-Chat/summary)

* [DeepSeek-V2-Chat](https://modelscope.cn/models/deepseek-ai/DeepSeek-V2-Chat/summary)

* [Ziya2-13B-Chat](https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Chat/summary)

* [Meta-Llama-3-8B-Instruct](https://modelscope.cn/models/LLM-Research/Meta-Llama-3-8B-Instruct/summary)

* [Phi-3-mini-128k-instruct](https://modelscope.cn/models/LLM-Research/Phi-3-mini-128k-instruct/summary)

Multi-Modal:

* [Qwen-VL-Chat](https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary)

* [Yi-VL-6B](https://modelscope.cn/models/01ai/Yi-VL-6B/summary)

* [InternVL-Chat-V1-5](https://modelscope.cn/models/AI-ModelScope/InternVL-Chat-V1-5/summary)

* [deepseek-vl-7b-chat](https://modelscope.cn/models/deepseek-ai/deepseek-vl-7b-chat/summary)

* [OpenSoraPlan](https://modelscope.cn/models/AI-ModelScope/Open-Sora-Plan-v1.0.0/summary)

* [OpenSora](https://modelscope.cn/models/luchentech/OpenSora-STDiT-v1-HQ-16x512x512/summary)

* [I2VGen-XL](https://modelscope.cn/models/iic/i2vgen-xl/summary)

CV:

* [DamoFD Face Detection Key Point Model - 0.5G](https://modelscope.cn/models/damo/cv_ddsar_face-detection_iclr23-damofd/summary)

* [BSHM Portrait Matting](https://modelscope.cn/models/damo/cv_unet_image-matting/summary)

* [DCT-Net Portrait Cartoonization - 3D](https://modelscope.cn/models/damo/cv_unet_person-image-cartoon-3d_compound-models/summary)

* [DCT-Net Portrait Cartoonization Model - 3D](https://modelscope.cn/models/damo/face_chain_control_model/summary)

* [DuGuang - Text Recognition - Line Recognition Model - Chinese and English - General Domain](https://modelscope.cn/models/damo/cv_convnextTiny_ocr-recognition-general_damo/summary)

* [DuGuang - Text Recognition - Line Recognition Model - Chinese and English - General Domain](https://modelscope.cn/models/damo/cv_resnet18_ocr-detection-line-level_damo/summary)

* [LaMa Image Inpainting](https://modelscope.cn/models/damo/cv_fft_inpainting_lama/summary)

Audio:

* [Paraformer Speech Recognition - Chinese - General - 16k - Offline - Large - Long Audio Version](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)

* [FSMN Voice Endpoint Detection - Chinese - General - 16k - onnx](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-onnx/summary)

* [Monotonic-Aligner Speech Timestamp Prediction - 16k - Offline](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary)

* [CT-Transformer Punctuation - Chinese - General - onnx](https://modelscope.cn/models/damo/punc_ct-transformer_zh-cn-common-vocab272727-onnx/summary)

* [Speech Synthesis - Chinese - Multiple Emotions Domain - 16k - Multiple Speakers](https://modelscope.cn/models/damo/speech_sambert-hifigan_tts_zh-cn_16k/summary)

* [CAM++ Speaker Verification - Chinese - General - 200k-Spkrs](https://modelscope.cn/models/damo/speech_campplus_sv_zh-cn_16k-common/summary)

AI for Science:

* [uni-fold-monomer](https://modelscope.cn/models/DPTech/uni-fold-monomer/summary)

* [uni-fold-multimer](https://modelscope.cn/models/DPTech/uni-fold-multimer/summary)

**Note:** Most models on ModelScope are public and can be downloaded without account registration on modelscope website([www.modelscope.cn](www.modelscope.cn)), please refer to instructions for [model download](https://modelscope.cn/docs/%E6%A8%A1%E5%9E%8B%E7%9A%84%E4%B8%8B%E8%BD%BD), for dowloading models with api provided by modelscope library or git.

# QuickTour

We provide unified interface for inference using `pipeline`, fine-tuning and evaluation using `Trainer` for different tasks.

For any given task with any type of input (image, text, audio, video...), inference pipeline can be implemented with only a few lines of code, which will automatically load the underlying model to get inference result, as is exemplified below:

```python
>>> from modelscope.pipelines import pipeline
>>> word_segmentation = pipeline('word-segmentation',model='damo/nlp_structbert_word-segmentation_chinese-base')
>>> word_segmentation('今天天气不错，适合出去游玩')
{'output': '今天 天气 不错 ， 适合 出去 游玩'}
```

Given an image, portrait matting (aka. background-removal) can be accomplished with the following code snippet:

![image](data/resource/portrait_input.png)

```python
>>> import cv2
>>> from modelscope.pipelines import pipeline

>>> portrait_matting = pipeline('portrait-matting')
>>> result = portrait_matting('https://modelscope.oss-cn-beijing.aliyuncs.com/test/images/image_matting.png')
>>> cv2.imwrite('result.png', result['output_img'])
```

The output image with the background removed is:
![image](data/resource/portrait_output.png)

Fine-tuning and evaluation can also be done with a few more lines of code to set up training dataset and trainer, with the heavy-lifting work of training and evaluation a model encapsulated in the implementation of  `traner.train()` and
`trainer.evaluate()`  interfaces.

For example, the gpt3 base model (1.3B) can be fine-tuned with the chinese-poetry dataset, resulting in a model that can be used for chinese-poetry generation.

```python
>>> from modelscope.metainfo import Trainers
>>> from modelscope.msdatasets import MsDataset
>>> from modelscope.trainers import build_trainer

>>> train_dataset = MsDataset.load('chinese-poetry-collection', split='train'). remap_columns({'text1': 'src_txt'})
>>> eval_dataset = MsDataset.load('chinese-poetry-collection', split='test').remap_columns({'text1': 'src_txt'})
>>> max_epochs = 10
>>> tmp_dir = './gpt3_poetry'

>>> kwargs = dict(
     model='damo/nlp_gpt3_text-generation_1.3B',
     train_dataset=train_dataset,
     eval_dataset=eval_dataset,
     max_epochs=max_epochs,
     work_dir=tmp_dir)

>>> trainer = build_trainer(name=Trainers.gpt3_trainer, default_args=kwargs)
>>> trainer.train()
```

# Why should I use ModelScope library

1. A unified and concise user interface is abstracted for different tasks and different models. Model inferences and training can be implemented by as few as 3 and 10 lines of code, respectively. It is convenient for users to explore models in different fields in the ModelScope community. All models integrated into ModelScope are ready to use, which makes it easy to get started with AI, in both educational and industrial settings.

2. ModelScope offers a model-centric development and application experience. It streamlines the support for model training, inference, export and deployment, and facilitates users to build their own MLOps based on the ModelScope ecosystem.

3. For the model inference and training process, a modular design is put in place, and a wealth of functional module implementations are provided, which is convenient for users to customize their own model inference, training and other processes.

4. For distributed model training, especially for large models, it provides rich training strategy support, including data parallel, model parallel, hybrid parallel and so on.

# Installation

## Docker

ModelScope Library currently supports popular deep learning framework for model training and inference, including PyTorch, TensorFlow and ONNX. All releases are tested and run on Python 3.7+, Pytorch 1.8+, Tensorflow1.15 or Tensorflow2.0+.

To allow out-of-box usage for all the models on ModelScope, official docker images are provided for all releases. Based on the docker image, developers can skip all environment installation and configuration and use it directly. Currently, the latest version of the CPU image and GPU image can be obtained from:

CPU docker image
```shell
# py37
registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py37-torch1.11.0-tf1.15.5-1.6.1

# py38
registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch2.0.1-tf2.13.0-1.9.5
```

GPU docker image
```shell
# py37
registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.3.0-py37-torch1.11.0-tf1.15.5-1.6.1

# py38
registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-cuda11.8.0-py38-torch2.0.1-tf2.13.0-1.9.5
```

## Setup Local Python Environment

One can also set up local ModelScope environment using pip and conda.  ModelScope supports python3.7 and above.
We suggest [anaconda](https://docs.anaconda.com/anaconda/install/) for creating local python environment:

```shell
conda create -n modelscope python=3.8
conda activate modelscope
```

PyTorch or TensorFlow can be installed separately according to each model's requirements.
* Install pytorch [doc](https://pytorch.org/get-started/locally/)
* Install tensorflow [doc](https://www.tensorflow.org/install/pip)

After installing the necessary machine-learning framework, you can install modelscope library as follows:

If you only want to play around with the modelscope framework, of trying out model/dataset download, you can install the core modelscope components:
```shell
pip install modelscope
```

If you want to use multi-modal models:
```shell
pip install modelscope[multi-modal]
```

If you want to use nlp models:
```shell
pip install modelscope[nlp] -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
```

If you want to use cv models:
```shell
pip install modelscope[cv] -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
```

If you want to use audio models:
```shell
pip install modelscope[audio] -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
```

If you want to use science models:
```shell
pip install modelscope[science] -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
```

`Notes`:
1. Currently, some audio-task models only support python3.7, tensorflow1.15.4 Linux environments. Most other models can be installed and used on Windows and Mac (x86).

2. Some models in the audio field use the third-party library SoundFile for wav file processing. On the Linux system, users need to manually install libsndfile of SoundFile([doc link](https://github.com/bastibe/python-soundfile#installation)). On Windows and MacOS, it will be installed automatically without user operation. For example, on Ubuntu, you can use following commands:
    ```shell
    sudo apt-get update
    sudo apt-get install libsndfile1
    ```

3. Some models in computer vision need mmcv-full, you can refer to mmcv [installation guide](https://github.com/open-mmlab/mmcv#installation), a minimal installation is as follows:

    ```shell
    pip uninstall mmcv # if you have installed mmcv, uninstall it
    pip install -U openmim
    mim install mmcv-full
    ```

# Learn More

We  provide additional documentations including:
* [More detailed Installation Guide](https://modelscope.cn/docs/%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85)
* [Introduction to tasks](https://modelscope.cn/docs/%E4%BB%BB%E5%8A%A1%E7%9A%84%E4%BB%8B%E7%BB%8D)
* [Use pipeline for model inference](https://modelscope.cn/docs/%E6%A8%A1%E5%9E%8B%E7%9A%84%E6%8E%A8%E7%90%86Pipeline)
* [Finetuning example](https://modelscope.cn/docs/%E6%A8%A1%E5%9E%8B%E7%9A%84%E8%AE%AD%E7%BB%83Train)
* [Preprocessing of data](https://modelscope.cn/docs/%E6%95%B0%E6%8D%AE%E7%9A%84%E9%A2%84%E5%A4%84%E7%90%86)
* [Evaluation](https://modelscope.cn/docs/%E6%A8%A1%E5%9E%8B%E7%9A%84%E8%AF%84%E4%BC%B0)
* [Contribute your own model to ModelScope](https://modelscope.cn/docs/ModelScope%E6%A8%A1%E5%9E%8B%E6%8E%A5%E5%85%A5%E6%B5%81%E7%A8%8B%E6%A6%82%E8%A7%88)

# License

This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).


# python-rucaptcha

![Logo](https://red-panda-dev.xyz/media/images/RuCaptchaHigh_zkkPoYF.original.png)

### [Capsolver](https://capsolver.com?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha)

[![Capsolver](files/capsolver.jpg)](https://capsolver.com?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha)

[Capsolver.com](https://www.capsolver.com/?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha) is an AI-powered service that specializes in solving various types of captchas automatically. It supports captchas such as [reCAPTCHA V2](https://docs.capsolver.com/guide/captcha/ReCaptchaV2.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [reCAPTCHA V3](https://docs.capsolver.com/guide/captcha/ReCaptchaV3.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [hCaptcha](https://docs.capsolver.com/guide/captcha/HCaptcha.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [FunCaptcha](https://docs.capsolver.com/guide/captcha/FunCaptcha.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [DataDome](https://docs.capsolver.com/guide/captcha/DataDome.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [AWS Captcha](https://docs.capsolver.com/guide/captcha/awsWaf.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [Geetest](https://docs.capsolver.com/guide/captcha/Geetest.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), and Cloudflare [Captcha](https://docs.capsolver.com/guide/antibots/cloudflare_turnstile.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha) / [Challenge 5s](https://docs.capsolver.com/guide/antibots/cloudflare_challenge.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), [Imperva / Incapsula](https://docs.capsolver.com/guide/antibots/imperva.html?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), among others.

For developers, Capsolver offers API integration options detailed in their [documentation](https://docs.capsolver.com/?utm_source=github&utm_medium=banner_github&utm_campaign=python-rucaptcha), facilitating the integration of captcha solving into applications. They also provide browser extensions for [Chrome](https://chromewebstore.google.com/detail/captcha-solver-auto-captc/pgojnojmmhpofjgdmaebadhbocahppod) and [Firefox](https://addons.mozilla.org/es/firefox/addon/capsolver-captcha-solver/), making it easy to use their service directly within a browser. Different pricing packages are available to accommodate varying needs, ensuring flexibility for users.

<hr>

[![PyPI version](https://badge.fury.io/py/python-rucaptcha.svg)](https://badge.fury.io/py/python-rucaptcha)
[![Python versions](https://img.shields.io/pypi/pyversions/python-rucaptcha.svg?logo=python&logoColor=FBE072)](https://badge.fury.io/py/python-rucaptcha)
[![Downloads](https://static.pepy.tech/badge/python-rucaptcha/month)](https://pepy.tech/project/python-rucaptcha)

[![Maintainability](https://api.codeclimate.com/v1/badges/aec93bb04a277cf0dde9/maintainability)](https://codeclimate.com/github/AndreiDrang/python-rucaptcha/maintainability)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/b4087362bd024b088b358b3e10e7a62f)](https://www.codacy.com/gh/AndreiDrang/python-rucaptcha/dashboard?utm_source=github.com&utm_medium=referral&utm_content=AndreiDrang/python-rucaptcha&utm_campaign=Badge_Grade)
[![codecov](https://codecov.io/gh/AndreiDrang/python-rucaptcha/branch/master/graph/badge.svg?token=doybTUCfbD)](https://codecov.io/gh/AndreiDrang/python-rucaptcha)

[![Sphinx docs](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/sphinx.yml/badge.svg?branch=release)](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/sphinx.yml)
[![Build](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/build.yml/badge.svg?branch=master)](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/build.yml)
[![Installation](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/install.yml/badge.svg?branch=master)](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/install.yml)
[![Tests](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/test.yml)
[![Lint](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/lint.yml/badge.svg?branch=master)](https://github.com/AndreiDrang/python-rucaptcha/actions/workflows/lint.yml)

Python3 library for [RuCaptcha](https://rucaptcha.com/) and [2Captcha](https://2captcha.com/) service API.

Tested on UNIX based OS.

The library is intended for software developers and is used to work with the [RuCaptcha](https://rucaptcha.com/) and [2Captcha](https://2captcha.com/) service API.

Support of the service [Death By Captcha](https://deathbycaptcha.com?refid=1237267242) is integrated into this library, more information in the library documentation or in the [service docs](https://deathbycaptcha.com/api/2captcha?refid=1237267242).
***

Application in [RuCaptcha software](https://rucaptcha.com/software/python-rucaptcha) and [2Captcha software](https://2captcha.com/software/python-rucaptcha).

If you have any questions, please send a message to the [Telegram](https://t.me/pythoncaptcha) chat room.

Or email python-captcha@pm.me

## How to install?

### pip

```bash
pip install python-rucaptcha
```

## How to use?

Is described in the [documentation-website](https://andreidrang.github.io/python-rucaptcha/).

## How to test?

1. You need set ``RUCAPTCHA_KEY`` in your environment(get this value from you account).
2. Run command ``make tests``, from root directory.

### Changelog

For full changelog info check - [Releases page](https://github.com/AndreiDrang/python-rucaptcha/releases).

- v.6.0 - Library refactoring. Stop using `pydantic`, start using `msgspec`. Move to API v2. Drop Python 3.8 support. More details at [Releases page](https://github.com/AndreiDrang/python-rucaptcha/releases).
- v.5.3 - Added support for [Death By Captcha](https://www.deathbycaptcha.com?refid=1237267242) and other services by changing `service_type` and `url_request` \ `url_response` parameters.
- v.5.2 - Added Audio captcha method.
- v.5.1 - Check [releases page](https://github.com/AndreiDrang/python-rucaptcha/releases).
- v.5.0 - Added AmazonWAF captcha method.
- v.4.2 - Added [Yandex Smart Captcha](https://rucaptcha.com/api-rucaptcha#yandex).

### Get API Key to work with the library
1. On the page - https://rucaptcha.com/enterpage
2. Find it: ![img.png](https://github.com/AndreiDrang/python-rucaptcha/blob/master/files/img.png)


**A Pure Python, React-style Framework for Scaling Your Jupyter and Web Apps**

[![solara logo](https://solara.dev/static/assets/images/logo.svg)](https://solara.dev)

Come chat with us on [Discord](https://discord.solara.dev) to ask questions or share your thoughts or creations!

[![Discord Shield](https://discordapp.com/api/guilds/1106593685241614489/widget.png?style=banner2)](https://discord.solara.dev)

## Introducing Solara

While there are many Python web frameworks out there, most are designed for small data apps or use paradigms unproven for larger scale. Code organization, reusability, and state tend to suffer as apps grow in complexity, resulting in either spaghetti code or offloading to a React application.

Solara addresses this gap. Using a React-like API, we don't need to worry about scalability. React has already proven its ability to support the world's largest web apps.

Solara uses a pure Python implementation of React (Reacton), creating ipywidget-based applications. These apps work both inside the Jupyter Notebook and as standalone web apps with frameworks like FastAPI. This paradigm enables component-based code and incredibly simple state management.

By building on top of ipywidgets, we automatically leverage an existing ecosystem of widgets and run on many platforms, including JupyterLab, Jupyter Notebook, Voilà, Google Colab, DataBricks, JetBrains Datalore, and more.

We care about developer experience. Solara will give your hot code reloading and type hints for faster development.

## Installation

Run:
```
pip install solara
```

Or follow the [Installation instructions](https://solara.dev/documentation/getting_started/installing) for more detailed instructions.

## First script

Put the following Python snippet in a file (we suggest `sol.py`), or put it in a Jupyter notebook cell:

```python
import solara

# Declare reactive variables at the top level. Components using these variables
# will be re-executed when their values change.
sentence = solara.reactive("Solara makes our team more productive.")
word_limit = solara.reactive(10)

@solara.component
def Page():
    # Calculate word_count within the component to ensure re-execution when reactive variables change.
    word_count = len(sentence.value.split())

    solara.SliderInt("Word limit", value=word_limit, min=2, max=20)
    solara.InputText(label="Your sentence", value=sentence, continuous_update=True)

    # Display messages based on the current word count and word limit.
    if word_count >= int(word_limit.value):
        solara.Error(f"With {word_count} words, you passed the word limit of {word_limit.value}.")
    elif word_count >= int(0.8 * word_limit.value):
        solara.Warning(f"With {word_count} words, you are close to the word limit of {word_limit.value}.")
    else:
        solara.Success("Great short writing!")

# The following line is required only when running the code in a Jupyter notebook:
Page()
```

Run from the command line in the same directory where you put your file (`sol.py`):

```bash
$ solara run sol.py
Solara server is starting at http://localhost:8765
```

Or copy-paste this to a Jupyter notebook cell and execute it (the `Page()` expression at the end
will cause it to automatically render the component in the notebook).

See this snippet run live at https://solara.dev/documentation/getting_started

## Demo

The following demo app can be used to explore a dataset (buildin or upload yourself) using
a scatter plot. The plot can be interacted with to filter the dataset, and the filtered dataset can
be downloaded.

 * [Source code](https://github.com/widgetti/solara/blob/master/solara/website/pages/apps/scatter.py)

### Running in solara-server

The solara server is build on top of Starlette/FastAPI and runs standalone. Ideal for production use.

![fastapi](https://global.discourse-cdn.com/standard11/uploads/jupyter/original/2X/9/9442fc70e2a1fcd201f4f900fa073698a1f8c937.gif)

### Running in Jupyter

By building on top of ipywidgets, we automatically leverage an existing ecosystem of widgets and run on many platforms, including JupyterLab, Jupyter Notebook, Voilà, Google Colab, DataBricks, JetBrains Datalore, and more. This means our app can also run in Jupyter:

![jupyter](https://global.discourse-cdn.com/standard11/uploads/jupyter/original/2X/8/8bc875c0c3845ae077168575a4f8a49cf1b35bc6.gif)

## Resources

Visit our main website or jump directly to the introduction

[![Introduction](https://dabuttonfactory.com/button.png?t=Introduction&f=Open+Sans-Bold&ts=20&tc=fff&hp=45&vp=12&c=8&bgt=unicolored&bgc=f19f41)](https://solara.dev/documentation/getting_started/introduction)
[![Quickstart](https://dabuttonfactory.com/button.png?t=Quickstart&f=Open+Sans-Bold&ts=20&tc=fff&hp=45&vp=12&c=8&bgt=unicolored&bgc=f19f41)](https://solara.dev/documentation/getting_started)

*Note that the solara.dev website is created using Solara*


# HaikuPorter

The HaikuPorter tool is provided to ease the fetching, patching and building of source code. It can be compared to a slim version of [Gentoo Portage](https://www.gentoo.org/main/en/about.xml). Each port contains the [Haiku](http://haiku-os.org) specific patches to the original source code. It fetches the original source code, applies the Haiku-specific patches, builds the software, and packages it.

Detailed information available on the [wiki](https://github.com/haikuports/haikuports/wiki/).

## Quick start

### Single Machine (Haiku)

A single machine installation is for building individual packages.

## Installation (Haiku)

HaikuPorts installation can be done via the following command sequence:

```shell
 $ git clone https://github.com/haikuports/haikuporter.git
 $ git clone https://github.com/haikuports/haikuports.git --depth=10
 $ cd haikuporter
 $ cp haikuports-sample.conf /boot/home/config/settings/haikuports.conf # Copy the config file
 $ lpe ~/config/settings/haikuports.conf # and edit it
```

### Build port
 - `./haikuporter mesa -j4`

### Build port and all outdated dependency ports
 - `./haikuporter mesa --all-dependencies -j4`

## Multi-node cluster (Linux + Haiku)

A multi-node cluster is for mass building large numbers of packages.

### Running buildmaster in a container with docker

 - `docker pull ghcr.io/haikuports/haikuporter/buildmaster`
 - `mkdir ~/buildmaster.x86`
 - `docker run -v ~/buildmaster.x86:/data -it -e ARCH=x86 ghcr.io/haikuports/haikuporter/buildmaster`
 - Provision builders
   - `createbuilder -n test01 -H 127.0.0.1`
   - copy generated public key to builder
   - `builderctl health`
 - exit
 - Copy the packages from a nightly to ports/packages on the buildmaster
 - `docker run -v ~/buildmaster.x86:/data -it -e ARCH=x86 ghcr.io/haikuports/haikuporter/buildmaster`
 - buildmaster everything

buildmaster.x86 will persist between build runs. Feel free to exit, update, or
erase the container without losing your work.

### Manually Deploy buildmaster (Linux)

 - Install requirements
   - `pip install paramiko` or `dnf install python-paramiko`
   - buildtools dependencies: autoconf, flex, bison, texinfo, zlib-devel
   - Haiku host tools dependencies: libstdc++-static, libcurl-devel
 - Bootstrap the buildmaster instance
   - `git clone https://github.com/haikuports/haikuporter.git`
   - `./haikuporter/buildmaster/bin/bootstrap_buildmaster.sh ...`
 - Configure your builders within instance ports tree with createbuilder
   - `cd buildmaster_<arch>/haikuports`
   - example: `../haikuporter/buildmaster/bin/createbuilder -n mybuilder01 -H 127.0.0.1`
 - Validate and provision your builders
   - `../haikuporter/buildmaster/bin/builderctl health`
   - `../haikuporter/buildmaster/bin/builderctl provision`
 - `../haikuporter/buildmaster/bin/buildmaster everything`

### Deploy buildslave (Haiku)

 - Checkout Haikuporter and Haikuports, matching the paths specified in createbuilder on buildmaster side
 - Add the public key from the buildmaster to authorized\_keys
 - useradd sshd ; ssh-keygen -A
 - Enable PermitRootLogin in /system/settings/ssh/sshd\_config and make sure the path to the sftp server is correct
 - install xz\_utils\_x86, lzip\_x86 (required for extracting packages), patch, dos2unix (required for PATCH() function in some packages)


![Maturity level-1](https://img.shields.io/badge/Maturity%20Level-ML--2-green)

[Find our docs here](https://astrazeneca.github.io/KAZU/index.html)

# Kazu - Biomedical NLP Framework

**Note: the recent 2.0 release has large elements of backwards incompatibility if you are using a custom model pack and curations.**

Welcome to Kazu (Korea AstraZeneca University), a python biomedical NLP framework built in collaboration with Korea University,
designed to handle production workloads.

This library aims to simplify the process of using state of the art NLP research in production systems. Some of the
research contained within are our own, but most of it comes from the community, for which we are immensely grateful.

If you want to use Kazu, please cite our [EMNLP 2022 publication](https://aclanthology.org/2022.emnlp-industry.63)!
([**citation link**](https://aclanthology.org/2022.emnlp-industry.63.bib))

[Please click here for the TinyBERN2 training and evaluation code](https://github.com/dmis-lab/KAZU-NER-module)

# Quickstart

## Install

Python version 3.9 or higher is required (tested with Python 3.11).

Either:

`pip install kazu`

or download the wheel from the release page and install locally.

If you intend to use [Mypy](https://mypy.readthedocs.io/en/stable/#) on your own codebase, consider installing Kazu using:

`pip install kazu[typed]`

This will pull in typing stubs for kazu's dependencies (such as [types-requests](https://pypi.org/project/types-requests/) for [Requests](https://requests.readthedocs.io/en/latest/))
so that mypy has access to as much relevant typing information as possible when type checking your codebase. Otherwise (depending on mypy config), you may see errors when running mypy like:

```
.venv/lib/python3.10/site-packages/kazu/steps/linking/post_processing/xref_manager.py:10: error: Library stubs not installed for "requests" [import]
```

## Getting the model pack

For most functionality, you will also need the Kazu model pack. This is tied to each release, and can be found on the [release page](https://github.com/astrazeneca/kazu/releases). Once downloaded,
extract the archive and:

`export KAZU_MODEL_PACK=<path to the extracted archive>`

Kazu is highly configurable (using [Hydra](https://hydra.cc/docs/intro/)), although it comes preconfigured with defaults appropriate for most literature processing use cases.
To make use of these, and process a simple document:

```python
import hydra
from hydra.utils import instantiate

from kazu.data import Document
from kazu.pipeline import Pipeline
from kazu.utils.constants import HYDRA_VERSION_BASE
from pathlib import Path
import os

# the hydra config is kept in the model pack
cdir = Path(os.environ["KAZU_MODEL_PACK"]).joinpath("conf")

@hydra.main(
    version_base=HYDRA_VERSION_BASE, config_path=str(cdir), config_name="config"
)
def kazu_test(cfg):
    pipeline: Pipeline = instantiate(cfg.Pipeline)
    text = "EGFR mutations are often implicated in lung cancer"
    doc = Document.create_simple_document(text)
    pipeline([doc])
    print(f"{doc.get_entities()}")

if __name__ == "__main__":
    kazu_test()
```

## License

Licensed under [Apache 2.0](https://github.com/AstraZeneca/KAZU/blob/main/LICENSE).

Kazu includes elements under compatible licenses (full licenses are in relevant files or as indicated):
- Some elements are a modification of code licensed under MIT by Explosion.AI - see the README [here](https://github.com/AstraZeneca/KAZU/blob/main/kazu/ontology_matching/README.md).
- The doc build process (conf.py's linkcode_resolve function) uses code modified from pandas, in turn modified from numpy. See [PANDAS_LICENSE.txt](https://github.com/AstraZeneca/KAZU/blob/main/docs/PANDAS_LICENSE.txt) and [NUMPY_LICENSE.txt](https://github.com/AstraZeneca/KAZU/blob/main/docs/NUMPY_LICENSE.txt)
- Elements of the model distillation code are inspired by or modified from Huawei Noah's Ark Lab [TinyBERT](https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/TinyBERT) and DMIS-Lab's [BioBERT](https://github.com/dmis-lab/biobert/tree/master).
  See the details in dataprocessor.py, models.py and tiny_transformer.py.
- PLSapbertModel is inspired by the code from [sapbert](https://github.com/cambridgeltl/sapbert), licensed under MIT. See the file for details, and see the [SapBert](#sapbert) section below regarding use of the model.
- GildaUtils in the string_normalizer.py file is modified from [Gilda](https://github.com/indralab/gilda). See the file for full details
  including the full BSD 2-Clause license.
- The AbbreviationFinderStep uses KazuAbbreviationDetector, which is a modified version of
  [SciSpacy](https://allenai.github.io/scispacy/)'s abbreviation finding algorithm, licensed under Apache 2.0 - see the files for full details.
- The JWTAuthenticationBackend Starlette Middleware in jwtauth.py is originally from [starlette-jwt](https://raw.githubusercontent.com/amitripshtos/starlette-jwt/master/starlette_jwt/middleware.py), licensed under BSD 3-Clause.
- The AddRequestIdMiddleware Starlette Middleware in req_id_header.py is modified from 'CustomHeaderMiddleware' in the [Starlette Middleware docs](https://www.starlette.io/middleware/#basehttpmiddleware).
  This is licensed under BSD 3-Clause along with the rest of Starlette.
- The kazu-jvm folder includes files like gradelw and gradelw.bat distributed by gradle under Apache 2.0 - see the files for details.
- [kazu/data.py](https://github.com/AstraZeneca/KAZU/blob/main/kazu/data.py) contains `AutoNameEnum`, which is `AutoName` from
  the [Python Enum Docs](https://docs.python.org/3/howto/enum.html#using-automatic-values) licensed under [Zero-Clause BSD](https://docs.python.org/3/license.html#zero-clause-bsd-license-for-code-in-the-python-release-documentation).

## Dataset licences

For the version of each ontology currently in use, please see the 'data_origin' field in kazu/conf/ontologies

### Under [Creative Commons Attribution-Share Alike 3.0 Unported Licence](https://creativecommons.org/licenses/by/3.0/legalcode)

#### Chembl
ChEMBL data is from http://www.ebi.ac.uk/chembl

#### CLO
CLO data is from http://www.ebi.ac.uk/ols/ontologies/clo

#### UBERON
UBERON data is from http://www.ebi.ac.uk/ols/ontologies/uberon

### Under [Creative Commons Attribution 4.0 Unported License](https://creativecommons.org/licenses/by/4.0/legalcode>)

#### MONDO
MONDO data is from http://www.ebi.ac.uk/ols/ontologies/mondo

#### CELLOSAURUS
CELLOSAURUS data is from https://www.cellosaurus.org/

#### Gene Ontology
Gene Ontology data is from http://purl.obolibrary.org/obo/go.owl

### Other licenced datasets and models

#### HPO

This service/product uses the Human Phenotype Ontology (version information). Find out more at http://www.human-phenotype-ontology.org

Freely licenced under https://hpo.jax.org/app/license

Sebastian Köhler, Michael Gargano, Nicolas Matentzoglu, Leigh C Carmody, David Lewis-Smith,
Nicole A Vasilevsky, Daniel Danis, Ganna Balagura, Gareth Baynam, Amy M Brower,
Tiffany J Callahan, Christopher G Chute, Johanna L Est, Peter D Galer, Shiva Ganesan,
Matthias Griese, Matthias Haimel, Julia Pazmandi, Marc Hanauer, Nomi L Harris,
Michael J Hartnett, Maximilian Hastreiter, Fabian Hauck, Yongqun He, Tim Jeske, Hugh Kearney,
Gerhard Kindle, Christoph Klein, Katrin Knoflach, Roland Krause, David Lagorce, Julie A McMurry,
Jillian A Miller, Monica C Munoz-Torres, Rebecca L Peters, Christina K Rapp, Ana M Rath,
Shahmir A Rind, Avi Z Rosenberg, Michael M Segal, Markus G Seidel, Damian Smedley,
Tomer Talmy, Yarlalu Thomas, Samuel A Wiafe, Julie Xian, Zafer Yüksel, Ingo Helbig,
Christopher J Mungall, Melissa A Haendel, Peter N Robinson,

The Human Phenotype Ontology in 2021,

Nucleic Acids Research, Volume 49, Issue D1, 8 January 2021, Pages D1207–D1217,

https://doi.org/10.1093/nar/gkaa1043

#### OPEN TARGETS
Open Targets datasets are kindly provided by www.opentargets.org, which are free for commercial use cases <https://platform-docs.opentargets.org/licence>

Ochoa, D. et al. (2021). Open Targets Platform: supporting systematic drug–target identification and prioritisation. Nucleic Acids Research.

https://doi.org/10.1093/nar/gkaa1027

#### STANZA

The Stanza framework:

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Association for Computational Linguistics (ACL) System Demonstrations. 2020.

https://arxiv.org/abs/2003.07082

Biomedical NLP models are derived from:

Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D. Manning, Curtis P. Langlotz.

Biomedical and Clinical English Model Packages in the Stanza Python NLP Library,

Journal of the American Medical Informatics Association. 2021.

https://doi.org/10.1093/jamia/ocab090

#### SCISPACY

Biomedical scispacy models are derived from

Mark Neumann, Daniel King, Iz Beltagy, Waleed Ammar

ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing

Proceedings of the 18th BioNLP Workshop and Shared Task

ACL 2019

https://www.aclweb.org/anthology/W19-5034

#### SAPBERT

Kazu uses a distilled form of SAPBERT, from

Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, Nigel Collier

Self-Alignment Pretraining for Biomedical Entity Representations

ACL 2021

https://aclanthology.org/2021.naacl-main.334/

#### GLINER

GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer.

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry Charnois

https://arxiv.org/abs/2311.08526

#### SETH

Kazu's SethStep uses Py4j to call the SETH mutation finder.

Thomas, P., Rocktäschel, T., Hakenberg, J., Mayer, L., and Leser, U. (2016).

[SETH detects and normalizes genetic variants in text](https://pubmed.ncbi.nlm.nih.gov/27256315/)

Bioinformatics (2016)

http://dx.doi.org/10.1093/bioinformatics/btw234

#### Opsin

Kazu's OpsinStep uses Py4j to call OPSIN: Open Parser for Systematic IUPAC nomenclature.

Daniel M. Lowe, Peter T. Corbett, Peter Murray-Rust, and Robert C. Glen

Chemical Name to Structure: OPSIN, an Open Source Solution

Journal of Chemical Information and Modeling 2011 51 (3), 739-753

DOI: [10.1021/ci100384d](https://doi.org/10.1021/ci100384d)


# Covidcast Indicators

[![License: MIT][mit-image]][mit-url]

In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals that would track in real-time and in fine geographic granularity all facets of the COVID-19 pandemic, aiding both nowcasting and forecasting. Delphi's long history in tracking and forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for user convenience and to provide data versioning for sources that do not track revisions themselves.

Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract informative aggregate signals, and output those signals---which we call **COVID-19 indicators**---in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html).

For client access to the API, along with a variety of other utilities, see our [R](https://cmu-delphi.github.io/covidcast/covidcastR/) and [Python](https://cmu-delphi.github.io/covidcast/covidcast-py/html/) packages.

For interactive visualizations (of a subset of the available indicators), see our [COVIDcast map](https://covidcast.cmu.edu).

## Organization

Utilities:

- `_delphi_utils_python` - common behaviors
- `_template_python` & `_template_r` - starting points for new data sources
- `ansible` & `jenkins` - automated testing and deployment
- `sir_complainsalot` - a Slack bot to check for missing data

Indicator pipelines: all remaining directories.

Each indicator pipeline includes its own documentation.

- Consult README.md for directions to install, lint, test, and run the pipeline for that indicator.
- Consult REVIEW.md for the checklist to use for code reviews.
- Consult DETAILS.md (if present) for implementation details, including handling of corner cases.

## Development

`prod` reflects what is currently in production. `main` is the staging branch for the next release.

1. Branch from `main` to develop a new change
2. PR into `main` and assign a reviewer (or tag someone) to get feedback on your change. List the issue number under `Fixes` if your change resolves an existing [GitHub Issue](https://github.com/cmu-delphi/covidcast-indicators/issues).
3. Add new commits to your branch in response to feedback.
4. When approved, tag an admin to merge the PR. Let them know if this change should be released immediately, at a set future date, or if it can just go along for the ride whenever the next release happens.

### Linting and Formatting

Each indicator has a `make lint` command to check for linting errors and a `make
format` command to incrementally format your code (using
[darker](https://github.com/akaihola/darker)). These are both automated with a
[Github Action](.github/workflows/python-ci.yml).

If you get the error `ERROR:darker.git:fatal: Not a valid commit name <hash>`,
then it's likely because your local main branch is not up to date; either you
need to rebase or merge. Note that `darker` reads from `pyproject.toml` for
default settings.

If the lines you change are in a file that uses 2 space indentation, `darker`
will indent the lines around your changes and not the rest, which will likely
break the code; in that case, you should probably just pass the whole file
through black. You can do that with the following command (using the same
virtual environment as above):

```sh
env/bin/black <file>
```

## Release Process

The release process consists of multiple steps which can all be done via the GitHub website:

1. Go to [create_release GitHub Action](https://github.com/cmu-delphi/covidcast-indicators/actions/workflows/create-release.yml) and click the `Run workflow` dropdown button. Leave branch as `main` unless you know what you're doing. Enter the type of release (patch: bugfixes, params file changes, new signals for existing indicators; minor: new indicators, new utilities; major: backwards-incompatible changes requiring substantial refactoring) and GitHub will automatically compute the next version number for you; alternately, specify the version number by hand. Hit the green `Run workflow` button.
2. The action will prepare a new release and generate an associated [Pull Request](https://github.com/cmu-delphi/covidcast-indicators/pulls).
3. Edit the PR description and **list all pull requests included in this release**. This is a manual step to make sure you are aware of 100% of the changes that will be deployed. You can use `#xxx` notation and GitHub will automatically render the title of each PR in Preview mode and when the edit is saved.
4. Verify that CI passes for the PR as a whole and for the most-recent/bottom-most commit in the PR. We're currently having problems where [python-ci does not run on release PRs](https://github.com/cmu-delphi/covidcast-indicators/issues/1310), but if you see a green check next to the most-recent commit you should be fine.
5. Approve the PR, merge it, and delete the branch.
6. Jenkins will automatically deploy the most-recently-built indicator packages to the pipeline servers
7. Another GitHub action will automatically
   1. Create a git tag
   2. Create another [Pull Request](https://github.com/cmu-delphi/covidcast-indicators/pulls) to merge the changes back into the `main` branch
   3. (if `delphi-utils` was updated) Upload the new version of `delphi-utils` to PyPI
8. Approve the sync PR, merge it, and delete the branch
9. Done

You may need to be an admin to perform some of the steps above.

## License

This repository is released under the **MIT License**.

[mit-image]: https://img.shields.io/badge/License-MIT-yellow.svg
[mit-url]: https://opensource.org/licenses/MIT


# cftime
Time-handling functionality from netcdf4-python

[![Build status](https://github.com/Unidata/cftime/workflows/cftime%20test/badge.svg)](https://github.com/Unidata/cftime/actions)
[![PyPI package](https://badge.fury.io/py/cftime.svg)](http://python.org/pypi/cftime)
[![Coverage Status](https://coveralls.io/repos/github/Unidata/cftime/badge.svg?branch=master)](https://coveralls.io/github/Unidata/cftime?branch=master)
[![Tag Status](https://img.shields.io/github/tag/UniData/cftime.svg)](https://github.com/Unidata/cftime/tags)
[![Release Status](https://img.shields.io/github/release/UniData/cftime.svg)](https://github.com/Unidata/cftime/releases)
[![Commits Status](https://img.shields.io/github/commits-since/UniData/cftime/latest.svg)](https://github.com/UniData/cftime/commits/master)
[![DOI](https://zenodo.org/badge/73107250.svg)](https://zenodo.org/badge/latestdoi/73107250)

## News
For details on the latest updates, see the [Changelog](https://github.com/Unidata/cftime/blob/master/Changelog).

6/7/2024:  Version 1.6.4 release.  Wheels for muslinux and aarch64, numpy 2.0 compatibility.

10/20/2023:  Version 1.6.3 released.  Support for python 3.12, cython 3.0, strptime formats without separators.

9/18/2022:  Version 1.6.2 released.  strptime method added, fix for num2date failure on
empty integer array, date2num 'longdouble' keyword added. New wheel building workflow.

6/30/2022:  Version 1.6.1 released.  Fixes for numpy 1.23.0, updated CI/CD.

3/4/2022:  Version 1.6.0 released.  Big speed-ups for num2date, date2index bugfix for select != 'exact' when select='exact' works, fix for date2num with masked array inputs.

1/22/2022: Version 1.5.2 released (wheels for Apple M1 available on pypi for python 3.8,3.9 and 3.10). is_leap_year
function added (issue #259).

10/31/2021: Version 1.5.1.1 released (new binary wheels for python 3.10).

10/1/2021:  Version 1.5.1 released. Changed default behavior of ``proleptic_gregorian``
to has_year_zero=T (since it is allowed in ISO-8601 and CF does not specify the
year zero convention for this calendar). Raise warning message when trying
to create a calendar that is not supported by CF version 1.9 (no years < 1
allowed for 'standard'/'gregorian' or 'julian'  calendars).
Added support for "common_year" and "common_years" units for "noleap"
and "365_day" calendars.

5/20/2021:  Version 1.5.0 released.  Includes support for astronomical year numbering
(including the year zero) for real-world calendars ('julian', 'gregorian'/'standard',
and 'proleptic_gregorian') using 'has_year_zero' `cftime.datetime` kwarg.
Added a 'change_calendar' `cftime.datetime` method to switch to another
'real-world' calendar to enable comparison of instances with different calendars.
Some legacy classes and functions removed (`utime`, `JulianDayFromDate` and
`DateFromJulianDay`). The functionality of `JulianDayFromDate` and
`DateFromJulianDay` is now available from the methods `cftime.datetime.toordinal`
and `cftime.datetime.fromordinal`.

2/2/2021:  Version 1.4.1 released. Restore use of calendar-specific subclasses
in `cftime.num2date`, `cftime.datetime.__add__`, and `cftime.datetime.__sub__`.
The use of this will be removed in a later release.
Add 'fromordinal' static method to create a cftime.datetime instance
from a julian day ordinal and calendar (inverse of 'toordinal').

2/1/2021:  Version 1.4.0 released.  License changed to MIT (GPL'ed code replaced).
Roundtrip accuracy improved for units other than microseconds. Added
cftime.datetime.toordinal method, returns integer julian day number.

1/17/2021: Version 1.3.1 released.

11/16/2020:  Version 1.3.0 released. **API change**: The `cftime.datetime` constructor now creates
 'calendar-aware' instances (default is `'standard'` calendar, if `calendar=''` or `None` the instance
 is not calendar aware and some methods, like `dayofwk`, `dayofyr`, `__add__` and `__sub__`, will not work)
 See discussion for issue [#198](https://github.com/Unidata/cftime/issues/198).
 The calendar specific sub-classes are now deprecated, but remain for now
 as stubs that just instantiate the base class and override `__repr__`.
 The default calendar in `cftime.date2num` has been changed from `'standard'` to `None`
 (the calendar associated with first input datetime object is used to define the calendar).

07/20/2020: Version 1.2.1 released.  Fixes a couple of regressions introduced in 1.2.0. See Changelog for details.

7/06/2020:  version 1.2.0 released. New microsecond accurate algorithm for date2num/num2date contributed by [spencerkclark](https://github.com/spencerkclark). Bugs fixed in masked array handling.

5/12/2020:  version 1.1.3 released.  Add isoformat method for compatibility with python datetime (issue #152).
 Make 'standard' default calendar for cftime.datetime so that dayofwk,dayofyr methods don't fail (issue #169).

4/20/2020:  version 1.1.2 released.  Code optimization, fix logic so `only_use_cftime_datetimes=False` works as
 expected (issues [#158](https://github.com/Unidata/cftime/issues/158) and [#165](https://github.com/Unidata/cftime/issues/165)).

3/16/2020:  version 1.1.1 released.  Fix bug in microsecond formatting, ensure identical num2date results if input is an array of times, or a single scalar ([issue #143](https://github.com/Unidata/cftime/issues/143)).

2/12/2020:  version 1.1.0 released.  `cftime.datetime` instances are returned by default from `num2date`
(instead of returning python datetime instances where possible ([issue #136](https://github.com/Unidata/cftime/issues/136))).  `num2pydate`
convenience function added (always returns python datetime instances, [issue #134](https://github.com/Unidata/cftime/issues/134)). Fix for
fraction seconds in reference date string ([issue #140](https://github.com/Unidata/cftime/issues/140)). Added `daysinmonth` attribute
([issue #137](https://github.com/Unidata/cftime/issues/137)).

10/25/2019:  version 1.0.4.2 released (fix for [issue #126](https://github.com/Unidata/cftime/issues/126)).

10/21/2019:  version 1.0.4 released.

12/05/2018:  version 1.0.3.4 released (just to fix a problem with the source
tarball on pypi).

12/05/2018:  version 1.0.3.1 released.  Bugfix release (fixed issue with installation
when cython not installed, regression on 32-bit platforms, workaround for pandas
compatibility).

12/01/2018:  version 1.0.3 released. Test coverage with coveralls.io, improved round-tripping accuracy for non-real world calendars (like `360_day`).

10/27/2018:  version 1.0.2 released. Improved accuracy (from approximately 1000 microseconds to 10 microseconds on x86
platforms). Refactored calendar calculations now allow for negative reference years. num2date function now more than an
order of magnitude faster. `months since` units now allowed, but only for `360_day` calendar.

08/15/2018:  version 1.0.1 released.

11/8/2016: `cftime` was split out of the [netcdf4-python](https://github.com/Unidata/netcdf4-python) package.

## Quick Start
* Clone GitHub repository (`git clone https://github.com/Unidata/cftime.git`), or get source tarball from [PyPI](https://pypi.python.org/pypi/cftime). Links to Windows and OS X precompiled binary packages are also available on [PyPI](https://pypi.python.org/pypi/cftime).

* Make sure [numpy](http://www.numpy.org/) and [Cython](http://cython.org/) are
  installed and you have [Python](https://www.python.org) 2.7 or newer.

* Run `python setup.py build`, then `python setup.py install` (with `sudo` if necessary).

* To run all the tests, execute `py.test`.

## Documentation
See the online [docs](http://unidata.github.io/cftime) for more details.


# Execution Spec Tests

![latest version](https://img.shields.io/github/v/release/ethereum/execution-spec-tests)

[ethereum/execution-spec-tests](https://github.com/ethereum/execution-spec-tests) is both a collection of test cases and a framework implemented in Python to generate tests for Ethereum execution clients.

The framework collects and executes the test cases in order to generate _test fixtures_ (JSON) which can be consumed by any execution client to verify their implementation of [ethereum/execution-specs](https://github.com/ethereum/execution-specs). The fixtures, which define state transition and block tests, are generated by the framework using one of the `t8n` command-line tools that are provided by most execution clients, see below for an overview of the supported `t8n` tools.

```mermaid
---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
  style C stroke:#333,stroke-width:2px
  style D stroke:#333,stroke-width:2px
  style G stroke:#F9A825,stroke-width:2px
  style H stroke:#F9A825,stroke-width:2px

  subgraph "ethereum/go-ethereum, ..."
    C[<code>evm t8n</code>\nexternal executable]
  end

  subgraph ethereum/solidity
    D[<code>solc</code>\nexternal executable]
  end

  subgraph ethereum/EIPs
    E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
  end

  subgraph "ethereum/execution-spec-tests"
    A(<code>./tests/**/*.py</code>\nPython Test Cases)
    B([<code>$ fill ./tests/</code>\nPython Framework])
  end

  subgraph Test Fixture Consumers
    subgraph ethereum/hive
      G([<code>$ hive ...</code>\nGo Test Framework])
    end
    H([Client executables])
  end

  C <-.-> B
  D <-.-> B
  A --> B
  E <-.-> |retrieve latest spec version\ncheck tested spec version| B
  B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
  F -->|input| G
  F -->|input| H
```

The generated test fixtures can be used:

1. Directly by client teams' test frameworks, and,
2. In the integration tests executed in the [ethereum/hive](https://github.com/ethereum/hive) framework.

## Transition Tool Support

The following transition tools are supported by the framework:

| Client | "t8n" Tool | Tracing Support |
| -------| ---------- | --------------- |
| [ethereum/evmone](https://github.com/ethereum/evmone) | `evmone-t8n` | Yes |
| [ethereum/execution-specs](https://github.com/ethereum/execution-specs) | `ethereum-spec-evm` | Yes |
| [ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) | [`evm t8n`](https://github.com/ethereum/go-ethereum/tree/master/cmd/evm) | Yes |
| [hyperledger/besu](https://github.com/hyperledger/besu/tree/main/ethereum/evmtool) | [`evm t8n-server`](https://github.com/hyperledger/besu/tree/main/ethereum/evmtool) | Yes |
| [status-im/nimbus-eth1](https://github.com/status-im/nimbus-eth1) | [`t8n`](https://github.com/status-im/nimbus-eth1/blob/master/tools/t8n/readme.md) | Yes |

### Upcoming EIP Development

Generally, specific `t8n` implementations and branches must be used when developing tests for upcoming EIPs.

We use named reference tags to point to the specific version of the `t8n` implementation that needs to be used fill the tests.

All current tags, their t8n implementation and branch they point to, are listed in [configs/evm.yaml](configs/evm.yaml).

## Getting Started

### Prerequisites

The following requires a Python 3.10, 3.11 or 3.12 installation.

### Quick Start

This guide installs stable versions of the required external (go-ethereum) `evm` and `solc` executables and will only enable generation of test fixtures for features deployed to mainnet. In order to generate fixtures for features under active development, you can follow the steps below and then follow the [additional steps in the online doc](https://ethereum.github.io/execution-spec-tests/getting_started/executing_tests_dev_fork/).

1. Ensure go-ethereum's `evm` tool and `solc` ([0.8.20](https://github.com/ethereum/solidity/releases/tag/v0.8.20), [0.8.21](https://github.com/ethereum/solidity/releases/tag/v0.8.21), [0.8.22](https://github.com/ethereum/solidity/releases/tag/v0.8.22), [0.8.23](https://github.com/ethereum/solidity/releases/tag/v0.8.23)  supported) are in your path. Either build the required versions, or alternatively:

    ```console
    sudo add-apt-repository -y ppa:ethereum/ethereum
    sudo apt-get update
    sudo apt-get install ethereum solc
    ```

    More help:

    - [geth installation doc](https://geth.ethereum.org/docs/getting-started/installing-geth#ubuntu-via-ppas).
    - [solc installation doc](https://docs.soliditylang.org/en/latest/installing-solidity.html#linux-packages).

    Help for other platforms is available in the [online doc](https://ethereum.github.io/execution-spec-tests/getting_started/quick_start/).

2. Clone the [execution-spec-tests](https://github.com/ethereum/execution-spec-tests) repo and install its dependencies (it's recommended to use a virtual environment for the installation):

   ```console
   git clone https://github.com/ethereum/execution-spec-tests
   cd execution-spec-tests
   python3 -m venv ./venv/
   source ./venv/bin/activate
   pip install -e '.[docs,lint,test]'
   ```

3. Verify the installation:
    1. Explore test cases:

       ```console
       fill --collect-only
       ```

       Expected console output:
         ![Screenshot of pytest test collection console output](docs/getting_started/img/pytest_collect_only.png)

    2. Execute the test cases (verbosely) in the `./tests/berlin/eip2930_access_list/test_acl.py` module:

        ```console
        fill -v tests/berlin/eip2930_access_list/test_acl.py
        ```

        Expected console output:
          ![Screenshot of pytest test collection console output](docs/getting_started/img/pytest_run_example.png)
        Check:

        1. The versions of the `evm` and `solc` tools are as expected (your versions may differ from those in the highlighted box).
        2. The corresponding fixture file has been generated:

           ```console
           head fixtures/blockchain_tests/berlin/eip2930_access_list/acl/access_list.json
           ```

## Usage

More information on how to obtain and consume the [released test fixtures](https://github.com/ethereum/execution-spec-tests/releases) can be found in the [documentation](https://ethereum.github.io/execution-spec-tests/main/consuming_tests/).

For further help with working with this codebase, see the [online documentation](https://ethereum.github.io/execution-spec-tests/):

1. Learn [useful command-line flags](https://ethereum.github.io/execution-spec-tests/getting_started/executing_tests_command_line/).
2. [Execute tests for features under development](https://ethereum.github.io/execution-spec-tests/getting_started/executing_tests_dev_fork/) via the `--from=FORK1` and `--until=FORK2` flags.
3. _Optional:_ [Configure VS Code](https://ethereum.github.io/execution-spec-tests/getting_started/setup_vs_code/) to auto-format Python code and [execute tests within VS Code](https://ethereum.github.io/execution-spec-tests/getting_started/executing_tests_vs_code/#executing-and-debugging-test-cases).
4. Implement a new test case, see [Writing Tests](https://ethereum.github.io/execution-spec-tests/writing_tests/).

## Coverage

The available test cases can be browsed in the [Test Case Reference doc](https://ethereum.github.io/execution-spec-tests/tests/).

## Installation Troubleshooting

If you encounter issues during the installation process, please refer to the [Installation Troubleshooting](https://ethereum.github.io/execution-spec-tests/main/getting_started/installation_troubleshooting/) page.

## Contributing

Contributions and feedback are welcome. Please see the [online documentation](https://ethereum.github.io/execution-spec-tests/writing_tests/) for this repository's coding standards and help on implementing new tests.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.


fv3net
======

[![CircleCI](https://circleci.com/gh/ai2cm/fv3net/tree/master.svg?style=svg)](https://circleci.com/gh/ai2cm/fv3net/tree/master)

Improving the GFDL FV3 model physics with machine learning. See the [documentation](https://vulcanclimatemodeling.com/docs/fv3net/) for more information on using this suite of tools.

Disclaimer: This is a work in progress.


[![Build Status](https://travis-ci.org/LSSTDESC/Spectractor.svg?branch=master)](https://travis-ci.org/LSSTDESC/Spectractor)
[![Coverage Status](https://coveralls.io/repos/github/LSSTDESC/Spectractor/badge.svg?branch=master)](https://coveralls.io/github/LSSTDESC/Spectractor?branch=master)
[![Documentation Status](https://readthedocs.org/projects/spectractor/badge/?version=latest)](https://spectractor.readthedocs.io/en/latest/?badge=latest)

# Spectractor

The goal of Spectractor is to measure the atmospheric transmission and intempt extracting spectra from slitless spectrophotometric images. It has been optimized on CTIO images but can be configured to analyse any kind of slitless data that contains the order 0 and the order 1 of a spectrum. In particular it can  be used to estimate the atmospheric transmission of the LSST site using the dedicated  Auxiliary Telescope.

Spectractor is structured in three subpackages:
- `spectractor.extractor`: extracts as most information as possible from a slitless data  image, as the amplitude of the spectrum, the PSF evolution with the wavelength, the pixel to wavelength calibration, an estimate of the background under the spectrum, the position of the order 0;
- `spectractor.simulation`: contains all the tools to simulate a spectrogram, as the atmospheric transmission simulation, the inclusion of instrumental throughput, the simulation of mock  slitless  data images;
- `spectractor.fit`: compares the extracted data with simulations to estimate the atmospheric transmission and refine the spectral extraction.

Some submodules complete the structures with generic functions:
- `spectractor.parameters`: contains all the global parameters of Spectractor to set its general behaviour, the instrumental characteritics, etc;
- `spectractor.config`: tools to read config `.ini` text files and set the global parameters;
- `spectractor.logbook`: tools to read logbook `.csv` text files and get some metadata relative to the data images that are not contained in the header;
- `spectractor.tools`: contains generic functions shared by all  the subpackages (fitting procedures, plotting functions, etc).

## Installation

Spectractor is written in Python 3.9. The mandatory dependencies are listed in the `requirements.txt` file. To install Spectractor, just run
```
pip install -r requirements.txt .
```

To simulate atmospheric transmission, Spectractor needs [libradtran](http://www.libradtran.org/doku.php) (optional). There are two ways to install this software:
- use the [Rubin observatory conda package](https://anaconda.org/conda-forge/rubin-libradtran)
```
conda install -c conda-forge rubin-libradtran
```
- compile it from [sources](http://www.libradtran.org/doku.php): it needs the installation of [netcdf](https://www.unidata.ucar.edu/software/netcdf/) and a python 2 environment (for the compilation only, not the usage); `uvpsec` executable must in the user `$PATH` or the user has to set an environmental variable `$LIBRADTRAN_DIR` pointing to the install directory.

To find accurate star centroids using astrometry, one can install [astrometry.net](https://astrometrynet.readthedocs.io/en/latest/) (optional). There are two ways to install this software:
- use the anaconda package
```
conda install -c conda-forge astrometry
```
and then download and move [index files](http://astrometry.net/doc/readme.html#getting-index-files) in the `$CONDA_PREFIX/data` folder.
- compile it from [sources](https://astrometrynet.readthedocs.io/en/latest/): it needs the installation of [netpbm](https://netpbm.sourceforge.net/) and [wcslib](https://www.atnf.csiro.au/people/mcalabre/WCS/wcslib/); `solve-field` executable must in the user `$PATH` or the user has to set an environmental variable `$ASTROMETRYNET_DIR` pointing to the install directory.

For faster matrix multiplications, if available for a given computer configuration, one can install [sparse_dot_mkl](https://anaconda.org/conda-forge/sparse_dot_mkl):
```
conda install -c conda-forge sparse_dot_mkl
```

Spectractor is able to perform parameter fits using the MCMC library [emcee](https://emcee.readthedocs.io/en/stable/) (optional) with [mpi4py](https://mpi4py.readthedocs.io/en/stable/) and [h5py](https://www.h5py.org/).  The latter might be better installed using `conda install ...` command to get their own dependencies (openmp and hdf5).

## Basic extraction

The main file is `spectractor/extractor/extractor.py` with the function `Spectractor`. It extracts the spectrum from a science data image (deflatted, debiased), given:
- the path to the FITS image from which to extract the image,
- the path of the output directory to save the extracted spectrum (created automatically if it does not exist yet),
- the rough pr exact position of the object in the image (in pixels),
- the name of the disperser (as it is named in the `spectractor/extractor/dispersers/` folder),
- the name of the config .ini file,
- optionally the name of the target (to search for the extra-atmospheric spectrum if available).

```
filename="./tests/data/reduc_20170530_134.fits"
output_directory="./outputs/"
guess = [745,643]
disperser_label = "HoloAmAg"
config = "./config/ctio.ini"
target = "HD111980"
```

Then the spectrum is simply extracted from the image and saved in a new fits file using the `Spectractor` function:
```
spectrum = Spectractor(filename, output_directory, guess=guess, target_label=target, disperser_label=disperser_label, config=config)
```

or typing the following command within a terminal:
```
python runExtractor.py ./tests/data/reduc_20170530_134.fits -o outputs --config ./config/ctio.ini --xy [745, 643] --target HD111980 --grating HoloAmAg
```

Spectractor comes with two verbosity modes, set in the `parameters.py` file:
- `VERBOSE` (or -v, --verbose) : the first level of verbosity that returns many information along the process to know what the program is doing; it also plots the output spectrum
- `DEBUG` (or -d, --debug) : in the debugging mode some intermediate plots are produced to see the performance of the program.

### Plot a spectrum

To see the result of the extraction process, it is possible to load it via the `Spectrum` class and plot it:
```
spectrum = Spectrum('./tests/data/reduc_20170530_134_spectrum.fits', config="./config/ctio.ini)
spectrum.plot_spectrum()
spectrum.plot_spectrogram()
```
This object is also returned by the `Spectractor` function.

### Tutorial notebook

A tutorial Jupyter notebook is available in the `notebooks` folder.


# Lookit-api

[![CircleCI](https://dl.circleci.com/status-badge/img/gh/lookit/lookit-api/tree/develop.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/lookit/lookit-api/tree/develop)

For full documentation, see <http://lookit.readthedocs.io> (source: <https://github.com/lookit/lookit-docs/>)

This is the codebase for the Lookit project, an online developmental laboratory. It includes an Experimenter platform for designing and administering research studies, and a participant-facing platform where users can signup and take part in studies. It is built using Django, PostgreSQL, and Ember.js (see Ember portion of codebase, [ ember-lookit-frameplayer](https://github.com/lookit/ember-lookit-frameplayer)), and is developed by [MIT](https://mit.edu) and [Center for Open Science](https://cos.io/).