# Registry of Open Data on AWS

A repository of publicly available datasets that are available for access from AWS resources. Note that datasets in this registry are available via AWS resources, but they are not provided by AWS; these datasets are owned and maintained by a variety of government organizations, researchers, businesses, and individuals.

## What is this for?

When data is shared on AWS, anyone can analyze it and build services on top of it using a broad range of compute and data analytics products, including [Amazon EC2](https://aws.amazon.com/ec2/), [Amazon Athena](https://aws.amazon.com/athena/), [AWS Lambda](https://aws.amazon.com/lambda/), and [Amazon EMR](https://aws.amazon.com/emr/). Sharing data in the cloud lets data users spend more time on data analysis rather than data acquisition. This repository exists to help people promote and discover datasets that are available via AWS resources.

## How are datasets added to the registry?

Each dataset in this repository is described with metadata saved in a YAML file in the [/datasets](/datasets) directory. We use these YAML files to provide three services:

- A [Registry of Open Data on AWS browser](https://registry.opendata.aws/).
- A hosted YAML file listing all of the dataset entries.
- Hosted YAML files for each dataset.

The YAML files use this structure:

```yaml
Name:
Description:
Documentation:
Contact:
ManagedBy:
UpdateFrequency:
Tags:
  -
License:
Citation:
Resources:
  - Description:
    ARN:
    Region:
    Type:
    Explore:
DataAtWork:
  Tutorials:
    - Title:
      URL:
      NotebookURL:
      AuthorName:
      AuthorURL:
      Services:
  Tools & Applications:
    - Title:
      URL:
      AuthorName:
      AuthorURL:
  Publications:
    - Title:
      URL:
      AuthorName:
      AuthorURL:
DeprecatedNotice:
ADXCategories:
  -
```

The metadata required for each dataset entry is as follows:

| Field | Type | Description & Style |
| --- | --- | --- |
| **Name** | String | The public facing name of the dataset. Spell out acronyms and abbreviations. We do not require "AWS" or "Open Data" to be in the dataset name. Must be between 5 and 130 characters.|
|**Description**|String|A high-level description of the dataset. Only the first 600 characters will be displayed on the homepage of the [Registry of Open Data on AWS](https://registry.opendata.aws)|
|**Documentation**|URL|A link to documentation of the dataset, preferably hosted on the data provider's website or Github repository.|
|**Contact**|String|May be an email address, a link to contact form, a link to GitHub issues page, or any other instructions to contact the producer of the dataset|
|**ManagedBy**|String|The name of the laboratory, institution, or organization who is responsible for the data ingest process. Avoid using individuals. If your institution manages several datasets hosted by the Public Dataset Program, please list the managing institution identically. For an example why, check out the Managed By section of the [TARGET dataset](https://registry.opendata.aws/target/)|
|**UpdateFrequency**|String|An explanation of how frequently the dataset is updated|
|**Tags**|List of strings|Select tags that are related to an intrinsic property or descriptor of the dataset. A list of supported tags is maintained in the tags.yaml file in this repo. If you want to recommend a tag that is not included in [tags.yaml](tags.yaml), please submit a pull request to add it to that file.|
|**License**|String|An explanation of the dataset license and/or a URL to more information about data terms of use of the dataset|
|**Citation** (Optional)|String|Custom citation language to be used when citing this dataset, which will be appended to the default citation used for all datasets. Default citation language is as follows: "[DATASET NAME] was accessed on [DATE] at registry.opendata.aws/[dataset]"|
|**Resources**|List of lists|A list of AWS resources that users can use to consume the data. Each resource entry requires the metadata below:|
|**Resources > Description**|String|A technical description of the data available within the AWS resource, including information about file formats and scope.|
|**Resources > ARN**|String|Amazon Resource Name for resource, e.g. arn:aws:s3:::commoncrawl|
|**Resources > Region**|String|AWS region unique identifier, e.g. us-east-1|
|**Resources > Type**|String|Can be _CloudFront Distribution_, _DB Snapshot_, _S3 Bucket_, or _SNS Topic_. A list of supported resources is maintained in the [resources.yaml](resources.yaml) file in this repo. If you want to recommend a resource that is not included in [resources.yaml](resources.yaml), please submit a pull request to add it to that file.|
|**Resources > RequesterPays** (Optional)|Boolean|Only appropriate for Amazon S3 buckets, indicates whether the bucket has [Requester Pays](https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) enabled or not.|
|**Resources > AccountRequired** (Optional)|String|Is an AWS account required to access this data? Note that while Requester Pays means you will need an account, this is meant for cases where an account is required outside of that scenario.|
|**Resources > ControlledAccess** (Optional)|String|Only appropriate for Amazon S3 buckets with controlled access. Please provide a URL to instructions on how to request and gain access to the S3 bucket.|
|**Resources > Explore** (Optional)|List of strings|Additional links that can be used to explore the bucket resource, i.e. links to S3 JS Explorer index.html for the bucket or the AWS S3 console.|
|**DataAtWork  [> Tutorials, Tools & Applications, Publications]**  (Optional)|List of lists|A list of links to example tutorials, tools & applications, publications that use the data.|
|**DataAtWork [> Tutorials, Tools & Applications, Publications] > Title**|String|The title of the tutorial, tool, application, or publication that uses the data.|
|**DataAtWork [> Tutorials, Tools & Applications, Publications] > URL**|URL|A link to the tutorial, tool, application, or publication that uses the data.|
|**DataAtWork [> Tutorials, Tools & Applications, Publications] > AuthorName**|String|Name(s) of person or entity that created  the tutorial, tool, application, or publication. Limit scientific publication author lists to the first six authors in the format Last Name First Initial, followed by 'et al'.|
|**DataAtWork [> Tutorials, Tools & Applications, Publications] > AuthorURL** (Optional)|String|URL for person or entity that created the tutorial, tool, application, or publication.|
|**DataAtWork [> Tutorials] > NotebookURL** (Optional)|URL|A link to a Jupyter notebook (.ipynb) on GitHub that shows how this data can be used.|
|**DataAtWork [> Tutorials] > Services** (Optional)|String|For tutorials only. List AWS Services applied in your tutorial. A list of supported AWS services is maintained in the [services.yaml](services.yaml) file in this repo. If you want to recommend a resource that is not included in [services.yaml](services.yaml), please submit a pull request to add it to that file.|
|**DeprecatedNotice** (Optional)|String|Only appropriate for datasets that are being retired, indicates to users that the dataset will soon be deprecated and should include the date that the dataset will no longer be available.|
|**ADXCategories**|List of strings|Allowed categories can be found in [adx_categories.yaml](adx_categories.yaml), at most, 2 can be added. Adding categories to your listing will improve searchability within the AWS Data Exchange.|

Note also that we use the name of each YAML file as the URL slug for each dataset on the [Registry of Open Data on AWS website](https://registry.opendata.aws). E.g. the metadata from `1000-genomes.yaml` is listed at `https://registry.opendata.aws/1000-genomes/`

### Example entry

Here is an example of the metadata behind this dataset registration: https://registry.opendata.aws/noaa-nexrad/

```yaml
Name: NEXRAD on AWS
Description: Real-time and archival data from the Next Generation Weather Radar (NEXRAD) network.
Documentation: https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-nexrad
Contact: noaa.bdp@noaa.gov
ManagedBy: "[NOAA](http://www.noaa.gov/)"
UpdateFrequency: New Level II data is added as soon as it is available.
Tags:
  - aws-pds
  - earth observation
  - natural resource
  - weather
  - meteorological
  - sustainability
License: There are no restrictions on the use of this data.
Resources:
  - Description: NEXRAD Level II archive data
    ARN: arn:aws:s3:::noaa-nexrad-level2
    Region: us-east-1
    Type: S3 Bucket
    Explore:
    - '[Browse Bucket](https://noaa-nexrad-level2.s3.amazonaws.com/index.html)'
  - Description: NEXRAD Level II real-time data
    ARN: arn:aws:s3:::unidata-nexrad-level2-chunks
    Region: us-east-1
    Type: S3 Bucket
  - Description: "[Rich notifications](https://github.com/awslabs/open-data-docs/tree/main/docs/noaa/noaa-nexrad#subscribing-to-nexrad-data-notifications) for real-time data with filterable fields"
    ARN: arn:aws:sns:us-east-1:684042711724:NewNEXRADLevel2ObjectFilterable
    Region: us-east-1
    Type: SNS Topic
  - Description: Notifications for archival data
    ARN: arn:aws:sns:us-east-1:811054952067:NewNEXRADLevel2Archive
    Region: us-east-1
    Type: SNS Topic
DataAtWork:
  Tutorials:
    - Title: NEXRAD on EC2 tutorial
      URL: https://github.com/openradar/AMS_radar_in_the_cloud
      Services: EC2
      AuthorName: openradar
      AuthorURL: https://github.com/openradar
    - Title: Using Python to Access NCEI Archived NEXRAD Level 2 Data (Jupyter notebook)
      URL: http://nbviewer.jupyter.org/gist/dopplershift/356f2e14832e9b676207
      AuthorName: Ryan May
      AuthorURL: http://dopplershift.github.io
    - Title: Mapping Noaa Nexrad Radar Data With CARTO
      URL: https://carto.com/blog/mapping-nexrad-radar-data/
      AuthorName: Stuart Lynn
      AuthorURL: https://carto.com/blog/author/stuart-lynn/
  Tools & Applications:
    - Title: nexradaws on pypi.python.org - python module to query and download Nexrad data from Amazon S3
      URL: https://pypi.org/project/nexradaws/
      AuthorName: Aaron Anderson
      AuthorURL: https://github.com/aarande
    - Title: WeatherPipe - Amazon EMR based analysis tool for NEXRAD data stored on Amazon S3
      URL: https://github.com/stephenlienharrell/WeatherPipe
      AuthorName: Stephen Lien Harrell
      AuthorURL: https://github.com/stephenlienharrell
  Publications:
    - Title: Seasonal abundance and survival of North America’s migratory avifauna determined by weather radar
      URL: https://www.nature.com/articles/s41559-018-0666-4
      AuthorName: Adriaan M. Dokter, Andrew Farnsworth, Daniel Fink, Viviana Ruiz-Gutierrez, Wesley M. Hochachka, Frank A. La Sorte, Orin J. Robinson, Kenneth V. Rosenberg & Steve Kelling
    - Title: Unlocking the Potential of NEXRAD Data through NOAA’s Big Data Partnership
      URL: https://journals.ametsoc.org/doi/full/10.1175/BAMS-D-16-0021.1
      AuthorName: Steve Ansari and Stephen Del Greco
```

## How can I contribute?

You are welcome to contribute dataset entries or usage examples to the Registry of Open Data on AWS. Please review our [contribution guidelines](CONTRIBUTING.md).

### New to Github and contributing via pull requests?
In addition to [Github's getting started documentation](https://docs.github.com/en/get-started), this [video tutorial shows the complete end-to-end process of contributing to this repository](https://youtu.be/5nocWdjN1DA).


MetPy
=====

[![MetPy Logo](https://github.com/Unidata/MetPy/raw/main/docs/_static/metpy_150x150.png)](https://unidata.github.io/MetPy/)
[![Unidata Logo](https://github.com/Unidata/MetPy/raw/main/docs/_static/unidata_150x150.png)](https://www.unidata.ucar.edu)

[![License](https://img.shields.io/pypi/l/metpy.svg)](https://pypi.python.org/pypi/MetPy/)
[![Gitter](https://badges.gitter.im/Unidata/MetPy.svg)](https://gitter.im/Unidata/MetPy?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=round-square)](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github)

[![Latest Docs](https://github.com/Unidata/MetPy/workflows/Build%20Docs/badge.svg)](http://unidata.github.io/MetPy)
[![PyPI Package](https://img.shields.io/pypi/v/metpy.svg)](https://pypi.python.org/pypi/MetPy/)
[![Conda Package](https://anaconda.org/conda-forge/metpy/badges/version.svg)](https://anaconda.org/conda-forge/metpy)
[![PyPI Downloads](https://img.shields.io/pypi/dm/metpy.svg)](https://pypi.python.org/pypi/MetPy/)
[![Conda Downloads](https://anaconda.org/conda-forge/metpy/badges/downloads.svg)](https://anaconda.org/conda-forge/metpy)

[![PyPI Tests](https://github.com/Unidata/MetPy/workflows/PyPI%20Tests/badge.svg)](https://github.com/Unidata/MetPy/actions?query=workflow%3A%22PyPI+Tests%22)
[![Conda Tests](https://github.com/Unidata/MetPy/workflows/Conda%20Tests/badge.svg)](https://github.com/Unidata/MetPy/actions?query=workflow%3A%22Conda+Tests%22)
[![Code Coverage Status](https://codecov.io/github/Unidata/MetPy/coverage.svg?branch=main)](https://codecov.io/github/Unidata/MetPy?branch=main)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/2e64843f595c42e991457cb76fcfa769)](https://www.codacy.com/gh/Unidata/MetPy/dashboard)
[![Code Climate](https://codeclimate.com/github/Unidata/MetPy/badges/gpa.svg)](https://codeclimate.com/github/Unidata/MetPy)

MetPy is a collection of tools in Python for reading, visualizing and
performing calculations with weather data.

MetPy follows [semantic versioning](https://semver.org) in its version number. This means
that any MetPy ``1.x`` release will be backwards compatible with an earlier ``1.y`` release. By
"backward compatible", we mean that **correct** code that works on a ``1.y`` version will work
on a future ``1.x`` version.

For additional MetPy examples not included in this repository, please see the [Unidata Python
Gallery](https://unidata.github.io/python-gallery/).

We support Python >= 3.9.

Need Help?
----------

Need help using MetPy? Found an issue? Have a feature request? Checkout our
[support page](https://github.com/Unidata/MetPy/blob/main/SUPPORT.md).

Important Links
---------------

- [HTML Documentation](http://unidata.github.io/MetPy)
- [Unidata Python Gallery](https://unidata.github.io/python-gallery/)
- "metpy" tagged questions on [Stack Overflow](https://stackoverflow.com/questions/tagged/metpy)
- [Gitter chat room](https://gitter.im/Unidata/MetPy)

Dependencies
------------

Other required packages:

- Numpy
- Scipy
- Matplotlib
- Pandas
- Pint
- Xarray

There is also an optional dependency on the pyproj library for geographic
projections (used with cross sections, grid spacing calculation, and the GiniFile interface).

See the [installation guide](https://unidata.github.io/MetPy/latest/userguide/installguide.html)
for more information.

Code of Conduct
---------------

We want everyone to feel welcome to contribute to MetPy and participate in discussions. In that
spirit please have a look at our [Code of Conduct](https://github.com/Unidata/MetPy/blob/main/CODE_OF_CONDUCT.md).

Contributing
------------

**Imposter syndrome disclaimer**: We want your help. No, really.

There may be a little voice inside your head that is telling you that you're not ready to be
an open source contributor; that your skills aren't nearly good enough to contribute. What
could you possibly offer a project like this one?

We assure you - the little voice in your head is wrong. If you can write code at all,
you can contribute code to open source. Contributing to open source projects is a fantastic
way to advance one's coding skills. Writing perfect code isn't the measure of a good developer
(that would disqualify all of us!); it's trying to create something, making mistakes, and
learning from those mistakes. That's how we all improve, and we are happy to help others learn.

Being an open source contributor doesn't just mean writing code, either. You can help out by
writing documentation, tests, or even giving feedback about the project (and yes - that
includes giving feedback about the contribution process). Some of these contributions may be
the most valuable to the project as a whole, because you're coming to the project with fresh
eyes, so you can see the errors and assumptions that seasoned contributors have glossed over.

For more information, please read the see the [contributing guide](https://github.com/Unidata/MetPy/blob/main/CONTRIBUTING.md).

Philosophy
----------

The space MetPy aims for is GEMPAK (and maybe NCL)-like functionality, in a way that plugs
easily into the existing scientific Python ecosystem (numpy, scipy, matplotlib). So, if you
take the average GEMPAK script for a weather map, you need to:

- read data
- calculate a derived field
- show on a map/skew-T

One of the benefits hoped to achieve over GEMPAK is to make it easier to use these routines for
any meteorological Python application; this means making it easy to pull out the LCL
calculation and just use that, or reuse the Skew-T with your own data code. MetPy also prides
itself on being well-documented and well-tested, so that on-going maintenance is easily
manageable.

The intended audience is that of GEMPAK: researchers, educators, and any one wanting to script
up weather analysis. It doesn't even have to be scripting; all python meteorology tools are
hoped to be able to benefit from MetPy. Conversely, it's hoped to be the meteorological
equivalent of the audience of scipy/scikit-learn/skimage.


^ Please read ^

    Fast to learn, fast to code, fast to run

![Test](https://github.com/vitalik/django-ninja/actions/workflows/test_full.yml/badge.svg)
![Coverage](https://img.shields.io/codecov/c/github/vitalik/django-ninja)
[![PyPI version](https://badge.fury.io/py/django-ninja.svg)](https://badge.fury.io/py/django-ninja)
[![Downloads](https://static.pepy.tech/personalized-badge/django-ninja?period=month&units=international_system&left_color=black&right_color=brightgreen&left_text=downloads/month)](https://pepy.tech/project/django-ninja)

# Django Ninja - Fast Django REST Framework

# v1.0 What's new
Read more details here - https://django-ninja.dev/whatsnew_v1/

Or Watch here:

**Django Ninja** is a web framework for building APIs with **Django** and Python 3.6+ **type hints**.

 **Key features:**

  - **Easy**: Designed to be easy to use and intuitive.
  - **FAST execution**: Very high performance thanks to **Pydantic** and **async support**.
  - **Fast to code**: Type hints and automatic docs lets you focus only on business logic.
  - **Standards-based**: Based on the open standards for APIs: **OpenAPI** (previously known as Swagger) and **JSON Schema**.
  - **Django friendly**: (obviously) has good integration with the Django core and ORM.
  - **Production ready**: Used by multiple companies on live projects (If you use django-ninja and would like to publish your feedback, please email ppr.vitaly@gmail.com).

![Django Ninja REST Framework](docs/docs/img/benchmark.png)

**Documentation**: https://django-ninja.dev

---

## Installation

```
pip install django-ninja
```

## Usage

In your django project next to urls.py create new `api.py` file:

```Python
from ninja import NinjaAPI

api = NinjaAPI()

@api.get("/add")
def add(request, a: int, b: int):
    return {"result": a + b}
```

Now go to `urls.py` and add the following:

```Python hl_lines="3 7"
...
from .api import api

urlpatterns = [
    path("admin/", admin.site.urls),
    path("api/", api.urls),  # <---------- !
]
```

**That's it !**

Now you've just created an API that:

 - receives an HTTP GET request at `/api/add`
 - takes, validates and type-casts GET parameters `a` and `b`
 - decodes the result to JSON
 - generates an OpenAPI schema for defined operation

### Interactive API docs

Now go to http://127.0.0.1:8000/api/docs

You will see the automatic interactive API documentation (provided by Swagger UI or Redoc):

![Swagger UI](docs/docs/img/index-swagger-ui.png)

## What next?

 - Read the full documentation here - https://django-ninja.dev
 - To support this project, please give star it on Github. ![github star](docs/docs/img/github-star.png)
 - Share it [via Twitter](https://twitter.com/intent/tweet?text=Check%20out%20Django%20Ninja%20-%20Fast%20Django%20REST%20Framework%20-%20https%3A%2F%2Fdjango-ninja.dev)
 - If you already using django-ninja, please share your feedback to ppr.vitaly@gmail.com


> ## THIS PROJECT IS OPEN FOR MAINTAINERS
> Development on the python-social-auth projects has been stagnated for a while,
> #445 was open a long time ago to discuss this matter and a plan (failed) was
> presented to fix the situation. For that reason, I'm opening the organization to
> new maintainers that will have the proper permissions to unstuck development.
>
> Those willing to join, contact me by email with the subject `[PSA Maintainer]
> <your name>` and please let me know what motivates you to join in such role.

# Python Social Auth - Core
![Build Status](https://github.com/python-social-auth/social-core/workflows/Flake8/badge.svg)
![Build Status](https://github.com/python-social-auth/social-core/workflows/Tests/badge.svg)
[![Build Status](https://travis-ci.org/python-social-auth/social-core.svg?branch=master)](https://travis-ci.org/python-social-auth/social-core)
[![PyPI version](https://badge.fury.io/py/social-auth-core.svg)](https://badge.fury.io/py/social-auth-core)
[![Donate](https://img.shields.io/badge/Donate-PayPal-orange.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=matiasaguirre%40gmail%2ecom&lc=US&item_name=Python%20Social%20Auth&no_note=0&currency_code=USD&bn=PP%2dDonationsBF%3abtn_donate_SM%2egif%3aNonHostedGuest)

Python Social Auth is an easy to setup social authentication/registration
mechanism with support for several frameworks and auth providers.

## Description

This is the core component of the python-social-auth ecosystem, it
implements the common interface to define new authentication backends
to third parties services, implement integrations with web frameworks
and storage solutions.

## Documentation

Project documentation is available at http://python-social-auth.readthedocs.org/.

## Setup

```shell
$ pip install social-auth-core
```

## Contributing

See the [CONTRIBUTING.md](CONTRIBUTING.md) document for details.

## Versioning

This project follows [Semantic Versioning 2.0.0](http://semver.org/spec/v2.0.0.html).

## License

This project follows the BSD license. See the [LICENSE](LICENSE) for details.

## Donations

This project is maintained on my spare time, consider donating to keep
it improving.

[![Donate](https://img.shields.io/badge/Donate-PayPal-orange.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=matiasaguirre%40gmail%2ecom&lc=US&item_name=Python%20Social%20Auth&no_note=0&currency_code=USD&bn=PP%2dDonationsBF%3abtn_donate_SM%2egif%3aNonHostedGuest)


# Intake: Take 2

**A general python package for describing, loading and processing data**

![Logo](https://github.com/intake/intake/raw/master/logo-small.png)

[![Build Status](https://github.com/intake/intake/workflows/CI/badge.svg)](https://github.com/intake/intake/actions)
[![Documentation Status](https://readthedocs.org/projects/intake/badge/?version=latest)](http://intake.readthedocs.io/en/latest/?badge=latest)

*Taking the pain out of data access and distribution*

Intake is an open-source package to:

- describe your data declaratively
- gather data sets into catalogs
- search catalogs and services to find the right data you need
- load, transform and output data in many formats
- work with third party remote storage and compute platforms

Documentation is available at [Read the Docs](http://intake.readthedocs.io/en/latest).

Please report issues at https://github.com/intake/intake/issues

Install
-------

Recommended method using conda:
```bash
conda install -c conda-forge intake
```

You can also install using `pip`, in which case you have a choice as to how many of the optional
dependencies you install, with the simplest having least requirements

```bash
pip install intake
```

Note that you may well need specific drivers and other plugins, which usually have additional
dependencies of their own.

Development
-----------
 * Create development Python environment with the required dependencies, ideally with `conda`.
   The requirements can be found in the yml files in the `scripts/ci/` directory of this repo.
   * e.g. `conda env create -f scripts/ci/environment-py311.yml` and then `conda activate test_env`
 * Install intake using `pip install -e .`
 * Use `pytest` to run tests.
 * Create a fork on github to be able to submit PRs.
 * We respect, but do not enforce, pep8 standards; all new code should be covered by tests.


# Datadog Integrations - Core

| | |
| --- | --- |
| CI/CD | [![CI - Test][1]][2] [![CI - Coverage][17]][18] |
| Docs | [![Docs - Release][19]][20] |
| Meta | [![Hatch project][26]][27] [![Linting - Ruff][24]][25] [![Code style - black][21]][22] [![Typing - Mypy][28]][29] [![License - BSD-3-Clause][30]][31] |

This repository contains open source integrations that Datadog officially develops and supports.
To add a new integration, please see the [Integrations Extras][5] repository and the
[accompanying documentation][6].

The [Datadog Agent][7] packages are equipped with all the Agent integrations from this
repository, so to get started using them, you can simply [install the Agent][8]
for your operating system. The [AGENT_CHANGELOG](AGENT_CHANGELOG.md) file shows
which Integrations have been updated in each Agent version.

## Contributing

Working with integrations is easy, the main page of the [development docs][6]
contains all the info you need to get your dev environment up and running in minutes
to run, test and build a Check. More advanced documentation can be found [here][3].

## Reporting Issues

For more information on integrations, please reference our [documentation][11] and
[knowledge base][12]. You can also visit our [help page][13] to connect with us.

[1]: https://raw.githubusercontent.com/DataDog/integrations-core/badges/test-results.svg
[2]: https://github.com/DataDog/integrations-core/actions/workflows/master.yml
[3]: https://datadoghq.dev/integrations-core/
[5]: https://github.com/DataDog/integrations-extras
[6]: https://docs.datadoghq.com/developers/integrations/
[7]: https://github.com/DataDog/datadog-agent
[8]: https://app.datadoghq.com/account/settings/agent/latest
[9]: https://docs.pytest.org/en/latest/
[10]: https://packaging.python.org/tutorials/distributing-packages/
[11]: https://docs.datadoghq.com
[12]: https://help.datadoghq.com/hc/en-us
[13]: https://docs.datadoghq.com/help/
[15]: https://github.com/DataDog/integrations-core/blob/6.2.1/requirements-integration-core.txt
[16]: https://github.com/DataDog/integrations-core/blob/ea2dfbf1e8859333af4c8db50553eb72a3b466f9/requirements-agent-release.txt
[17]: https://codecov.io/github/DataDog/integrations-core/coverage.svg?branch=master
[18]: https://codecov.io/github/DataDog/integrations-core?branch=master
[19]: https://github.com/DataDog/integrations-core/workflows/docs/badge.svg
[20]: https://github.com/DataDog/integrations-core/actions?workflow=docs
[21]: https://img.shields.io/badge/code%20style-black-000000.svg
[22]: https://github.com/ambv/black
[24]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v0.json
[25]: https://github.com/charliermarsh/ruff
[26]: https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg
[27]: https://github.com/pypa/hatch
[28]: https://img.shields.io/badge/typing-Mypy-blue.svg
[29]: https://github.com/python/mypy
[30]: https://img.shields.io/badge/license-BSD--3--Clause-9400d3.svg
[31]: https://spdx.org/licenses/BSD-3-Clause.html


.. image:: https://circleci.com/gh/mozilla/ichnaea.svg?style=svg
    :target: https://circleci.com/gh/mozilla/ichnaea

=======
Ichnaea
=======

``Ichnaea`` is an application that provides geolocation coordinates
from other sources of data (Bluetooth, cell or WiFi networks, GeoIP, etc.).

For more information look at
`the full docs <https://ichnaea.readthedocs.io/>`_.

Please use `our github tracker <https://github.com/mozilla/ichnaea/issues>`_
to report issues.

License
=======

``ichnaea`` is offered under the Apache License 2.0.


# OpenTelemetry Python
[![Slack](https://img.shields.io/badge/slack-@cncf/otel/python-brightgreen.svg?logo=slack)](https://cloud-native.slack.com/archives/C01PD4HUVBL)
[![Build Status](https://github.com/open-telemetry/opentelemetry-python/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/open-telemetry/opentelemetry-python/actions)
[![Minimum Python Version](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Release](https://img.shields.io/github/v/release/open-telemetry/opentelemetry-python?include_prereleases&style=)](https://github.com/open-telemetry/opentelemetry-python/releases/)
[![Read the Docs](https://readthedocs.org/projects/opentelemetry-python/badge/?version=latest)](https://opentelemetry-python.readthedocs.io/en/latest/)

## Project Status

See the [OpenTelemetry Instrumentation for Python](https://opentelemetry.io/docs/instrumentation/python/#status-and-releases).

| Signal  | Status       | Project |
| ------- | ------------ | ------- |
| Traces  | Stable       | N/A     |
| Metrics | Stable       | N/A     |
| Logs    | Experimental | N/A     |

Project versioning information and stability guarantees can be found [here](./rationale.md#versioning-and-releasing).

## Getting started

You can find the getting started guide for OpenTelemetry Python [here](https://opentelemetry.io/docs/instrumentation/python/getting-started/).

If you are looking for **examples** on how to use the OpenTelemetry API to
instrument your code manually, or how to set up the OpenTelemetry
Python SDK, see https://opentelemetry.io/docs/instrumentation/python/manual/.

## Python Version Support

This project ensures compatibility with the current supported versions of the Python. As new Python versions are released, support for them is added and
as old Python versions reach their end of life, support for them is removed.

We add support for new Python versions no later than 3 months after they become stable.

We remove support for old Python versions 6 months after they reach their [end of life](https://devguide.python.org/devcycle/#end-of-life-branches).

## Documentation

The online documentation is available at https://opentelemetry-python.readthedocs.io/.
To access the latest version of the documentation, see
https://opentelemetry-python.readthedocs.io/en/latest/.

## Install

This repository includes multiple installable packages. The `opentelemetry-api`
package includes abstract classes and no-op implementations that comprise the OpenTelemetry API following the
[OpenTelemetry specification](https://github.com/open-telemetry/opentelemetry-specification).
The `opentelemetry-sdk` package is the reference implementation of the API.

Libraries that produce telemetry data should only depend on `opentelemetry-api`,
and defer the choice of the SDK to the application developer. Applications may
depend on `opentelemetry-sdk` or another package that implements the API.

The API and SDK packages are available on the Python Package Index (PyPI). You can install them via `pip` with the following commands:

```sh
pip install opentelemetry-api
pip install opentelemetry-sdk
```

The
[`exporter/`](https://github.com/open-telemetry/opentelemetry-python/tree/main/exporter)
directory includes OpenTelemetry exporter packages. You can install the packages separately with the following command:

```sh
pip install opentelemetry-exporter-{exporter}
```

The
[`propagator/`](https://github.com/open-telemetry/opentelemetry-python/tree/main/propagator)
directory includes OpenTelemetry propagator packages. You can install the packages separately with the following command:

```sh
pip install opentelemetry-propagator-{propagator}
```

To install the development versions of these packages instead, clone or fork
this repository and perform an [editable
install](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs):

```sh
pip install -e ./opentelemetry-api
pip install -e ./opentelemetry-sdk
pip install -e ./instrumentation/opentelemetry-instrumentation-{instrumentation}
```

For additional exporter and instrumentation packages, see the
[`opentelemetry-python-contrib`](https://github.com/open-telemetry/opentelemetry-python-contrib) repository.

## Contributing

For information about contributing to OpenTelemetry Python, see [CONTRIBUTING.md](CONTRIBUTING.md).

We meet weekly on Thursdays at 9AM PST. The meeting is subject to change depending on contributors' availability. Check the [OpenTelemetry community calendar](https://calendar.google.com/calendar/embed?src=google.com_b79e3e90j7bbsa2n2p5an5lf60%40group.calendar.google.com) for specific dates and Zoom meeting links.

Meeting notes are available as a public [Google doc](https://docs.google.com/document/d/1CIMGoIOZ-c3-igzbd6_Pnxx1SjAkjwqoYSUWxPY8XIs/edit).

Approvers ([@open-telemetry/python-approvers](https://github.com/orgs/open-telemetry/teams/python-approvers)):

- [Aaron Abbott](https://github.com/aabmass), Google
- [Jeremy Voss](https://github.com/jeremydvoss), Microsoft
- [Owais Lone](https://github.com/owais), Splunk
- [Pablo Collins](https://github.com/pmcollins), Splunk
- [Riccardo Magliocchetti](https://github.com/xrmx), Elastic
- [Shalev Roda](https://github.com/shalevr), Cisco
- [Srikanth Chekuri](https://github.com/srikanthccv), signoz.io
- [Tammy Baylis](https://github.com/tammy-baylis-swi), SolarWinds

Emeritus Approvers

- [Ashutosh Goel](https://github.com/ashu658), Cisco
- [Carlos Alberto Cortez](https://github.com/carlosalberto), Lightstep
- [Christian Neumüller](https://github.com/Oberon00), Dynatrace
- [Héctor Hernández](https://github.com/hectorhdzg), Microsoft
- [Mauricio Vásquez](https://github.com/mauriciovasquezbernal), Kinvolk
- [Nathaniel Ruiz Nowell](https://github.com/NathanielRN), AWS
- [Nikolay Sokolik](https://github.com/oxeye-nikolay), Oxeye
- [Sanket Mehta](https://github.com/sanketmehta28), Cisco
- [Tahir H. Butt](https://github.com/majorgreys), DataDog

*For more information about the approver role, see the [community repository](https://github.com/open-telemetry/community/blob/main/community-membership.md#approver).*

Maintainers ([@open-telemetry/python-maintainers](https://github.com/orgs/open-telemetry/teams/python-maintainers)):

- [Diego Hurtado](https://github.com/ocelotl), Lightstep
- [Leighton Chen](https://github.com/lzchen), Microsoft

Emeritus Maintainers:

- [Alex Boten](https://github.com/codeboten), Lightstep
- [Chris Kleinknecht](https://github.com/c24t), Google
- [Owais Lone](https://github.com/owais), Splunk
- [Reiley Yang](https://github.com/reyang), Microsoft
- [Srikanth Chekuri](https://github.com/srikanthccv), signoz.io
- [Yusuke Tsutsumi](https://github.com/toumorokoshi), Google

*For more information about the maintainer role, see the [community repository](https://github.com/open-telemetry/community/blob/main/community-membership.md#maintainer).*

### Thanks to all the people who already contributed!


**********
MushroomRL
**********

.. image:: https://github.com/MushroomRL/mushroom-rl/actions/workflows/continuous_integration.yml/badge.svg?branch=dev
   :target: https://github.com/MushroomRL/mushroom-rl/actions/workflows/continuous_integration.yml
   :alt: Continuous Integration

.. image:: https://readthedocs.org/projects/mushroomrl/badge/?version=latest
   :target: https://mushroomrl.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status

.. image:: https://api.codeclimate.com/v1/badges/3b0e7167358a661ed882/maintainability
   :target: https://codeclimate.com/github/MushroomRL/mushroom-rl/maintainability
   :alt: Maintainability

.. image:: https://api.codeclimate.com/v1/badges/3b0e7167358a661ed882/test_coverage
   :target: https://codeclimate.com/github/MushroomRL/mushroom-rl/test_coverage
   :alt: Test Coverage

**MushroomRL: Reinforcement Learning Python library.**

.. contents:: **Contents of this document:**
   :depth: 2

What is MushroomRL
==================
MushroomRL is a Python Reinforcement Learning (RL) library whose modularity allows
to easily use well-known Python libraries for tensor computation (e.g. PyTorch,
Tensorflow) and RL benchmarks (e.g. OpenAI Gym, PyBullet, Deepmind Control Suite).
It allows to perform RL experiments in a simple way providing classical RL algorithms
(e.g. Q-Learning, SARSA, FQI), and deep RL algorithms (e.g. DQN, DDPG, SAC, TD3,
TRPO, PPO).

`Full documentation and tutorials available here <http://mushroomrl.readthedocs.io/en/latest/>`_.

Installation
============

You can do a minimal installation of ``MushroomRL`` with:

.. code:: shell

    pip3 install mushroom_rl

Installing everything
---------------------
``MushroomRL`` contains also some optional components e.g., support for ``OpenAI Gym``
environments, Atari 2600 games from the ``Arcade Learning Environment``, and the support
for physics simulators such as ``Pybullet`` and ``MuJoCo``.
Support for these classes is not enabled by default.

To install the whole set of features, you will need additional packages installed.
You can install everything by running:

.. code:: shell

    pip3 install mushroom_rl[all]

This will install every dependency of MushroomRL, except the Plots dependency.
For ubuntu>20.04, you may need to install pygame and gym dependencies:

.. code:: shell

    sudo apt -y install libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev \
                     libsdl1.2-dev libsmpeg-dev libportmidi-dev ffmpeg libswscale-dev \
                     libavformat-dev libavcodec-dev swig

Notice that you still need to install some of these dependencies for different operating systems, e.g. swig for macOS

Below is the code that you need to run to install the Plots dependencies:

.. code:: shell

    sudo apt -y install python3-pyqt5
    pip3 install mushroom_rl[plots]

You might need to install external dependencies first. For more information about mujoco-py
installation follow the instructions on the `project page <https://github.com/openai/mujoco-py>`_

    WARNING! when using conda, there may be issues with QT. You can fix them by adding the following lines to the code, replacing ``<conda_base_path>`` with the path to your conda distribution and ``<env_name>`` with the name of the conda environment you are using:

.. code:: python

   import os
   os.environ['QT_QPA_PLATFORM_PLUGIN_PATH'] = '<conda_base_path>/envs/<env_name>/bin/platforms'

To use dm_control MushroomRL interface, install ``dm_control`` following the instruction that can
be found `here <https://github.com/deepmind/dm_control>`_

Using Habitat and iGibson with MushroomRL
-----------------------------------------

`Habitat <https://aihabitat.org/>`__ and `iGibson <http://svl.stanford.edu/igibson/>`__
are simulation platforms providing realistic and sensory-rich learning environments.
In MushroomRL, the agent's default observations are RGB images, but RGBD,
agent sensory data, and other information can also be used.

    If you have previous versions of iGibson or Habitat already installed, we recommend to remove them and do clean installs.

iGibson Installation
^^^^^^^^^^^^^^^^^^^^
Follow the `official guide <http://svl.stanford.edu/igibson/#install_env>`__ and install its
`assets <http://svl.stanford.edu/igibson/docs/assets.html>`__ and
`datasets <http://svl.stanford.edu/igibson/docs/dataset.html>`__.

For ``<MUSHROOM_RL PATH>/mushroom-rl/examples/igibson_dqn.py`` you need to run

.. code:: shell

    python -m igibson.utils.assets_utils --download_assets
    python -m igibson.utils.assets_utils --download_demo_data
    python -m igibson.utils.assets_utils --download_ig_dataset

You can also use `third party datasets <https://github.com/StanfordVL/iGibson/tree/master/igibson/utils/data_utils/ext_scene>`__.

The scene details are defined in a YAML file, that needs to be passed to the agent.
See ``<IGIBSON PATH>/igibson/test/test_house.YAML`` for an example.

Habitat Installation
^^^^^^^^^^^^^^^^^^^^
Follow the `official guide <https://github.com/facebookresearch/habitat-lab/#installation>`__
and do a **full install** with `habitat_baselines`.
Then you can download interactive datasets following
`this <https://github.com/facebookresearch/habitat-lab#data>`__ and
`this <https://github.com/facebookresearch/habitat-lab#task-datasets>`__.
If you need to download other datasets, you can use
`this utility <https://github.com/facebookresearch/habitat-sim/blob/master/habitat_sim/utils/datasets_download.py>`__.

Basic Usage of Habitat
^^^^^^^^^^^^^^^^^^^^^^
When you create a ``Habitat`` environment, you need to pass a wrapper name and two
YAML files: ``Habitat(wrapper, config_file, base_config_file)``.

* The wrapper has to be among the ones defined in ``<MUSHROOM_RL PATH>/mushroom-rl/environments/habitat_env.py``,
  and takes care of converting actions and observations in a gym-like format. If your task / robot requires it,
  you may need to define new wrappers.

* The YAML files define every detail: the Habitat environment, the scene, the
  sensors available to the robot, the rewards, the action discretization, and any
  additional information you may need. The second YAML file is optional, and
  overwrites whatever was already defined in the first YAML.

    If you use YAMLs from ``habitat-lab``, check if they define a YAML for
    ``BASE_TASK_CONFIG_PATH``. If they do, you need to pass it as ``base_config_file`` to
    ``Habitat()``. ``habitat-lab`` YAMLs, in fact, use relative paths, and calling them
    from outside its root folder will cause errors.

* If you use a dataset, be sure that the path defined in the YAML file is correct,
  especially if you use relative paths. ``habitat-lab`` YAMLs use relative paths, so
  be careful with that. By default, the path defined in the YAML file will be
  relative to where you launched the python code. If your `data` folder is
  somewhere else, you may also create a symbolic link.

Rearrange Task Example
^^^^^^^^^^^^^^^^^^^^^^
* Download the ReplicaCAD datasets (``--data-path data`` downloads them in the folder
  from where you are launching your code)

.. code:: shell

    python -m habitat_sim.utils.datasets_download --uids replica_cad_dataset --data-path data

* For this task we use ``<HABITAT_LAB PATH>/habitat_baselines/config/rearrange/rl_pick.yaml``.
  This YAML defines ``BASE_TASK_CONFIG_PATH: configs/tasks/rearrange/pick.yaml``,
  and since this is a relative path we need to overwrite it by passing its absolute path
  as ``base_config_file`` argument to ``Habitat()``.

* Then, ``pick.yaml`` defines the dataset to be used with respect to ``<HABITAT_LAB PATH>``.
  If you have not used ``--data-path`` argument with the previous download command,
  the ReplicaCAD datasets is now in ``<HABITAT_LAB PATH>/data`` and you need to
  make a link to it

.. code:: shell

    ln -s <HABITAT_LAB PATH>/data/ <MUSHROOM_RL PATH>/mushroom-rl/examples/habitat

* Finally, you can launch ``python habitat_rearrange_sac.py``.

Navigation Task Example
^^^^^^^^^^^^^^^^^^^^^^^
* Download and extract Replica scenes

    WARNING! The dataset is very large!

.. code:: shell

    sudo apt-get install pigz
    git clone https://github.com/facebookresearch/Replica-Dataset.git
    cd Replica-Dataset
    ./download.sh replica-path

* For this task we only use the custom YAML file ``pointnav_apartment-0.yaml``.

* ``DATA_PATH: "replica_{split}_apartment-0.json.gz"`` defines the JSON file with
  some scene details, such as the agent's initial position and orientation.
  The ``{split}`` value is defined in the ``SPLIT`` key.

    If you want to try new positions, you can sample some from the set of the scene's navigable points.
    After initializing a ``habitat`` environment, for example ``mdp = Habitat(...)``,
    run ``mdp.env._env._sim.sample_navigable_point()``.

* ``SCENES_DIR: "Replica-Dataset/replica-path/apartment_0"`` defines the scene.
  As said before, this path is relative to where you launch the script, thus we need to link the Replica folder.
  If you launch ``habitat_nav_dqn.py`` from its example folder, run

.. code:: shell

    ln -s <PATH TO>/Replica-Dataset/ <MUSHROOM_RL PATH>/mushroom-rl/examples/habitat

* Finally, you can launch ``python habitat_nav_dqn.py``.

Editable Installation
---------------------

You can also perform a local editable installation by using:

.. code:: shell

    pip install --no-use-pep517 -e .

To install also optional dependencies:

.. code:: shell

    pip install --no-use-pep517 -e .[all]

How to set and run and experiment
=================================
To run experiments, MushroomRL requires a script file that provides the necessary information
for the experiment. Follow the scripts in the "examples" folder to have an idea
of how an experiment can be run.

For instance, to run a quick experiment with one of the provided example scripts, run:

.. code:: shell

    python3 examples/car_on_hill_fqi.py

Cite MushroomRL
===============
If you are using MushroomRL for your scientific publications, please cite:

.. code:: bibtex

    @article{JMLR:v22:18-056,
        author  = {Carlo D'Eramo and Davide Tateo and Andrea Bonarini and Marcello Restelli and Jan Peters},
        title   = {MushroomRL: Simplifying Reinforcement Learning Research},
        journal = {Journal of Machine Learning Research},
        year    = {2021},
        volume  = {22},
        number  = {131},
        pages   = {1-5},
        url     = {http://jmlr.org/papers/v22/18-056.html}
    }

How to contact us
=================
For any question, drop an e-mail at mushroom4rl@gmail.com.

Follow us on Twitter `@Mushroom_RL <https://twitter.com/mushroom_rl>`_!


# fastavro
[![Build Status](https://github.com/fastavro/fastavro/workflows/Build/badge.svg)](https://github.com/fastavro/fastavro/actions)
[![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest)
[![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python `avro` package is written in pure Python, it is
relatively slow. In one test case, it takes about 14 seconds to iterate through
a file of 10,000 records. By comparison, the JAVA `avro` SDK reads the same file in
1.9 seconds.

The `fastavro` library was written to offer performance comparable to the Java
library. With regular CPython, `fastavro` uses C extensions which allow it to
iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5
seconds (to be fair, the JAVA benchmark is doing some extra JSON
encoding/decoding).

`fastavro` supports the following Python versions:

* Python 3.8
* Python 3.9
* Python 3.10
* Python 3.11
* Python 3.12
* PyPy3

## Supported Features

* File Writer
* File Reader (iterating via records or blocks)
* Schemaless Writer
* Schemaless Reader
* JSON Writer
* JSON Reader
* Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
* Schema resolution
* Aliases
* Logical Types
* Parsing schemas into the canonical form
* Schema fingerprinting

## Missing Features

* Anything involving Avro's RPC features

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing
`fastavro` is available both on [PyPI](http://pypi.python.org/pypi)

    pip install fastavro

and on [conda-forge](https://conda-forge.github.io) `conda` channel.

    conda install -c conda-forge fastavro

# Contributing

* Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
* Get approval from discussing on the GitHub issue before opening the pull request
* Tests must be passing for pull request to be considered

Developer requirements can be installed with `pip install -r developer_requirements.txt`.
If those are installed, you can run the tests with `./run-tests.sh`. If you have trouble
installing those dependencies, you can run `docker build .` to run the tests inside
a Docker container. This won't test on all versions of Python or on PyPy, so it's possible
to still get CI failures after making a pull request, but we can work through those errors
if/when they happen. `.run-tests.sh` only covers the Cython tests. In order to test the
pure Python implementation, comment out `python setup.py build_ext --inplace`
and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this
is this codec tests. If the supporting codec library is not available, the test
will fail. These failures can be ignored since the tests will on pull requests
and will be run in the correct environments with the correct dependencies set up.

### Releasing

We release both to [PyPI][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you've created your own
fork of [fastavro-feedstock][feedstock].

* Make sure the tests pass
* Run `make tag`
* Wait for all artifacts to be built and published the the Github release
* Run `make publish`
* The conda-forge PR should get created and merged automatically

[conda-forge]: https://conda-forge.org/
[feedstock]: https://github.com/conda-forge/fastavro-feedstock
[pypi]: https://pypi.python.org/pypi
[twine]: https://pypi.python.org/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)


# PackageDev - The Sublime Text Meta Package

PackageDev provides syntax highlighting
and other helpful utility for Sublime Text resource files.

Resource files are ways of configuring the Sublime Text text editor
to various extends,
including but not limited to:
custom syntax definitions,
context menus (and the main menu),
and key bindings.

Thus, this package is ideal for package developers,
but even normal users of Sublime Text
who want to configure it to their liking
should find it very useful.

## Installation

Install the package using [Package Control][].
The package's name is PackageDev.

PackageDev was made for **Sublime Text 3**.
An older unmaintained version
can still be installed for Sublime Text 2, however.

[Package Control]: https://packagecontrol.io/

## Getting Started

Syntax highlighting for various resource files
is available immediately.

Various commands,
for example creating a new resource,
are made available
through the command palette (Primary+Shift+P)
with the "PackageDev:" prefix
and in the main menu under *Tools → Packages → Package Development*,

*Note*:
The Primary key refers to
Ctrl on Windows and Linux
and Command (⌘) on macOS.

## Documentation

Detailed documentation is available [at the repository's wiki][wiki].

[wiki]: https://github.com/SublimeText/PackageDev/wiki

## Features Overview

- Syntax definitions for:

  - Build System files
  - Color Scheme files
  - Commands files
  - Completions files
  - Keymap files
  - Macro files
  - Menu files
  - Mousemap files
  - Project files
  - Settings files
  - Snippet files
  - Syntax definition files (both Sublime Text and TextMate)
  - TextMate Preferences (`.tmPreferences`)

- Static and dynamic completions for all the above,
  where sensible.

  Dynamic completions are provided for
  all files expecting command names,
  syntax definitions,
  color schemes,
  and settings files.

- Snippets for some of the above.

  In files holding multiple "entries",
  snippets are available with the trigger `e` and `ee`
  for a short and a long entry, respectively.

- Commands to create (or open) a package
  and various resource files with standard templates.

- Checking for unknown settings keys.

- Hover popups for setting keys with a description and their default value.

- And more!


|logo| Girder |build-status| |docs-status| |license-badge| |codecov-badge|
==========================================================================

**Data Management Platform**

Girder is a free and open source web-based data management platform developed by
`Kitware <https://kitware.com>`_ as part of the `Resonant <http://resonant.kitware.com>`_
data and analytics ecosystem.

|kitware-logo|

Documentation of the Girder platform can be found at
https://girder.readthedocs.io.

For questions, comments, or to get in touch with the maintainers, head to our `Discourse forum <https://discourse.girder.org>`_.

We'd love for you to `contribute to Girder <CONTRIBUTING.rst>`_.

.. |logo| image:: girder/web_client/static/img/Girder_Favicon.png

.. |kitware-logo| image:: https://www.kitware.com/main/wp-content/uploads/2021/10/Kitware-Wordmark.svg
    :target: https://kitware.com
    :alt: Kitware Logo
    :width: 180

.. |build-status| image:: https://circleci.com/gh/girder/girder.png?style=shield
    :target: https://circleci.com/gh/girder/girder
    :alt: Build Status

.. |docs-status| image:: https://readthedocs.org/projects/girder/badge?version=latest
    :target: https://girder.readthedocs.org
    :alt: Documentation Status

.. |license-badge| image:: docs/license.png
    :target: https://pypi.python.org/pypi/girder
    :alt: License

.. |codecov-badge| image:: https://img.shields.io/codecov/c/github/girder/girder.svg
    :target: https://codecov.io/gh/girder/girder
    :alt: Coverage Status


![logo](./doc/_static/logo/keops_logo.png)

---------------------------------------------------------------------------------

[![PyPI version](https://img.shields.io/pypi/v/pykeops?color=blue)](https://pypi.org/project/pykeops/)
[![PyPI downloads](https://pepy.tech/badge/pykeops?color=green)](https://pypi.org/project/pykeops/)
[![CRAN version](https://img.shields.io/cran/v/rkeops?color=yellowgreen)](https://cran.r-project.org/web/packages/rkeops/index.html)
[![CRAN downloads](http://cranlogs.r-pkg.org/badges/grand-total/rkeops?color=yellowgreen)](https://cran.r-project.org/web/packages/rkeops/index.html)

Please visit our [website](https://www.kernel-operations.io/) for
documentation, contribution guidelines and tutorials.

# Kernel Operations on the GPU, with autodiff, without memory overflows

The KeOps library lets you compute reductions of **large arrays**
whose entries are given by a **mathematical formula** or a **neural network**.
It combines **efficient C++ routines** with an **automatic differentiation**
engine and can be used with **Python** (NumPy, PyTorch), **Matlab** and **R**.

It is perfectly suited to the computation of **kernel** matrix-vector products,
**K-nearest neighbors** queries, **N-body** interactions,
point cloud **convolutions** and the associated **gradients**.
Crucially, it performs well even when the corresponding kernel or distance
matrices do *not* fit into the RAM or GPU memory.
Compared with a PyTorch GPU baseline, KeOps provides
a **x10-x100 speed-up** on a wide range of geometric applications,
from kernel methods to geometric deep learning.

![Symbolic matrices](./doc/_static/symbolic_matrix.svg)

**Why using KeOps?**
Math libraries represent most objects as matrices and tensors:

- **(a) Dense matrices.** Variables are often encoded as
  dense numerical arrays Mi,j = M[i,j].
  This representation is convenient and well-supported,
  but also puts a **heavy load** on the memories of our computers.
  Unfortunately, **large arrays are expensive** to move around and
  may not even fit in RAM or GPU memories.

  In practice, this means that a majority of scientific programs are **memory-bound**.
  Run times for most neural networks and mathematical computations
  are not limited by the raw capabilities of our CPUs and CUDA cores,
  but by the **time-consuming transfers** of large arrays
  from memory circuits to arithmetic computing units.

- **(b) Sparse matrices.** To work around this problem, a common solution is to rely on
  sparse matrices: tensors that have few non-zero coefficients.
  We represent these objects using lists of indices
  (in,jn) and values Mn = Min,jn
  that correspond to a small number of non-zero entries.
  Matrix-vector operations are then implemented with
  indexing methods and scattered memory accesses.

  This method is elegant and allows us to represent large arrays
  with a small memory footprint.
  But unfortunately, it **does not stream well on GPUs**:
  parallel computing devices are wired to perform *block-wise* memory accesses
  and have a hard time dealing with lists of *random* indices (in,jn).
  As a consequence, when compared with dense arrays,
  sparse encodings only speed up computations for matrices
  that have **less than 1% non-zero coefficients**.
  This restrictive condition prevents sparse matrices
  from being very useful outside of graph and mesh processing.

- **(c) Symbolic matrices.**
  KeOps provides another solution to **speed up tensor programs**.
  Our key remark is that most of the large arrays that are used
  in machine learning and applied mathematics share a **common mathematical structure**.
  *Distance* matrices, *kernel* matrices, point cloud *convolutions*
  and *attention* layers can all be described as **symbolic** tensors:
  given two collections of vectors (xi) and (yj),
  their coefficients Mi,j at location (i,j) are given by
  **mathematical formulas** F(xi,yj)
  that are evaluated on data samples xi and yj.

  These objects are not "sparse" in the traditional sense...
  but can nevertheless be described efficiently using
  a mathematical formula F and relatively
  small data arrays (xi) and (yj).
  The main purpose of the KeOps library
  is to provide support for this abstraction
  **with all the perks of a deep learning library**:

  - A transparent [interface](https://www.kernel-operations.io/keops/python/api/index.html) with CPU and GPU integration.
  - Numerous [tutorials](https://www.kernel-operations.io/keops/_auto_tutorials/index.html) and [benchmarks](https://www.kernel-operations.io/keops/_auto_benchmarks/index.html).
  - Full support for **automatic differentiation**, **batch processing**
    and approximate computations.

In practice, KeOps symbolic tensors are both
**fast** and **memory-efficient**.
We take advantage of the structure of CUDA registers
to bypass costly memory transfers between
arithmetic and memory circuits. This allows us to provide a
**x10-x100 speed-up** to PyTorch GPU programs
in a wide range of settings.

Using our **Python interface**, a typical sample of code looks like:

```python
# Create two arrays with 3 columns and a (huge) number of lines, on the GPU
import torch  # NumPy, Matlab and R are also supported
M, N, D = 1000000, 2000000, 3
x = torch.randn(M, D, requires_grad=True).cuda()  # x.shape = (1e6, 3)
y = torch.randn(N, D).cuda()                      # y.shape = (2e6, 3)

# Turn our dense Tensors into KeOps symbolic variables with "virtual"
# dimensions at positions 0 and 1 (for "i" and "j" indices):
from pykeops.torch import LazyTensor
x_i = LazyTensor(x.view(M, 1, D))  # x_i.shape = (1e6, 1, 3)
y_j = LazyTensor(y.view(1, N, D))  # y_j.shape = ( 1, 2e6,3)

# We can now perform large-scale computations, without memory overflows:
D_ij = ((x_i - y_j)**2).sum(dim=2)  # Symbolic (1e6,2e6,1) matrix of squared distances
K_ij = (- D_ij).exp()               # Symbolic (1e6,2e6,1) Gaussian kernel matrix

# We come back to vanilla PyTorch Tensors or NumPy arrays using
# reduction operations such as .sum(), .logsumexp() or .argmin()
# on one of the two "symbolic" dimensions 0 and 1.
# Here, the kernel density estimation   a_i = sum_j exp(-|x_i-y_j|^2)
# is computed using a CUDA scheme that has a linear memory footprint and
# outperforms standard PyTorch implementations by two orders of magnitude.
a_i = K_ij.sum(dim=1)  # Genuine torch.cuda.FloatTensor, a_i.shape = (1e6, 1),

# Crucially, KeOps fully supports automatic differentiation!
g_x = torch.autograd.grad((a_i ** 2).sum(), [x])
```

KeOps allows you to **get the most out of your hardware** without compromising on **usability**.
It provides:

- **Linear** (instead of quadratic) **memory footprint** for numerous types of computations.
- Support for a wide range of mathematical **formulas** that can be composed at will.
- Seamless computation of **derivatives** and **gradients**, up to arbitrary orders.
- Sum, LogSumExp, Min, Max but also ArgMin, ArgMax or K-min **reductions**.
- A **conjugate gradient solver** for large-scale spline interpolation and Gaussian process regression.
- Transparent integration with **standard packages**, such as the
  [SciPy solvers](https://docs.scipy.org/doc/scipy/reference/sparse.linalg.html) for linear algebra.
- An interface for **block-sparse** and coarse-to-fine strategies.
- Support for **multi GPU** configurations.

More details are provided below:

- [Installation](http://www.kernel-operations.io/keops/introduction/installation.html)
- [Documentation](http://www.kernel-operations.io/keops/introduction/why_using_keops.html)
- [Learning KeOps with tutorials](http://www.kernel-operations.io/keops/_auto_tutorials/index.html)
- [Gallery of examples](http://www.kernel-operations.io/keops/_auto_examples/index.html)
- [Benchmarks](http://www.kernel-operations.io/keops/_auto_benchmarks/index.html)

# Projects using KeOps

**Symbolic** matrices are to **geometric** learning what **sparse** matrices are to **graph** processing.
KeOps can thus be used in a wide range of settings,
from **shape analysis** (registration, geometric deep learning, optimal transport...)
to **machine learning** (kernel methods, k-means, UMAP...),
**Gaussian processes**, computational **biology** and **physics**.

KeOps provides core routines for the following projects and libraries:

- [GPyTorch](https://docs.gpytorch.ai/en/stable/examples/02_Scalable_Exact_GPs/KeOps_GP_Regression.html>)
  (from the universities of Cornell, Columbia, Pennsylvania) and
  [Falkon](https://falkonml.github.io/falkon/)
  (from the university of Genoa and the [Sierra](https://www.di.ens.fr/sierra/) Inria team),
  two libraries for **Gaussian Process regression** that now scale
  up to **billion-scale datasets**.

- [Deformetrica](http://www.deformetrica.org), a **computational anatomy** software
  from the [Aramis](https://www.inria.fr/en/teams/aramis) Inria team.

- The [Gudhi library](https://gudhi.inria.fr) for **topological data analysis**
  and higher dimensional geometry understanding,
  from the [DataShape](https://team.inria.fr/datashape/) Inria team.

- [GeomLoss](http://www.kernel-operations.io/geomloss), a PyTorch package for Chamfer (Hausdorff) distances, Kernel (Sobolev) divergences and **Earth Mover's (Wasserstein) distances**.
  It provides **optimal transport** solvers that scale up to
  **millions of samples** in seconds.

- The [deep graph matching consensus](https://github.com/rusty1s/deep-graph-matching-consensus) module,
  for learning and refining structural correspondences between graphs.

- [FshapesTk](https://plmlab.math.cnrs.fr/benjamin.charlier/fshapesTk) and the
  [Shapes toolbox](https://plmlab.math.cnrs.fr/jeanfeydy/shapes_toolbox),
  two research-oriented [LDDMM](https://en.wikipedia.org/wiki/Large_deformation_diffeomorphic_metric_mapping) toolkits.

- [HyenaDNA](https://github.com/HazyResearch/hyena-dna/) for parallel computations of the Vandermonde matrix multiplication kernel and reductions used in the S4D kernel.

# Licensing, citation, academic use

This library is licensed under the permissive [MIT license](https://en.wikipedia.org/wiki/MIT_License),
which is fully compatible with both **academic** and **commercial** applications.

If you use this code in a research paper, **please cite** our
[original publication](https://jmlr.org/papers/v22/20-275.html):

> Charlier, B., Feydy, J., Glaunès, J. A., Collin, F.-D. & Durif, G. Kernel Operations on the GPU, with Autodiff, without Memory Overflows. Journal of Machine Learning Research 22, 1–6 (2021).

```tex
@article{JMLR:v22:20-275,
  author  = {Benjamin Charlier and Jean Feydy and Joan Alexis Glaunès and François-David Collin and Ghislain Durif},
  title   = {Kernel Operations on the GPU, with Autodiff, without Memory Overflows},
  journal = {Journal of Machine Learning Research},
  year    = {2021},
  volume  = {22},
  number  = {74},
  pages   = {1-6},
  url     = {http://jmlr.org/papers/v22/20-275.html}
}
```

For applications to **geometric (deep) learning**,
you may also consider our [NeurIPS 2020 paper](https://www.jeanfeydy.com/Papers/KeOps_NeurIPS_2020.pdf):

```tex
@article{feydy2020fast,
    title={Fast geometric learning with symbolic matrices},
    author={Feydy, Jean and Glaun{\`e}s, Joan and Charlier, Benjamin and Bronstein, Michael},
    journal={Advances in Neural Information Processing Systems},
    volume={33},
    year={2020}
}
```

# Authors

Please contact us for any **bug report**, **question** or **feature request** by filing
a report on our [GitHub issue tracker](https://github.com/getkeops/keops/issues)!

**Core library - KeOps, PyKeOps, KeOpsLab:**

- [Benjamin Charlier](https://imag.umontpellier.fr/~charlier/), from the University of Montpellier.
- [Jean Feydy](https://www.jeanfeydy.com), from the [HeKA team](https://team.inria.fr/heka/fr/) (Inria Paris, Inserm, Université Paris Cité).
- [Joan Alexis Glaunès](http://helios.mi.parisdescartes.fr/~glaunes/), from MAP5 laboratory, Université Paris Cité.

**R bindings - RKeOps:**

- [Ghislain Durif](https://gdurif.perso.math.cnrs.fr/), from the University of Montpellier.

**Contributors:**

- [François-David Collin](https://github.com/fradav), from the University of Montpellier: Tensordot operation, CI setup.
- [Tanguy Lefort](https://github.com/tanglef), from the University of Montpellier: conjugate gradient solver.
- [Amélie Vernay](https://github.com/AmelieVernay) and [Chloé Serre-Combe](https://github.com/chloesrcb), from the University of Montpellier: support for LazyTensors in RKeOps.
- [Mauricio Diaz](https://github.com/mdiazmel), from Inria of Paris: CI setup.
- [Benoît Martin](https://github.com/benoitmartin88), from the Aramis Inria team: multi-GPU support.
- [Francis Williams](https://www.fwilliams.info), from New York University: maths operations.
- [Kshiteej Kalambarkar](https://github.com/kshitij12345), from Quansight: maths operations.
- [Hugo Aguettaz](https://github.com/haguettaz), from ETH Zürich: trigonometric functions.
- [D. J. Sutherland](https://djsutherland.ml), from the TTI-Chicago: bug fix in the Python package.
- [David Völgyes](https://scholar.google.no/citations?user=ngT2GvMAAAAJ&hl=en), from the Norwegian Institute of Science and Technology: bug fix in the formula parser.
- [Jean-Baptiste Keck](https://www.keckj.fr/), from the Univeristy Grenoble-Alpes: bug fix in the Python package.

Beyond explicit code contributions, KeOps has grown out of numerous discussions with applied mathematicians and machine learning experts. We would especially like to thank
[Alain Trouvé](https://atrouve.perso.math.cnrs.fr/),
[Stanley Durrleman](https://who.rocq.inria.fr/Stanley.Durrleman/),
[Gabriel Peyré](http://www.gpeyre.com/) and
[Michael Bronstein](https://people.lu.usi.ch/bronstem/)
for their valuable suggestions and financial support.

KeOps was awarded an [open science prize](https://www.enseignementsup-recherche.gouv.fr/fr/remise-des-prix-science-ouverte-du-logiciel-libre-de-la-recherche-2023-93732) by the French Ministry of Higher Education and Research in 2023 ("Espoir - Documentation").


[![PyPI version fury.io](https://badge.fury.io/py/simplemonitor.svg)](https://pypi.python.org/pypi/simplemonitor/) [![Downloads](https://pepy.tech/badge/simplemonitor)](https://pepy.tech/project/simplemonitor)

![Tests (Windows)](https://github.com/jamesoff/simplemonitor/workflows/Tests%20(Windows)/badge.svg) ![Tests (Linux)](https://github.com/jamesoff/simplemonitor/workflows/Tests%20(Linux)/badge.svg) ![Linting](https://github.com/jamesoff/simplemonitor/workflows/Linting/badge.svg)

[![codecov](https://codecov.io/gh/jamesoff/simplemonitor/branch/master/graph/badge.svg)](https://codecov.io/gh/jamesoff/simplemonitor) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

SimpleMonitor is a Python script which monitors hosts and network connectivity. It is designed to be quick and easy to set up and lacks complex features that can make things like Nagios, OpenNMS and Zenoss overkill for a small business or home network. Remote monitor instances can send their results back to a central location.

Requires Python >= 3.6.2.

Documentation is here:
https://simplemonitor.readthedocs.io/en/latest/


# datamodel-code-generator

This code generator creates [pydantic v1 and v2](https://docs.pydantic.dev/) model, [dataclasses.dataclass](https://docs.python.org/3/library/dataclasses.html), [typing.TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict)
and [msgspec.Struct](https://github.com/jcrist/msgspec) from an openapi file and others.

[![PyPI version](https://badge.fury.io/py/datamodel-code-generator.svg)](https://pypi.python.org/pypi/datamodel-code-generator)
[![Conda-forge](https://img.shields.io/conda/v/conda-forge/datamodel-code-generator)](https://anaconda.org/conda-forge/datamodel-code-generator)
[![Downloads](https://pepy.tech/badge/datamodel-code-generator/month)](https://pepy.tech/project/datamodel-code-generator)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/datamodel-code-generator)](https://pypi.python.org/pypi/datamodel-code-generator)
[![codecov](https://codecov.io/gh/koxudaxi/datamodel-code-generator/graph/badge.svg?token=plzSSFb9Li)](https://codecov.io/gh/koxudaxi/datamodel-code-generator)
![license](https://img.shields.io/github/license/koxudaxi/datamodel-code-generator.svg)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Pydantic v1](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v1.json)](https://pydantic.dev)
[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)

## Help
See [documentation](https://koxudaxi.github.io/datamodel-code-generator) for more details.

## Quick Installation

To install `datamodel-code-generator`:
```bash
$ pip install datamodel-code-generator
```

## Simple Usage
You can generate models from a local file.
```bash
$ datamodel-codegen --input api.yaml --output model.py
```

api.yaml

```yaml
openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  license:
    name: MIT
servers:
  - url: http://petstore.swagger.io/v1
paths:
  /pets:
    get:
      summary: List all pets
      operationId: listPets
      tags:
        - pets
      parameters:
        - name: limit
          in: query
          description: How many items to return at one time (max 100)
          required: false
          schema:
            type: integer
            format: int32
      responses:
        '200':
          description: A paged array of pets
          headers:
            x-next:
              description: A link to the next page of responses
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
                x-amazon-apigateway-integration:
                  uri:
                    Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
                  passthroughBehavior: when_no_templates
                  httpMethod: POST
                  type: aws_proxy
    post:
      summary: Create a pet
      operationId: createPets
      tags:
        - pets
      responses:
        '201':
          description: Null response
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
                x-amazon-apigateway-integration:
                  uri:
                    Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
                  passthroughBehavior: when_no_templates
                  httpMethod: POST
                  type: aws_proxy
  /pets/{petId}:
    get:
      summary: Info for a specific pet
      operationId: showPetById
      tags:
        - pets
      parameters:
        - name: petId
          in: path
          required: true
          description: The id of the pet to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Expected response to a valid request
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
    x-amazon-apigateway-integration:
      uri:
        Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
      passthroughBehavior: when_no_templates
      httpMethod: POST
      type: aws_proxy
components:
  schemas:
    Pet:
      required:
        - id
        - name
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        tag:
          type: string
    Pets:
      type: array
      items:
        $ref: "#/components/schemas/Pet"
    Error:
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string
    apis:
      type: array
      items:
        type: object
        properties:
          apiKey:
            type: string
            description: To be used as a dataset parameter value
          apiVersionNumber:
            type: string
            description: To be used as a version parameter value
          apiUrl:
            type: string
            format: uri
            description: "The URL describing the dataset's fields"
          apiDocumentationUrl:
            type: string
            format: uri
            description: A URL to the API console for each API
```

model.py

```python
# generated by datamodel-codegen:
#   filename:  api.yaml
#   timestamp: 2020-06-02T05:28:24+00:00

from __future__ import annotations

from typing import List, Optional

from pydantic import AnyUrl, BaseModel, Field

class Pet(BaseModel):
    id: int
    name: str
    tag: Optional[str] = None

class Pets(BaseModel):
    __root__: List[Pet]

class Error(BaseModel):
    code: int
    message: str

class Api(BaseModel):
    apiKey: Optional[str] = Field(
        None, description='To be used as a dataset parameter value'
    )
    apiVersionNumber: Optional[str] = Field(
        None, description='To be used as a version parameter value'
    )
    apiUrl: Optional[AnyUrl] = Field(
        None, description="The URL describing the dataset's fields"
    )
    apiDocumentationUrl: Optional[AnyUrl] = Field(
        None, description='A URL to the API console for each API'
    )

class Apis(BaseModel):
    __root__: List[Api]
```

## Supported input types
-  OpenAPI 3 (YAML/JSON, [OpenAPI Data Type](https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.2.md#data-types));
-  JSON Schema ([JSON Schema Core](http://json-schema.org/draft/2019-09/json-schema-validation.html)/[JSON Schema Validation](http://json-schema.org/draft/2019-09/json-schema-validation.html));
-  JSON/YAML/CSV Data (it will be converted to JSON Schema);
-  Python dictionary (it will be converted to JSON Schema);
-  GraphQL schema ([GraphQL Schemas and Types](https://graphql.org/learn/schema/));

## Supported output types
- [pydantic](https://docs.pydantic.dev/1.10/).BaseModel;
- [pydantic_v2](https://docs.pydantic.dev/2.0/).BaseModel;
- [dataclasses.dataclass](https://docs.python.org/3/library/dataclasses.html);
- [typing.TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict);
- [msgspec.Struct](https://github.com/jcrist/msgspec);
- Custom type from your [jinja2](https://jinja.palletsprojects.com/en/3.1.x/) template;

## Sponsors

JetBrains

Astral

Datadog, Inc.

## Projects that use datamodel-code-generator

These OSS projects use datamodel-code-generator to generate many models.
See the following linked projects for real world examples and inspiration.

- [airbytehq/airbyte](https://github.com/airbytehq/airbyte)
  - *[code-generator/Dockerfile](https://github.com/airbytehq/airbyte/blob/master/tools/code-generator/Dockerfile)*
- [apache/iceberg](https://github.com/apache/iceberg)
  - *[Generate Python code](https://github.com/apache/iceberg/blob/d2e1094ee0cc6239d43f63ba5114272f59d605d2/open-api/README.md?plain=1#L39)*
    *[`make generate`](https://github.com/apache/iceberg/blob/d2e1094ee0cc6239d43f63ba5114272f59d605d2/open-api/Makefile#L24-L34)*
- [argoproj-labs/hera](https://github.com/argoproj-labs/hera)
  - *[`Makefile`](https://github.com/argoproj-labs/hera/blob/c8cbf0c7a676de57469ca3d6aeacde7a5e84f8b7/Makefile#L53-L62)*
- [awslabs/aws-lambda-powertools-python](https://github.com/awslabs/aws-lambda-powertools-python)
  - *Recommended for [advanced-use-cases](https://awslabs.github.io/aws-lambda-powertools-python/2.6.0/utilities/parser/#advanced-use-cases) in the official documentation*
- [DataDog/integrations-core](https://github.com/DataDog/integrations-core)
  - *[Config models](https://github.com/DataDog/integrations-core/blob/master/docs/developer/meta/config-models.md)*
- [hashintel/hash](https://github.com/hashintel/hash)
  - *[`codegen.sh`](https://github.com/hashintel/hash/blob/9762b1a1937e14f6b387677e4c7fe4a5f3d4a1e1/libs/%40local/hash-graph-client/python/scripts/codegen.sh#L21-L39)*
- [IBM/compliance-trestle](https://github.com/IBM/compliance-trestle)
  - *[Building the models from the OSCAL schemas.](https://github.com/IBM/compliance-trestle/blob/develop/docs/contributing/website.md#building-the-models-from-the-oscal-schemas)*
- [Netflix/consoleme](https://github.com/Netflix/consoleme)
  - *[How do I generate models from the Swagger specification?](https://github.com/Netflix/consoleme/blob/master/docs/gitbook/faq.md#how-do-i-generate-models-from-the-swagger-specification)*
- [Nike-Inc/brickflow](https://github.com/Nike-Inc/brickflow)
  - *[Code generate tools](https://github.com/Nike-Inc/brickflow/blob/e3245bf638588867b831820a6675ada76b2010bf/tools/README.md?plain=1#L8)[`./tools/gen-bundle.sh`](https://github.com/Nike-Inc/brickflow/blob/e3245bf638588867b831820a6675ada76b2010bf/tools/gen-bundle.sh#L15-L22)*
- [open-metadata/OpenMetadata](https://github.com/open-metadata/OpenMetadata)
  - *[Makefile](https://github.com/open-metadata/OpenMetadata/blob/main/Makefile)*
- [PostHog/posthog](https://github.com/PostHog/posthog)
  - *[Generate models via `npm run`](https://github.com/PostHog/posthog/blob/e1a55b9cb38d01225224bebf8f0c1e28faa22399/package.json#L41)*
- [SeldonIO/MLServer](https://github.com/SeldonIO/MLServer)
  - *[generate-types.sh](https://github.com/SeldonIO/MLServer/blob/master/hack/generate-types.sh)*

## Installation

To install `datamodel-code-generator`:
```bash
$ pip install datamodel-code-generator
```

### `http` extra option
If you want to resolve `$ref` for remote files then you should specify `http` extra option.
```bash
$ pip install 'datamodel-code-generator[http]'
```

### `graphql` extra option

If you want to generate data model from a GraphQL schema then you should specify `graphql` extra option.
```bash
$ pip install 'datamodel-code-generator[graphql]'
```

### Docker Image
The docker image is in [Docker Hub](https://hub.docker.com/r/koxudaxi/datamodel-code-generator)
```bash
$ docker pull koxudaxi/datamodel-code-generator
```

## Advanced Uses
You can generate models from a URL.
```bash
$ datamodel-codegen --url https://<INPUT FILE URL> --output model.py
```
This method needs the [http extra option](#http-extra-option)

## All Command Options

The `datamodel-codegen` command:

```bash
usage:
  datamodel-codegen [options]

Generate Python data models from schema definitions or structured data

Options:
  --http-headers HTTP_HEADER [HTTP_HEADER ...]
                        Set headers in HTTP requests to the remote host.
                        (example: "Authorization: Basic dXNlcjpwYXNz")
  --http-ignore-tls     Disable verification of the remote host's TLS
                        certificate
  --http-query-parameters QUERY_PARAMETER [QUERY_PARAMETER ...]
                        Set query parameters in HTTP requests to the remote host.
                        (example: "ref=branch")
  --input INPUT         Input file/directory (default: stdin)
  --input-file-type {auto,openapi,graphql,jsonschema,json,yaml,dict,csv}
                        Input file type (default: auto)
  --output OUTPUT       Output file (default: stdout)
  --output-model-type {pydantic.BaseModel,pydantic_v2.BaseModel,dataclasses.dataclass,typing.TypedDict,msgspec.Struct}
  --url URL             Input file URL. `--input` is ignored when `--url` is
                        used

Typing customization:
  --base-class BASE_CLASS
                        Base Class (default: pydantic.BaseModel)
  --enum-field-as-literal {all,one}
                        Parse enum field as literal. all: all enum field type
                        are Literal. one: field type is Literal when an enum
                        has only one possible value
  --field-constraints   Use field constraints and not con* annotations
  --set-default-enum-member
                        Set enum members as default values for enum field
  --strict-types {str,bytes,int,float,bool} [{str,bytes,int,float,bool} ...]
                        Use strict types
  --use-annotated       Use typing.Annotated for Field(). Also, `--field-
                        constraints` option will be enabled.
  --use-generic-container-types
                        Use generic container types for type hinting
                        (typing.Sequence, typing.Mapping). If `--use-standard-
                        collections` option is set, then import from
                        collections.abc instead of typing
  --use-non-positive-negative-number-constrained-types
                        Use the Non{Positive,Negative}{FloatInt} types instead
                        of the corresponding con* constrained types.
  --use-one-literal-as-default
                        Use one literal as default value for one literal field
  --use-standard-collections
                        Use standard collections for type hinting (list, dict)
  --use-subclass-enum   Define Enum class as subclass with field type when
                        enum has type (int, float, bytes, str)
  --use-union-operator  Use | operator for Union type (PEP 604).
  --use-unique-items-as-set
                        define field type as `set` when the field attribute
                        has `uniqueItems`

Field customization:
  --capitalise-enum-members, --capitalize-enum-members
                        Capitalize field names on enum
  --empty-enum-field-name EMPTY_ENUM_FIELD_NAME
                        Set field name when enum value is empty (default: `_`)
  --field-extra-keys FIELD_EXTRA_KEYS [FIELD_EXTRA_KEYS ...]
                        Add extra keys to field parameters
  --field-extra-keys-without-x-prefix FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX [FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX ...]
                        Add extra keys with `x-` prefix to field parameters.
                        The extra keys are stripped of the `x-` prefix.
  --field-include-all-keys
                        Add all keys to field parameters
  --force-optional      Force optional for required fields
  --original-field-name-delimiter ORIGINAL_FIELD_NAME_DELIMITER
                        Set delimiter to convert to snake case. This option
                        only can be used with --snake-case-field (default: `_`
                        )
  --remove-special-field-name-prefix
                        Remove field name prefix if it has a special meaning
                        e.g. underscores
  --snake-case-field    Change camel-case field name to snake-case
  --special-field-name-prefix SPECIAL_FIELD_NAME_PREFIX
                        Set field name prefix when first character can't be
                        used as Python field name (default: `field`)
  --strip-default-none  Strip default None on fields
  --use-default         Use default value even if a field is required
  --use-default-kwarg   Use `default=` instead of a positional argument for
                        Fields that have default values.
  --use-field-description
                        Use schema description to populate field docstring
  --use-pendulum
                        Use pendulum instead of `datetime` for `date`,
                        `datetime`, and `time` data types

Model customization:
  --allow-extra-fields  Allow to pass extra fields, if this flag is not
                        passed, extra fields are forbidden.
  --allow-population-by-field-name
                        Allow population by field name
  --class-name CLASS_NAME
                        Set class name of root model
  --collapse-root-models
                        Models generated with a root-type field will be
                        mergedinto the models using that root-type model
  --disable-appending-item-suffix
                        Disable appending `Item` suffix to model name in an
                        array
  --disable-timestamp   Disable timestamp on file headers
  --enable-faux-immutability
                        Enable faux immutability
  --enable-version-header
                        Enable package version on file headers
  --keep-model-order    Keep generated models' order
  --reuse-model         Reuse models on the field when a module has the model
                        with the same content
  --target-python-version {3.6,3.7,3.8,3.9,3.10,3.11}
                        target python version (default: 3.7)
  --use-schema-description
                        Use schema description to populate class docstring
  --use-title-as-name   use titles as class names of models
  --use-exact-imports   Import exact types instead of modules, for example:
                        `from .foo import Bar` instead of
                        `from . import foo` with `foo.Bar`

Template customization:
  --aliases ALIASES     Alias mapping file
  --custom-file-header CUSTOM_FILE_HEADER
                        Custom file header
  --custom-file-header-path CUSTOM_FILE_HEADER_PATH
                        Custom file header file path
  --custom-template-dir CUSTOM_TEMPLATE_DIR
                        Custom template directory
  --encoding ENCODING   The encoding of input and output (default: UTF-8)
  --extra-template-data EXTRA_TEMPLATE_DATA
                        Extra template data
  --use-double-quotes   Model generated with double quotes. Single quotes or
                        your black config skip_string_normalization value will
                        be used without this option.
  --wrap-string-literal
                        Wrap string literal by using black `experimental-
                        string-processing` option (require black 20.8b0 or
                        later)
  --additional-imports  Custom imports for output (delimited list input).
                        For example "datetime.date,datetime.datetime"
  --custom-formatters   List of modules with custom formatter (delimited list input).
  --custom-formatters-kwargs A file with kwargs for custom formatters.

OpenAPI-only options:
  --openapi-scopes {schemas,paths,tags,parameters} [{schemas,paths,tags,parameters} ...]
                        Scopes of OpenAPI model generation (default: schemas)
  --strict-nullable     Treat default field as a non-nullable field (Only
                        OpenAPI)
  --use-operation-id-as-name
                        use operation id of OpenAPI as class names of models
  --validation          Deprecated: Enable validation (Only OpenAPI). this
                        option is deprecated. it will be removed in future
                        releases

General options:
  --debug               show debug message (require "debug". `$ pip install 'datamodel-code-generator[debug]'`)
  --disable-warnings    disable warnings
  --no-color            disable colorized output
  --version             show version
  -h, --help            show this help message and exit
```

## Related projects
### fastapi-code-generator
This code generator creates [FastAPI](https://github.com/tiangolo/fastapi) app from an openapi file.

[https://github.com/koxudaxi/fastapi-code-generator](https://github.com/koxudaxi/fastapi-code-generator)

### pydantic-pycharm-plugin
[A JetBrains PyCharm plugin](https://plugins.jetbrains.com/plugin/12861-pydantic) for [`pydantic`](https://github.com/samuelcolvin/pydantic).

[https://github.com/koxudaxi/pydantic-pycharm-plugin](https://github.com/koxudaxi/pydantic-pycharm-plugin)

## PyPi

[https://pypi.org/project/datamodel-code-generator](https://pypi.org/project/datamodel-code-generator)

## Contributing

See `docs/development-contributing.md` for how to get started!

## License

datamodel-code-generator is released under the MIT License. http://www.opensource.org/licenses/mit-license


*pliers*: a python package for automated feature extraction
===========================================================

|PyPI version fury.io| |pytest| |Coverage Status| |docs|
|DOI:10.1145/3097983.3098075|

Pliers is a Python package for automated extraction of features from
multimodal stimuli. It provides a unified, standardized interface to
dozens of different feature extraction tools and services--including
many state-of-the-art deep learning-based models and content analysis
APIs. It's designed to let you rapidly and flexibly extract all kinds of
useful information from videos, images, audio, and text.

You might benefit from pliers if you need to accomplish any of the
following tasks (and many others!):

-  Identify objects or faces in a series of images
-  Transcribe the speech in an audio or video file
-  Apply sentiment analysis to text
-  Extract musical features from an audio clip
-  Apply a part-of-speech tagger to a block of text

Each of the above tasks can typically be accomplished in 2 - 3 lines of
code with pliers. Combining them *all*--and returning a single,
standardized DataFrame--might take a bit more
work. Say maybe 5 or 6 lines.

In a nutshell, pliers provides a high-level, unified interface to a
large number of feature extraction tools spanning a wide range of
modalities.

Documentation
-------------

The official pliers documentation on ReadTheDocs is comprehensive, and contains a
`quickstart <https://pliers.readthedocs.io/en/latest/quickstart.html>`__,
`API Reference <https://pliers.readthedocs.io/en/latest/reference.html>`__, and more.

Pliers overview (with application to naturalistic fMRI)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Pliers is a general purpose tool, this is just one domain where it's useful.

.. raw:: html

   <iframe width="560" height="315" src="https://www.youtube.com/embed/4mQjtyQPu_c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

`Tutorial Video <https://www.youtube.com/watch?v=4mQjtyQPu_c>`__

The above video is from a `tutorial <https://naturalistic-data.org/content/Pliers_Tutorial.html>`__
as a part of a `course about naturalistic data <https://naturalistic-data.org/>`__.

How to cite
-----------

If you use pliers in your work, please cite both the pliers and the following paper:

   McNamara, Q., De La Vega, A., & Yarkoni, T. (2017, August).
   `Developing a comprehensive framework for multimodal feature
   extraction <https://dl.acm.org/citation.cfm?id=3098075>`__. In
   Proceedings of the 23rd ACM SIGKDD International Conference on
   Knowledge Discovery and Data Mining (pp. 1567-1574). ACM.

.. |PyPI version fury.io| image:: https://badge.fury.io/py/pliers.svg
   :target: https://pypi.python.org/pypi/pliers/
.. |pytest| image:: https://github.com/PsychoinformaticsLab/pliers/actions/workflows/python-package.yml/badge.svg
.. |Coverage Status| image:: https://codecov.io/gh/PsychoinformaticsLab/pliers/branch/master/graph/badge.svg?token=ShZRZ9aqZv
   :target: https://codecov.io/gh/PsychoinformaticsLab/pliers
.. |DOI:10.1145/3097983.3098075| image:: https://zenodo.org/badge/DOI/10.1145/3097983.3098075.svg
   :target: https://doi.org/10.1145/3097983.3098075
.. |docs| image:: https://readthedocs.org/projects/pliers/badge/?version=latest
    :target: https://pliers.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status