=========================
h2: HTTP/2 Protocol Stack
=========================

.. image:: https://github.com/python-hyper/h2/workflows/CI/badge.svg
    :target: https://github.com/python-hyper/h2/actions
    :alt: Build Status
.. image:: https://codecov.io/gh/python-hyper/h2/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/python-hyper/h2
    :alt: Code Coverage
.. image:: https://readthedocs.org/projects/h2/badge/?version=latest
    :target: https://h2.readthedocs.io/en/latest/
    :alt: Documentation Status
.. image:: https://img.shields.io/badge/chat-join_now-brightgreen.svg
    :target: https://gitter.im/python-hyper/community
    :alt: Chat community

.. image:: https://raw.github.com/python-hyper/documentation/master/source/logo/hyper-black-bg-white.png

This repository contains a pure-Python implementation of a HTTP/2 protocol
stack. It's written from the ground up to be embeddable in whatever program you
choose to use, ensuring that you can speak HTTP/2 regardless of your
programming paradigm.

You use it like this:

.. code-block:: python

    import h2.connection
    import h2.config

    config = h2.config.H2Configuration()
    conn = h2.connection.H2Connection(config=config)
    conn.send_headers(stream_id=stream_id, headers=headers)
    conn.send_data(stream_id, data)
    socket.sendall(conn.data_to_send())
    events = conn.receive_data(socket_data)

This repository does not provide a parsing layer, a network layer, or any rules
about concurrency. Instead, it's a purely in-memory solution, defined in terms
of data actions and HTTP/2 frames. This is one building block of a full Python
HTTP implementation.

To install it, just run:

.. code-block:: console

    $ python -m pip install h2

Documentation
=============

Documentation is available at https://h2.readthedocs.io .

Contributing
============

``h2`` welcomes contributions from anyone! Unlike many other projects we
are happy to accept cosmetic contributions and small contributions, in addition
to large feature requests and changes.

Before you contribute (either by opening an issue or filing a pull request),
please `read the contribution guidelines`_.

.. _read the contribution guidelines: http://python-hyper.org/en/latest/contributing.html

License
=======

``h2`` is made available under the MIT License. For more details, see the
``LICENSE`` file in the repository.

Authors
=======

``h2`` was authored by Cory Benfield and is maintained
by the members of `python-hyper <https://github.com/orgs/python-hyper/people>`_.


=============================================
Weaver
=============================================

**Implementations**

* |ogc-proc-long|
* |wps-long|
* |esgf| processes
* |cwl-long| for |ogc-apppkg|_
* |ems-long| for dispatching distrubted workflow processing
* |ades-long| for processing close to the data

Weaver (the nest-builder)
  *Weaver birds build exquisite and elaborate nest structures that are a rival to any human feat of engineering.
  Some of these nests are the largest structures to be built by birds.*
  [`Eden <https://eden.uktv.co.uk/animals/birds/article/weaver-birds/>`_].

  *Although weavers are named for their elaborately woven nests, some are notable for their selective parasitic
  nesting habits instead.*
  [`Wikipedia <https://en.wikipedia.org/wiki/Ploceidae>`_]

`Weaver` is an OGC-API flavored |ems| that allows the execution of workflows chaining various
applications and |wps| inputs and outputs. Remote execution is deferred by the `EMS` to one or many
|ades| or remote service providers, and employs |cwl-long| configurations to define an |ogc-apppkg|_ deployed
for each process.

.. start-badges

.. list-table::
    :stub-columns: 1
    :widths: 20,80

    * - dependencies
      - | |py_ver| |deps| |pyup|
    * - license
      - | |license| |license_scan|
    * - build status
      - | |readthedocs| |docker_build_mode| |docker_build_status|
    * - tests status
      - | |github_latest| |github_tagged| |coverage| |codacy|
    * - releases
      - | |version| |commits-since| |docker_image|

.. |py_ver| image:: https://img.shields.io/badge/python-3.8%2B-blue.svg
    :alt: Requires Python 3.8+
    :target: https://www.python.org/getit

.. |commits-since| image:: https://img.shields.io/github/commits-since/crim-ca/weaver/5.6.1.svg
    :alt: Commits since latest release
    :target: https://github.com/crim-ca/weaver/compare/5.6.1...master

.. |version| image:: https://img.shields.io/badge/latest%20version-5.6.1-blue
    :alt: Latest Tagged Version
    :target: https://github.com/crim-ca/weaver/tree/5.6.1

.. |deps| image:: https://img.shields.io/librariesio/github/crim-ca/weaver
    :alt: Libraries.io Dependencies Status
    :target: https://libraries.io/github/crim-ca/weaver

.. |pyup| image:: https://pyup.io/repos/github/crim-ca/weaver/shield.svg
    :alt: PyUp Dependencies Status
    :target: https://pyup.io/account/repos/github/crim-ca/weaver/

.. |github_latest| image:: https://img.shields.io/github/actions/workflow/status/crim-ca/weaver/tests.yml?label=master&branch=master
    :alt: Github Actions CI Build Status (master branch)
    :target: https://github.com/crim-ca/weaver/actions?query=workflow%3ATests+branch%3Amaster

.. |github_tagged| image:: https://img.shields.io/github/actions/workflow/status/crim-ca/weaver/tests.yml?label=5.6.1&branch=5.6.1
    :alt: Github Actions CI Build Status (latest tag)
    :target: https://github.com/crim-ca/weaver/actions?query=workflow%3ATests+branch%3A5.6.1

.. |readthedocs| image:: https://img.shields.io/readthedocs/pavics-weaver
    :alt: ReadTheDocs Build Status (master branch)
    :target: `ReadTheDocs`_

.. |docker_build_mode| image:: https://img.shields.io/docker/automated/pavics/weaver.svg?label=build
    :alt: Docker Build Mode (latest version)
    :target: https://hub.docker.com/r/pavics/weaver/tags

.. below shield will either indicate the targeted version or 'tag not found'
.. since docker tags are pushed following manual builds by CI, they are not automatic and no build artifact exists
.. |docker_build_status| image:: https://img.shields.io/docker/v/pavics/weaver/5.6.1?label=tag%20status
    :alt: Docker Build Status (latest version)
    :target: https://hub.docker.com/r/pavics/weaver/tags

.. |docker_image| image:: https://img.shields.io/badge/docker-pavics%2Fweaver-blue
    :alt: Docker Image
    :target: https://hub.docker.com/r/pavics/weaver/tags

.. |coverage| image:: https://img.shields.io/codecov/c/gh/crim-ca/weaver.svg?label=coverage
    :alt: Code Coverage
    :target: https://codecov.io/gh/crim-ca/weaver

.. |codacy| image:: https://app.codacy.com/project/badge/Grade/2b340010b41b4401acc9618a437a43b8
    :alt: Codacy Badge
    :target: https://app.codacy.com/gh/crim-ca/weaver/dashboard

.. |license| image:: https://img.shields.io/github/license/crim-ca/weaver.svg
    :target: https://github.com/crim-ca/weaver/blob/master/LICENSE.txt
    :alt: GitHub License

.. |license_scan| image:: https://app.fossa.com/api/projects/git%2Bgithub.com%2Fcrim-ca%2Fweaver.svg?type=shield&issueType=license
    :target: https://app.fossa.com/projects/git%2Bgithub.com%2Fcrim-ca%2Fweaver?ref=badge_shield&issueType=license
    :alt: FOSSA Status

.. end-badges

----------------
Summary
----------------

`Weaver` is primarily an |ems| that allows the execution of workflows chaining various
applications and |wps| inputs and outputs. Remote execution of each process in a workflow
chain is dispatched by the *EMS* to one or many registered |ades| by
ensuring the transfer of files accordingly between instances when located across multiple remote locations.

`Weaver` can also accomplish the `ADES` role in order to perform application deployment at the data source using
the application definition provided by |cwl-long| configuration. It can then directly execute
a registered process |ogc-apppkg|_ with received inputs from a WPS request to expose output results for a
following `ADES` in a `EMS` workflow execution chain.

`Weaver` **extends** |ogc-api-proc|_ by providing additional functionalities such as more detailed job logs
endpoints, adding more process management and search request options than required by the standard, and supporting
*remote providers* registration for dynamic process definitions, to name a few.
Because of this, not all features offered in `Weaver` are guaranteed to be applicable on other similarly
behaving `ADES` and/or `EMS` instances. The reference specification is tracked to preserve the minimal conformance
requirements and provide feedback to |ogc|_ (OGC) in this effect.

`Weaver` can be launched either as an `EMS`, an `ADES` or an `HYBRID` of both according to its configuration.
For more details, see `Configuration`_ and `Documentation`_ sections.

----------------
Features Preview
----------------

Following videos present some of the features and potential capabilities of servicing and executing processes
offered by |ades| and |ems| instances like `Weaver`.

**Keywords**:
Big Data, software architecture, Earth Observation, satellite data, processing, climate change, machine learning,
climate services.

Applications
~~~~~~~~~~~~~~~~

The video shares the fundamental ideas behind the architecture, illustrates how application stores for Earth
Observation data processing can evolve, and illustrates the advantages with applications based on machine learning.

.. Tag iframe renders the embedded video in ReadTheDocs/Sphinx generated build,
   but it is filtered out by GitHub (https://github.github.com/gfm/#disallowed-raw-html-extension-).
   The following div displays instead video thumbnail with an external link only for GitHub.
   When iframe properly renders, the image/link div is masked under it to avoid seeing two "video displays".
.. raw:: html

            Watch the Application video on YouTube

Platform
~~~~~~~~~~~~~~~~

The video shares the fundamental ideas behind the architecture, illustrates how platform managers can benefit from
application stores, and shows the potential for multidisciplinary workflows in thematic platforms.

.. see other video comment
.. raw:: html

            Watch the Platform video on YouTube

----------------
Links
----------------

Docker image repositories:

.. list-table::
    :header-rows: 1

    * - Name
      - Reference
      - Access
    * - DockerHub
      - `pavics/weaver <https://hub.docker.com/r/pavics/weaver>`_
      - |public|
    * - CRIM registry
      - `ogc/weaver <https://docker-registry.crim.ca/repositories/3463>`_
      - |restricted|
    * - CRIM OGC Processes
      - `ogc-public <https://docker-registry.crim.ca/namespaces/39>`_
      - |restricted|

.. |public| image:: https://img.shields.io/badge/public-green
.. |restricted| image:: https://img.shields.io/badge/restricted-orange

For a prebuilt image, pull as follows:

.. code-block:: shell

    docker pull pavics/weaver:5.6.1

For convenience, following tags are also available:

- ``weaver:5.6.1-manager``: `Weaver` image that will run the API for WPS process and job management.
- ``weaver:5.6.1-worker``: `Weaver` image that will run the process job runner application.

Following links correspond to existing servers with `Weaver` configured as *EMS*/*ADES* instances respectively.

.. list-table::
    :header-rows: 1

    * - Description
      - Version
      - Entrypoint
    * - CRIM Services Portal
      - |crim-services-version|
      - `https://services.crim.ca/weaver <https://services.crim.ca/weaver>`_
    * - `DACCS Project <https://github.com/DACCS-Climate>`_ - CRIM Development Instance
      - |hirondelle-version|
      - `https://hirondelle.crim.ca/weaver <https://hirondelle.crim.ca/weaver>`_

.. |crim-services-version| image:: https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fservices.crim.ca%2Fweaver%2Fversions&query=%24.versions%5B0%5D.version&label=version
.. |hirondelle-version| image:: https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhirondelle.crim.ca%2Fweaver%2Fversions&query=%24.versions%5B0%5D.version&label=version

.. note::
    The test servers will **not** necessarily be up-to-date with the *latest* version.

----------------
Configuration
----------------

All configuration settings can be overridden using a ``weaver.ini`` file that will be picked during
instantiation of the application. An example of such file is provided here: `weaver.ini.example`_.

Setting the operational mode of `Weaver` (`EMS`/`ADES`/`HYBRID`) is accomplished using the
``weaver.configuration`` field of ``weaver.ini``. For more configuration details, please refer to Documentation_.

.. _weaver.ini.example: ./config/weaver.ini.example

----------------
Documentation
----------------

The REST API documentation is auto-generated and served under any running `Weaver` application on route
``{WEAVER_URL}/api/``. This documentation will correspond to the version of the executed `Weaver` application.
For the latest documentation, you can refer to the `OpenAPI Specification`_ served directly on `ReadTheDocs`_.

More ample details about installation, configuration and usage are also provided on `ReadTheDocs`_.
These are generated from corresponding information provided in `docs`_ source directory.

.. _ReadTheDocs: https://pavics-weaver.readthedocs.io
.. _`OpenAPI Specification`: https://pavics-weaver.readthedocs.io/en/latest/api.html
.. _docs: ./docs

-------------------------
Extra Details & Sponsors
-------------------------

The project was initially developed upon *OGC Testbed-14 – ESA Sponsored Threads – Exploitation Platform* findings and
improvements following from previous |ogc-tb13-cloud-er|_ architecture designs.
It was also built upon sponsorship from the *U.S. Department of Energy* to support common
API of the |esgf|. The findings are reported on the |ogc-tb14|_ thread, and more
explicitly in the |ogc-tb14-platform-er|_.

The project has been employed for |ogc-tb15-ml|_ to demonstrate the use of Machine Learning interactions with OGC web
standards in the context of natural resources applications. The advancements are reported through the |ogc-tb15-ml-er|_.

Developments are continued in |ogc-tb16|_ to improve methodologies in order to provide better
interoperable geospatial data processing in the areas of Earth Observation Application Packages.
Findings and recommendations are presented in the |ogc-tb16-data-access-proc-er|_.

.. fixme:
.. todo::
   deploy from ipynb, add |ogc-tb16-ipynb-er| (https://github.com/crim-ca/weaver/issues/63)

Videos and more functionalities were introduced in `Weaver` following |ogc-eo-apps-pilot|_.
Corresponding developments are reported in the |ogc-eo-apps-pilot-er|_.

`Weaver` has been used to participate in interoperability testing effort that lead to |ogc-best-practices-eo-apppkg|_
technical report. This resulted, along with previous efforts, in the definition of |ogc-api-proc-part2|_ backed by
validated test cases using |cwl-long| as the representation method for the deployment and execution of |ogc-apppkg|_
close to the data.

`Weaver` is being employed in the |ogc-tb20-gdc|_ initiative to improve and work on the alignment of multiple
community standards involved in workflow design, such as |cwl|_, `openEO`_ and |ogc-api-proc-part3|_, for
processing of multidimensional data involved through GeoDataCube interactions.

The project is furthermore developed through the *Data Analytics for Canadian Climate Services* (`DACCS`_) initiative
and is employed by the ClimateData.ca_/DonneesClimatiques.ca_ portal.

`Weaver` is implemented in Python with the `Pyramid`_ web framework.
It is part of `PAVICS`_ and `Birdhouse`_ ecosystems and is available within the `birdhouse-deploy`_ server stack.

.. NOTE: all references in this file must remain local (instead of imported from 'references.rst')
..       to allow Github to directly referring to them from the repository HTML page.
.. |cwl-long| replace:: `Common Workflow Language`_ (CWL)
.. _`Common Workflow Language`: https://www.commonwl.org/
.. |cwl| replace:: CWL
.. _cwl: https://www.commonwl.org/
.. _openEO: https://openeo.org/
.. |esgf| replace:: `Earth System Grid Federation`_ (ESGF)
.. _`Earth System Grid Federation`: https://esgf.llnl.gov/
.. |ems| replace:: Execution Management Service
.. _ems: https://docs.ogc.org/per/18-050r1.html#_crim
.. |ems-long| replace:: |ems|_ (EMS)
.. |ades| replace:: Application, Deployment and Execution Service
.. _ades: https://docs.ogc.org/per/18-050r1.html#_application_deployment_and_execution_service
.. |ades-long| replace:: |ades|_ (ADES)
.. |wps| replace:: `Web Processing Services`
.. _wps: https://www.ogc.org/standard/wps/
.. |wps-long| replace:: |wps|_ (WPS)
.. |ogc| replace:: Open Geospatial Consortium
.. _ogc: https://www.ogc.org/
.. |ogc-api-proc| replace:: `OGC API - Processes`
.. _ogc-api-proc: https://github.com/opengeospatial/ogcapi-processes
.. |ogc-proc-long| replace:: |ogc-api-proc|_ (WPS-REST bindings)
.. |ogc-tb13-cloud-er| replace:: OGC Testbed-13 - Cloud Engineering Report
.. _ogc-tb13-cloud-er: https://docs.ogc.org/per/17-035.html
.. |ogc-tb14| replace:: OGC Testbed-14
.. _ogc-tb14: https://www.ogc.org/initiatives/testbed-14/
.. |ogc-tb14-platform-er| replace:: ADES & EMS Results and Best Practices Engineering Report
.. _ogc-tb14-platform-er: http://docs.opengeospatial.org/per/18-050r1.html
.. |ogc-tb15-ml| replace:: OGC Testbed-15 - ML Thread
.. _ogc-tb15-ml: https://www.ogc.org/initiatives/testbed-15/#MachineLearning
.. |ogc-tb15-ml-er| replace:: OGC Testbed-15: Machine Learning Engineering Report
.. _ogc-tb15-ml-er: http://docs.opengeospatial.org/per/19-027r2.html
.. |ogc-tb16| replace:: OGC Testbed-16
.. _ogc-tb16: https://www.ogc.org/initiatives/t-16/
.. |ogc-tb16-data-access-proc-er| replace:: OGC Testbed-16: Data Access and Processing Engineering Report
.. _ogc-tb16-data-access-proc-er: http://docs.opengeospatial.org/per/20-016.html
.. |ogc-tb16-ipynb-er| replace:: OGC Testbed-16: Earth Observation Application Packages with Jupyter Notebooks Engineering Report
.. _ogc-tb16-ipynb-er: http://docs.opengeospatial.org/per/20-035.html
.. |ogc-tb20-gdc| replace:: OGC Testbed-20 - GeoDataCubes
.. _ogc-tb20-gdc: https://www.ogc.org/initiatives/ogc-testbed-20/
.. |ogc-eo-apps-pilot| replace:: OGC Earth Observation Applications Pilot
.. _ogc-eo-apps-pilot: https://www.ogc.org/initiatives/eoa-pilot/
.. |ogc-eo-apps-pilot-er| replace:: OGC Earth Observation Applications Pilot: CRIM Engineering Report
.. _ogc-eo-apps-pilot-er: http://docs.opengeospatial.org/per/20-045.html
.. |ogc-best-practices-eo-apppkg| replace:: OGC Best Practice for Earth Observation Application Package
.. _ogc-best-practices-eo-apppkg: https://docs.ogc.org/bp/20-089r1.html
.. |ogc-api-proc-part2| replace:: OGC API - Processes - Part 2: Deploy, Replace, Undeploy (DRU)
.. _ogc-api-proc-part2: https://docs.ogc.org/DRAFTS/20-044.html
.. |ogc-api-proc-part3| replace:: OGC API - Processes - Part 3: Workflows and Chaining
.. _ogc-api-proc-part3: https://docs.ogc.org/DRAFTS/21-009.html
.. |ogc-apppkg| replace:: `OGC Application Package`
.. _ogc-apppkg: https://github.com/opengeospatial/ogcapi-processes/blob/master/openapi/schemas/processes-dru/ogcapppkg.yaml
.. _PAVICS: https://ouranosinc.github.io/pavics-sdi/index.html
.. _Birdhouse: http://bird-house.github.io/
.. _birdhouse-deploy: https://github.com/bird-house/birdhouse-deploy
.. _DACCS: https://app.dimensions.ai/details/grant/grant.8105745
.. _ClimateData.ca: https://ClimateData.ca
.. _DonneesClimatiques.ca: https://DonneesClimatiques.ca
.. _Pyramid: http://www.pylonsproject.org


<img align="right" src="https://raw.githubusercontent.com/mcbeet/beet/main/logo.png?sanitize=true" alt="logo" width="76">

# Beet

[![GitHub Actions](https://github.com/mcbeet/beet/workflows/CI/badge.svg)](https://github.com/mcbeet/beet/actions)
[![PyPI](https://img.shields.io/pypi/v/beet.svg)](https://pypi.org/project/beet/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/beet.svg)](https://pypi.org/project/beet/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
[![Discord](https://img.shields.io/discord/900530660677156924?color=7289DA&label=discord&logo=discord&logoColor=fff)](https://discord.gg/98MdSGMm8j)

> The Minecraft pack development kit.

## Introduction

Minecraft [resource packs](https://minecraft.gamepedia.com/Resource_Pack) and [data packs](https://minecraft.gamepedia.com/Data_Pack) work well as _distribution_ formats but can be pretty limiting as _authoring_ formats. You can quickly end up having to manage hundreds of files, some of which might be buried within the bundled output of various generators.

The `beet` project is a development kit that tries to unify data pack and resource pack tooling into a single pipeline. The community is always coming up with pre-processors, frameworks, and generators of all kinds to make the developer experience more ergonomic. With `beet` you can seamlessly integrate all these tools in your project.

### Screencasts

- **Quick start** [https://youtu.be/JGrJTOhG3pY](https://youtu.be/JGrJTOhG3pY)
- **Command-line** [https://youtu.be/fQ9up0ELPNE](https://youtu.be/fQ9up0ELPNE)
- **Library overview** [https://youtu.be/LDvV4_l-PSc](https://youtu.be/LDvV4_l-PSc)
- **Plugins basics** [https://youtu.be/XTzKmvHqd1g](https://youtu.be/XTzKmvHqd1g)
- **Pipeline configuration** [https://youtu.be/QsnQncGxAAs](https://youtu.be/QsnQncGxAAs)

### Library

```python
from beet import ResourcePack, Texture

# Open a zipped resource pack and add a custom stone texture
with ResourcePack(path="stone.zip") as assets:
    assets["minecraft:block/stone"] = Texture(source_path="custom.png")
```

The `beet` library provides carefully crafted primitives for working with Minecraft resource packs and data packs.

- Create, read, edit and merge resource packs and data packs
- Handle zipped and unzipped packs
- Fast and lazy by default, files are transparently loaded when needed
- Statically typed API enabling rich intellisense and autocompletion
- First-class [`pytest`](https://github.com/pytest-dev/pytest/) integration with detailed assertion explanations

### Toolchain

```python
from beet import Context, Function

def greet(ctx: Context):
    """Plugin that adds a function for greeting the player."""
    ctx.data["greet:hello"] = Function(["say hello"], tags=["minecraft:load"])
```

The `beet` toolchain is designed to support a wide range of use-cases. The most basic pipeline will let you create configurable resource packs and data packs, but plugins make it easy to implement arbitrarily advanced workflows and tools like linters, asset generators and function pre-processors.

- Compose plugins that can inspect and edit the generated resource pack and data pack
- Configure powerful build systems for development and creating releases
- First-class template integration approachable without prior Python knowledge
- Link the generated resource pack and data pack to Minecraft
- Automatically rebuild the project on file changes with watch mode
- Batteries-included package that comes with a few handy plugins out of the box
- Rich ecosystem, extensible CLI, and powerful generator and worker API

## Installation

The package can be installed with `pip`.

```bash
$ pip install beet
```

To create and edit images programmatically you should install `beet` with the `image` extra or install `Pillow` separately.

```bash
$ pip install beet[image]
$ pip install beet Pillow
```

You can make sure that `beet` was successfully installed by trying to use the toolchain from the command-line.

```bash
$ beet --help
Usage: beet [OPTIONS] COMMAND [ARGS]...

  The beet toolchain.

Options:
  -p, --project PATH  Select project.
  -s, --set OPTION    Set config option.
  -l, --log LEVEL     Configure output verbosity.
  -v, --version       Show the version and exit.
  -h, --help          Show this message and exit.

Commands:
  build  Build the current project.
  cache  Inspect or clear the cache.
  link   Link the generated resource pack and data pack to Minecraft.
  watch  Watch the project directory and build on file changes.
```

## Contributing

Contributions are welcome. Make sure to first open an issue discussing the problem or the new feature before creating a pull request. The project uses [`poetry`](https://python-poetry.org).

```bash
$ poetry install --extras image
```

You can run the tests with `poetry run pytest`. We use [`pytest-minecraft`](https://github.com/vberlier/pytest-minecraft) to run tests against actual Minecraft releases.

```bash
$ poetry run pytest
$ poetry run pytest --minecraft-latest
```

We also use [`pytest-insta`](https://github.com/vberlier/pytest-minecraft) for snapshot testing. Data pack and resource pack snapshots make it easy to monitor and review changes.

```bash
$ poetry run pytest --insta review
```

The project must type-check with [`pyright`](https://github.com/microsoft/pyright). If you're using VSCode the [`pylance`](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance) extension should report diagnostics automatically. You can also install the type-checker locally with `npm install` and run it from the command-line.

```bash
$ npm run watch
$ npm run check
```

The code follows the [`black`](https://github.com/psf/black) code style. Import statements are sorted with [`isort`](https://pycqa.github.io/isort/).

```bash
$ poetry run isort beet tests
$ poetry run black beet tests
$ poetry run black --check beet tests
```

---

License - [MIT](https://github.com/mcbeet/beet/blob/main/LICENSE)


# Ara API

Restful API for Ara, KAIST's official community service

![GitHub Pipenv locked Python version (master)][badge/python-version]
[![Code style: black][badge/black]][repo/black]
[![Imports: isort][badge/isort]][isort]
[![pre-commit][badge/pre-commit]][repo/pre-commit]
[![Conventional Commits][badge/conventional-commits]][conventional-commits]

## Project Setup

Use pipenv to install packages. (e.g., `pipenv install <package>`)

```bash
pipenv --python 3.11 # Use python 3.11
pipenv shell # Activate virtual environment
pipenv install --dev # Install packages (`--dev` flag for development)

# Run `pre-commit` automatically on `git commit`
pre-commit install
pre-commit install --hook-type commit-msg
```

## How to Run

```bash
docker compose -f docker-compose.local.yml up -d
pipenv shell
make run
```

## URLs

- Base URL: `/api`
- Admin page: `/api/admin/`
- API documentations:
  - `/api/schema/swagger/`
  - `/api/schema/redoc/`

## How to Contribute

1. Follow [Conventional Commits][conventional-commits] for writing commit messages.
2. Use type hints strictly. (Check [PEP 484][pep-484].)

[badge/python-version]: https://img.shields.io/github/pipenv/locked/python-version/sparcs-kaist/new-ara-api/master
[badge/black]: https://img.shields.io/badge/code%20style-black-000000
[badge/isort]: https://img.shields.io/badge/%20imports-isort-%231674b1?labelColor=ef8336
[badge/pre-commit]: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit
[badge/conventional-commits]: https://img.shields.io/badge/Conventional%20Commits-1.0.0-%23FE5196?logo=conventionalcommits&logoColor=white
[isort]: https://pycqa.github.io/isort
[conventional-commits]: https://conventionalcommits.org
[pep-484]: https://peps.python.org/pep-0484/
[repo/black]: https://github.com/psf/black
[repo/pre-commit]: https://github.com/pre-commit/pre-commit


.. image:: https://img.shields.io/pypi/v/importlib_metadata.svg
   :target: https://pypi.org/project/importlib_metadata

.. image:: https://img.shields.io/pypi/pyversions/importlib_metadata.svg

.. image:: https://github.com/python/importlib_metadata/actions/workflows/main.yml/badge.svg
   :target: https://github.com/python/importlib_metadata/actions?query=workflow%3A%22tests%22
   :alt: tests

.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json
    :target: https://github.com/astral-sh/ruff
    :alt: Ruff

.. image:: https://readthedocs.org/projects/importlib-metadata/badge/?version=latest
   :target: https://importlib-metadata.readthedocs.io/en/latest/?badge=latest

.. image:: https://img.shields.io/badge/skeleton-2024-informational
   :target: https://blog.jaraco.com/skeleton

.. image:: https://tidelift.com/badges/package/pypi/importlib-metadata
   :target: https://tidelift.com/subscription/pkg/pypi-importlib-metadata?utm_source=pypi-importlib-metadata&utm_medium=readme

Library to access the metadata for a Python package.

This package supplies third-party access to the functionality of
`importlib.metadata <https://docs.python.org/3/library/importlib.metadata.html>`_
including improvements added to subsequent Python versions.

Compatibility
=============

New features are introduced in this third-party library and later merged
into CPython. The following table indicates which versions of this library
were contributed to different versions in the standard library:

.. list-table::
   :header-rows: 1

   * - importlib_metadata
     - stdlib
   * - 7.0
     - 3.13
   * - 6.5
     - 3.12
   * - 4.13
     - 3.11
   * - 4.6
     - 3.10
   * - 1.4
     - 3.8

Usage
=====

See the `online documentation <https://importlib-metadata.readthedocs.io/>`_
for usage details.

`Finder authors
<https://docs.python.org/3/reference/import.html#finders-and-loaders>`_ can
also add support for custom package installers.  See the above documentation
for details.

Caveats
=======

This project primarily supports third-party packages installed by PyPA
tools (or other conforming packages). It does not support:

- Packages in the stdlib.
- Packages installed without metadata.

Project details
===============

 * Project home: https://github.com/python/importlib_metadata
 * Report bugs at: https://github.com/python/importlib_metadata/issues
 * Code hosting: https://github.com/python/importlib_metadata
 * Documentation: https://importlib-metadata.readthedocs.io/

For Enterprise
==============

Available as part of the Tidelift Subscription.

This project and the maintainers of thousands of other packages are working with Tidelift to deliver one enterprise subscription that covers all of the open source you use.

`Learn more <https://tidelift.com/subscription/pkg/pypi-importlib-metadata?utm_source=pypi-importlib-metadata&utm_medium=referral&utm_campaign=github>`_.


[![GridPath Test Suite Status](https://github.com/blue-marble/gridpath/actions/workflows/test_gridpath.yml/badge.svg?branch=main)](https://github.com/blue-marble/gridpath/actions/workflows/test_gridpath.yml)
[![Documentation Status](https://readthedocs.org/projects/gridpath/badge/?version=latest)](https://gridpath.readthedocs.io/en/latest/?badge=latest)
[![Coverage Status](https://coveralls.io/repos/github/blue-marble/gridpath/badge.svg?branch=main)](https://coveralls.io/github/blue-marble/gridpath?branch=main)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Lint Black](https://github.com/blue-marble/gridpath/actions/workflows/black.yml/badge.svg?branch=main)](https://github.com/blue-marble/gridpath/actions/workflows/black.yml)
[![DOI](https://zenodo.org/badge/65574330.svg)](https://zenodo.org/badge/latestdoi/65574330)

# Welcome to GridPath

![Approaches](https://github.com/blue-marble/gridpath/blob/main/doc/graphics/approaches.png?raw=true)

GridPath is a versatile power-system planning platform capable of a range of
planning approaches including production-cost, capacity-expansion,
asset-valuation, and reliability modeling.

# Documentation
GridPath's documentation is hosted on [Read the Docs](https://gridpath.readthedocs.io/en/latest/).

# Installation

## Python
GridPath is tested on Python 3.9, 3.10, and 3.11. Get one of those Python versions [here](https://www.python.org/downloads/ "Python download").

## GridPath Python environment
You should create a Python environment for your GridPath installation, e.g. via
`venv`, [a lightweight environment manager](https://docs.python.org/3/library/venv.html "venv")
that is part of the standard Python distribution. Make sure to [create](https://docs.python.org/3/library/venv.html#creating-virtual-environments "create") [activate](https://docs.python.org/3/library/venv.html#how-venvs-work "activate") the environment before installing GridPath.

## Install GridPath from PyPi

Once you have _created and activated_ the GridPath Python environment, you
can install the latest version of GridPath from PyPi with:

```bash
pip install GridPath
```

## Install GridPath from source

You can alternatively download the GridPath source code and install from
source.

```bash
pip install .[all]
```

**NOTE:** If you plan to edit the GridPath code, you should install with the `-e` flag.

## Solver
You will need a solver to use this platform. GridPath assumes you will be using Cbc (Coin-or branch and cut) by default, but you can specify a
different solver.

# Usage

## The gridpath_run and gridpath_run_e2e commands
If you install GridPath via the setup script following the instructions above,
you can use the command `gridpath_run` to run a scenario from any directory
-- as long as your GridPath Python environment is enabled -- as follows:
```bash
gridpath_run --scenario SCENARIO_NAME --scenario_location
/PATH/TO/SCENARIO
```

If you are using the database, you can use the command `gridpath_run_e2e` to
run GridPath end-to-end, i.e. get inputs for the scenario from the database,
solve the scenario problem, import the results into the database, and
process them. Refer to the documentation for how to build the database.

```bash
gridpath_run_e2e --scenario SCENARIO_NAME --scenario_location
/PATH/TO/SCENARIO
```

To see usage and other optional arguments, e.g. how to specify a
solver, check the help menu, e.g.:
```bash
gridpath_run --help
```


# Prometheus RPM Packages

![Build Status](https://github.com/lest/prometheus-rpm/actions/workflows/build.yml/badge.svg) ![Lint Status](https://github.com/lest/prometheus-rpm/actions/workflows/linter.yml/badge.svg) ![Update Status](https://github.com/lest/prometheus-rpm/actions/workflows/check_new_versions.yml/badge.svg)

The repository contains the files needed to build [Prometheus][1] RPM packages
for CentOS 7 & 8 and CentOS Stream 9.

## Installing
The packages are available in [the packagecloud repository][2] and can be used
by adding the following `/etc/yum.repos.d/prometheus.repo`:

### CentOS
``` conf
[prometheus]
name=prometheus
baseurl=https://packagecloud.io/prometheus-rpm/release/el/$releasever/$basearch
repo_gpgcheck=1
enabled=1
gpgkey=https://packagecloud.io/prometheus-rpm/release/gpgkey
       https://raw.githubusercontent.com/lest/prometheus-rpm/master/RPM-GPG-KEY-prometheus-rpm
gpgcheck=1
metadata_expire=300
```

## Adding a new exporter
### Auto generation
1. Add a new section under `packages` in `templating.yaml` with the required information (overriding any defaults if required).
2. Create a new directory with the name of the exporter and populate it with a file named `<exporter_name>.default` which will contain the default environment variables passed to the init and unit files.
3. Once this is done add this exporter to the list of `AUTO_GENERATED` exporters in `Makefile`.
4. Test that you can build this RPM using the command `make <exporter_name>`.

### Custom
1. Add the exporter to the list of `MANUAL` in the file `Makefile`.
2. Make a new directory with the same name as the exporter.
3. Populate this directory with all the required files to build the RPM.
4. Test that you can build this RPM using the command `make <exporter_name>`.

## Build RPMs manually

Build all packages with:

``` shell
make all
```

or build a single package only, e.g.:

``` shell
make node_exporter
```

The resulting RPMs will be created in the `_dist7` or `_dist8` directories depending on the version of CentOS that they were built for.

## Ansible role

An [Ansible][3] role which installs Prometheus packages from these RPMs is
available in [Github][4] or in [Galaxy][5].

[1]: https://prometheus.io
[2]: https://packagecloud.io/prometheus-rpm/release
[3]: https://www.ansible.com/
[4]: https://github.com/cogini/ansible-role-prometheus-rpm
[5]: https://galaxy.ansible.com/cogini/prometheus-rpm/


Run your *raw* PyTorch training script on any kind of device

## Easy to integrate

🤗 Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boilerplate code needed to use multi-GPUs/TPU/fp16.

🤗 Accelerate abstracts exactly and only the boilerplate code related to multi-GPUs/TPU/fp16 and leaves the rest of your code unchanged.

Here is an example:

```diff
  import torch
  import torch.nn.functional as F
  from datasets import load_dataset
+ from accelerate import Accelerator

+ accelerator = Accelerator()
- device = 'cpu'
+ device = accelerator.device

  model = torch.nn.Transformer().to(device)
  optimizer = torch.optim.Adam(model.parameters())

  dataset = load_dataset('my_dataset')
  data = torch.utils.data.DataLoader(dataset, shuffle=True)

+ model, optimizer, data = accelerator.prepare(model, optimizer, data)

  model.train()
  for epoch in range(10):
      for source, targets in data:
          source = source.to(device)
          targets = targets.to(device)

          optimizer.zero_grad()

          output = model(source)
          loss = F.cross_entropy(output, targets)

-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
```

As you can see in this example, by adding 5-lines to any standard PyTorch training script you can now run on any kind of single or distributed node setting (single CPU, single GPU, multi-GPUs and TPUs) as well as with or without mixed precision (fp8, fp16, bf16).

In particular, the same code can then be run without modification on your local machine for debugging or your training environment.

🤗 Accelerate even handles the device placement for you (which requires a few more changes to your code, but is safer in general), so you can even simplify your training loop further:

```diff
  import torch
  import torch.nn.functional as F
  from datasets import load_dataset
+ from accelerate import Accelerator

- device = 'cpu'
+ accelerator = Accelerator()

- model = torch.nn.Transformer().to(device)
+ model = torch.nn.Transformer()
  optimizer = torch.optim.Adam(model.parameters())

  dataset = load_dataset('my_dataset')
  data = torch.utils.data.DataLoader(dataset, shuffle=True)

+ model, optimizer, data = accelerator.prepare(model, optimizer, data)

  model.train()
  for epoch in range(10):
      for source, targets in data:
-         source = source.to(device)
-         targets = targets.to(device)

          optimizer.zero_grad()

          output = model(source)
          loss = F.cross_entropy(output, targets)

-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
```

Want to learn more? Check out the [documentation](https://huggingface.co/docs/accelerate) or have a look at our [examples](https://github.com/huggingface/accelerate/tree/main/examples).

## Launching script

🤗 Accelerate also provides an optional CLI tool that allows you to quickly configure and test your training environment before launching the scripts. No need to remember how to use `torch.distributed.run` or to write a specific launcher for TPU training!
On your machine(s) just run:

```bash
accelerate config
```

and answer the questions asked. This will generate a config file that will be used automatically to properly set the default options when doing

```bash
accelerate launch my_script.py --args_to_my_script
```

For instance, here is how you would run the GLUE example on the MRPC task (from the root of the repo):

```bash
accelerate launch examples/nlp_example.py
```

This CLI tool is **optional**, and you can still use `python my_script.py` or `python -m torchrun my_script.py` at your convenience.

You can also directly pass in the arguments you would to `torchrun` as arguments to `accelerate launch` if you wish to not run` accelerate config`.

For example, here is how to launch on two GPUs:

```bash
accelerate launch --multi_gpu --num_processes 2 examples/nlp_example.py
```

To learn more, check the CLI documentation available [here](https://huggingface.co/docs/accelerate/package_reference/cli).

## Launching multi-CPU run using MPI

🤗 Here is another way to launch multi-CPU run using MPI. You can learn how to install Open MPI on [this page](https://www.open-mpi.org/faq/?category=building#easy-build). You can use Intel MPI or MVAPICH as well.
Once you have MPI setup on your cluster, just run:
```bash
accelerate config
```
Answer the questions that are asked, selecting to run using multi-CPU, and answer "yes" when asked if you want accelerate to launch mpirun.
Then, use `accelerate launch` with your script like:
```bash
accelerate launch examples/nlp_example.py
```
Alternatively, you can use mpirun directly, without using the CLI like:
```bash
mpirun -np 2 python examples/nlp_example.py
```

## Launching training using DeepSpeed

🤗 Accelerate supports training on single/multiple GPUs using DeepSpeed. To use it, you don't need to change anything in your training code; you can set everything using just `accelerate config`. However, if you desire to tweak your DeepSpeed related args from your Python script, we provide you the `DeepSpeedPlugin`.

```python
from accelerate import Accelerator, DeepSpeedPlugin

# deepspeed needs to know your gradient accumulation steps beforehand, so don't forget to pass it
# Remember you still need to do gradient accumulation by yourself, just like you would have done without deepspeed
deepspeed_plugin = DeepSpeedPlugin(zero_stage=2, gradient_accumulation_steps=2)
accelerator = Accelerator(mixed_precision='fp16', deepspeed_plugin=deepspeed_plugin)

# How to save your 🤗 Transformer?
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(model)
unwrapped_model.save_pretrained(save_dir, save_function=accelerator.save, state_dict=accelerator.get_state_dict(model))
```

Note: DeepSpeed support is experimental for now. In case you get into some problem, please open an issue.

## Launching your training from a notebook

🤗 Accelerate also provides a `notebook_launcher` function you can use in a notebook to launch a distributed training. This is especially useful for Colab or Kaggle notebooks with a TPU backend. Just define your training loop in a `training_function` then in your last cell, add:

```python
from accelerate import notebook_launcher

notebook_launcher(training_function)
```

An example can be found in [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/accelerate_examples/simple_nlp_example.ipynb). [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/accelerate_examples/simple_nlp_example.ipynb)

## Why should I use 🤗 Accelerate?

You should use 🤗 Accelerate when you want to easily run your training scripts in a distributed environment without having to renounce full control over your training loop. This is not a high-level framework above PyTorch, just a thin wrapper so you don't have to learn a new library. In fact, the whole API of 🤗 Accelerate is in one class, the `Accelerator` object.

## Why shouldn't I use 🤗 Accelerate?

You shouldn't use 🤗 Accelerate if you don't want to write a training loop yourself. There are plenty of high-level libraries above PyTorch that will offer you that, 🤗 Accelerate is not one of them.

## Frameworks using 🤗 Accelerate

If you like the simplicity of 🤗 Accelerate but would prefer a higher-level abstraction around its capabilities, some frameworks and libraries that are built on top of 🤗 Accelerate are listed below:

* [Amphion](https://github.com/open-mmlab/Amphion) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engineers get started in the field of audio, music, and speech generation research and development.
* [Animus](https://github.com/Scitator/animus) is a minimalistic framework to run machine learning experiments. Animus highlights common "breakpoints" in ML experiments and provides a unified interface for them within [IExperiment](https://github.com/Scitator/animus/blob/main/animus/core.py#L76).
* [Catalyst](https://github.com/catalyst-team/catalyst#getting-started) is a PyTorch framework for Deep Learning Research and Development. It focuses on reproducibility, rapid experimentation, and codebase reuse so you can create something new rather than write yet another train loop. Catalyst provides a [Runner](https://catalyst-team.github.io/catalyst/api/core.html#runner) to connect all parts of the experiment: hardware backend, data transformations, model training, and inference logic.
* [fastai](https://github.com/fastai/fastai#installing) is a PyTorch framework for Deep Learning that simplifies training fast and accurate neural nets using modern best practices. fastai provides a [Learner](https://docs.fast.ai/learner.html#Learner) to handle the training, fine-tuning, and inference of deep learning algorithms.
* [Finetuner](https://github.com/jina-ai/finetuner) is a service that enables models to create higher-quality embeddings for semantic search, visual similarity search, cross-modal text<->image search, recommendation systems, clustering, duplication detection, anomaly detection, or other uses.
* [InvokeAI](https://github.com/invoke-ai/InvokeAI) is a creative engine for Stable Diffusion models, offering industry-leading WebUI, terminal usage support, and serves as the foundation for many commercial products.
* [Kornia](https://kornia.readthedocs.io/en/latest/get-started/introduction.html) is a differentiable library that allows classical computer vision to be integrated into deep learning models. Kornia provides a [Trainer](https://kornia.readthedocs.io/en/latest/x.html#kornia.x.Trainer) with the specific purpose to train and fine-tune the supported deep learning algorithms within the library.
* [Open Assistant](https://projects.laion.ai/Open-Assistant/) is a chat-based assistant that understands tasks, can interact with their party systems, and retrieve information dynamically to do so.
* [pytorch-accelerated](https://github.com/Chris-hughes10/pytorch-accelerated) is a lightweight training library, with a streamlined feature set centered around a general-purpose [Trainer](https://pytorch-accelerated.readthedocs.io/en/latest/trainer.html), that places a huge emphasis on simplicity and transparency; enabling users to understand exactly what is going on under the hood, but without having to write and maintain the boilerplate themselves!
* [Stable Diffusion web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) is an open-source browser-based easy-to-use interface based on the Gradio library for Stable Diffusion.
* [torchkeras](https://github.com/lyhue1991/torchkeras) is a simple tool for training pytorch model just in a keras style, a dynamic and beautiful plot is provided in notebook to monitor your loss or metric.
* [transformers](https://github.com/huggingface/transformers) as a tool for helping train state-of-the-art machine learning models in PyTorch, Tensorflow, and JAX. (Accelerate is the backend for the PyTorch side).

## Installation

This repository is tested on Python 3.8+ and PyTorch 1.10.0+

You should install 🤗 Accelerate in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).

First, create a virtual environment with the version of Python you're going to use and activate it.

Then, you will need to install PyTorch: refer to the [official installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform. Then 🤗 Accelerate can be installed using pip as follows:

```bash
pip install accelerate
```

## Supported integrations

- CPU only
- multi-CPU on one node (machine)
- multi-CPU on several nodes (machines)
- single GPU
- multi-GPU on one node (machine)
- multi-GPU on several nodes (machines)
- TPU
- FP16/BFloat16 mixed precision
- FP8 mixed precision with [Transformer Engine](https://github.com/NVIDIA/TransformerEngine)
- DeepSpeed support (Experimental)
- PyTorch Fully Sharded Data Parallel (FSDP) support (Experimental)
- Megatron-LM support (Experimental)

## Citing 🤗 Accelerate

If you use 🤗 Accelerate in your publication, please cite it by using the following BibTeX entry.

```bibtex
@Misc{accelerate,
  title =        {Accelerate: Training and inference at scale made simple, efficient and adaptable.},
  author =       {Sylvain Gugger and Lysandre Debut and Thomas Wolf and Philipp Schmid and Zachary Mueller and Sourab Mangrulkar and Marc Sun and Benjamin Bossan},
  howpublished = {\url{https://github.com/huggingface/accelerate}},
  year =         {2022}
}
```


# DEA Config
Config files for Digital Earth Australia (DEA) services

This is a central space used to store configuration for our services and products.

Below are notable folders:

- [configs for DEV ENV](https://github.com/GeoscienceAustralia/dea-config/tree/master/dev)
- [configs for PROD ENV](https://github.com/GeoscienceAustralia/dea-config/tree/master/prod)
- [products definition](https://github.com/GeoscienceAustralia/dea-config/tree/master/products)
- [product metadatas](https://github.com/GeoscienceAustralia/dea-config/tree/master/product_metadata)

## Docker image

image name: `geoscienceaustralia/dea-datakube-config`
up to version `1.5.38-13-g43d4569a` images are available from: [dockerhub](https://hub.docker.com/r/geoscienceaustralia/dea-datakube-config)
newer images are only available from AWS ECR

## Formatting for Terria Data Description section in `ows config`
#### Bold
```
**Overview**
```
#### Italic
```
*Overview*
```

## Testing

### products metadata and products definition testing

#### testing workspace csv
First, create an `.env` file for the `docker-compose` setup
```bash
    vi .env
    # inside .env
ENV=prod
EXPLORER_ENDPOINT=explorer.sandbox.dea.ga.gov.au
```
then bring up the testing tool
```bash
    docker-compose -f docker-compose.productscsv.yaml up -d
    docker exec -it dea-config_datacube_1 bash
    # within the docker container run
    datacube system init
    cd /env/config
    ./datacube_init_metadata.sh
    ./check-product-csv.sh
```

#### audit checks (local-check-script.sh)
```bash
    docker-compose -f docker-compose.productscsv.yaml up -d
    docker exec -it dea-config_datacube_1 bash
    # within the docker container run
    datacube system init
    cd /env/config
    ./datacube_init_metadata.sh
    ./local-check-script.sh
```

### ows config testing
First, create an `.env` file for the `docker-compose` setup

```bash
    vi .env
    # inside .env
OWS_CFG_PATH=dev/services/wms/ows_refactored
DATACUBE_OWS_CFG=ows_refactored.ows_root_cfg.ows_cfg
PYTHON_PATH=dev/services/wms/
```

then bring up the testing tool

```bash
    docker-compose -f docker-compose.ows.yaml up -d
    docker exec -it dea-config_ows_1 bash
    # within the docker container run
    cd /env/config
    datacube system init
    ./compare-cfg.sh
```

## Pull Request Template
There are three pull request Templates available for the following changes:
- ows config changes (ows_cfg.md)
- terria json changes (terria.md)
- product yaml file changes (product.md)

To access the PR templates
- open up a new pull request page and add `&template=ows_cfg.md`, `&template=terria.md` or `&template=product.md` at the end of the url

for example https://github.com/GeoscienceAustralia/dea-config/compare/ows-fix-depreciated?expand=1&template=ows_cfg.md
# Usage
Files should be referenced by the raw tag on the master branch i.e.

https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/prod/products/nrt/landsat/products.yaml

# Dependants

[Datacube EKS](https://github.com/opendatacube/datacube-k8s-eks) - Containerised Web Services for DEA running on
AWS EKS.


![scalene](https://github.com/plasma-umass/scalene/raw/master/docs/scalene-icon-white.png)

# Scalene: a Python CPU+GPU+memory profiler with AI-powered optimization proposals

by [Emery Berger](https://emeryberger.com), [Sam Stern](https://samstern.me/), and [Juan Altmayer Pizzorno](https://github.com/jaltmayerpizzorno).

[![Scalene community Slack](https://github.com/plasma-umass/scalene/raw/master/docs/images/slack-logo.png)](https://join.slack.com/t/scaleneprofil-jge3234/shared_invite/zt-110vzrdck-xJh5d4gHnp5vKXIjYD3Uwg)[Scalene community Slack](https://join.slack.com/t/scaleneprofil-jge3234/shared_invite/zt-110vzrdck-xJh5d4gHnp5vKXIjYD3Uwg)

[![PyPI Latest Release](https://img.shields.io/pypi/v/scalene.svg)](https://pypi.org/project/scalene/)[![Anaconda-Server Badge](https://img.shields.io/conda/v/conda-forge/scalene)](https://anaconda.org/conda-forge/scalene) [![Downloads](https://static.pepy.tech/badge/scalene)](https://pepy.tech/project/scalene)[![Anaconda downloads](https://img.shields.io/conda/d/conda-forge/scalene?logo=conda)](https://anaconda.org/conda-forge/scalene) [![Downloads](https://static.pepy.tech/badge/scalene/month)](https://pepy.tech/project/scalene) ![Python versions](https://img.shields.io/pypi/pyversions/scalene.svg?style=flat-square)[![Visual Studio Code Extension version](https://img.shields.io/visual-studio-marketplace/v/emeryberger.scalene?logo=visualstudiocode)](https://marketplace.visualstudio.com/items?itemName=EmeryBerger.scalene) ![License](https://img.shields.io/github/license/plasma-umass/scalene)

![Ozsvald tweet](https://github.com/plasma-umass/scalene/raw/master/docs/Ozsvald-tweet.png)

(tweet from Ian Ozsvald, author of [_High Performance Python_](https://smile.amazon.com/High-Performance-Python-Performant-Programming/dp/1492055026/ref=sr_1_1?crid=texbooks))

![Semantic Scholar success story](https://github.com/plasma-umass/scalene/raw/master/docs/semantic-scholar-success.png)

***Scalene web-based user interface:*** [http://plasma-umass.org/scalene-gui/](http://plasma-umass.org/scalene-gui/)

## About Scalene

Scalene is a high-performance CPU, GPU *and* memory profiler for
Python that does a number of things that other Python profilers do not
and cannot do.  It runs orders of magnitude faster than many other
profilers while delivering far more detailed information. It is also
the first profiler ever to incorporate AI-powered proposed
optimizations.

### AI-powered optimization suggestions

> **Note**
>
> To enable AI-powered optimization suggestions, you need to enter an [OpenAI key](https://openai.com/api/) in the box under "Advanced options". _Your account will need to have a positive balance for this to work_ (check your balance at https://platform.openai.com/account/usage).
>
>

Once you've entered your OpenAI key (see above), click on the lightning bolt (⚡) beside any line or the explosion (💥) for an entire region of code to generate a proposed optimization. Click on a proposed optimization to copy it to the clipboard.

You can click as many times as you like on the lightning bolt or explosion, and it will generate different suggested optimizations. Your mileage may vary, but in some cases, the suggestions are quite impressive (e.g., order-of-magnitude improvements).

### Quick Start

#### Installing Scalene:

```console
python3 -m pip install -U scalene
```

or

```console
conda install -c conda-forge scalene
```

#### Using Scalene:

After installing Scalene, you can use Scalene at the command line, or as a Visual Studio Code extension.

    Using the Scalene VS Code Extension:

First, install the Scalene extension from the VS Code Marketplace or by searching for it within VS Code by typing Command-Shift-X (Mac) or Ctrl-Shift-X (Windows). Once that's installed, click Command-Shift-P or Ctrl-Shift-P to open the Command Palette. Then select "Scalene: AI-powered profiling..." (you can start typing Scalene and it will pop up if it's installed). Run that and, assuming your code runs for at least a second, a Scalene profile will appear in a webview.

Commonly used command-line options:

```console
scalene your_prog.py                             # full profile (outputs to web interface)
python3 -m scalene your_prog.py                  # equivalent alternative

scalene --cli your_prog.py                       # use the command-line only (no web interface)

scalene --cpu your_prog.py                       # only profile CPU
scalene --cpu --gpu your_prog.py                 # only profile CPU and GPU
scalene --cpu --gpu --memory your_prog.py        # profile everything (same as no options)

scalene --reduced-profile your_prog.py           # only profile lines with significant usage
scalene --profile-interval 5.0 your_prog.py      # output a new profile every five seconds

scalene (Scalene options) --- your_prog.py (...) # use --- to tell Scalene to ignore options after that point
scalene --help                                   # lists all options
```

Using Scalene programmatically in your code:

Invoke using `scalene` as above and then:

```Python
from scalene import scalene_profiler

# Turn profiling on
scalene_profiler.start()

# your code

# Turn profiling off
scalene_profiler.stop()
```

Using Scalene to profile only specific functions via @profile:

Just preface any functions you want to profile with the `@profile` decorator and run it with Scalene:

```Python
# do not import profile!

@profile
def slow_function():
    import time
    time.sleep(3)
```

#### Web-based GUI

Scalene has both a CLI and a web-based GUI [(demo here)](http://plasma-umass.org/scalene-gui/).

By default, once Scalene has profiled your program, it will open a
tab in a web browser with an interactive user interface (all processing is done
locally). Hover over bars to see breakdowns of CPU and memory
consumption, and click on underlined column headers to sort the
columns. The generated file `profile.html` is self-contained and can be saved for later use.

[![Scalene web GUI](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/scalene-gui-example.png)](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/scalene-gui-example-full.png)

## Scalene Overview

### Scalene talk (PyCon US 2021)

[This talk](https://youtu.be/5iEf-_7mM1k) presented at PyCon 2021 walks through Scalene's advantages and how to use it to debug the performance of an application (and provides some technical details on its internals). We highly recommend watching this video!

[![Scalene presentation at PyCon 2021](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/images/scalene-video-img.png)](https://youtu.be/5iEf-_7mM1k "Scalene presentation at PyCon 2021")

### Fast and Accurate

- Scalene is **_fast_**. It uses sampling instead of instrumentation or relying on Python's tracing facilities. Its overhead is typically no more than 10-20% (and often less).

- Scalene is **accurate**. We tested CPU profiler accuracy and found that Scalene is among the most accurate profilers, correctly measuring time taken.

![Profiler accuracy](https://github.com/plasma-umass/scalene/raw/master/docs/cpu-accuracy-comparison.png)

- Scalene performs profiling **_at the line level_** _and_ **_per function_**, pointing to the functions and the specific lines of code responsible for the execution time in your program.

### CPU profiling

- Scalene **separates out time spent in Python from time in native code** (including libraries). Most Python programmers aren't going to optimize the performance of native code (which is usually either in the Python implementation or external libraries), so this helps developers focus their optimization efforts on the code they can actually improve.
- Scalene **highlights hotspots** (code accounting for significant percentages of CPU time or memory allocation) in red, making them even easier to spot.
- Scalene also separates out **system time**, making it easy to find I/O bottlenecks.

### GPU profiling

- Scalene reports **GPU time** (currently limited to NVIDIA-based systems).

### Memory profiling

- Scalene **profiles memory usage**. In addition to tracking CPU usage, Scalene also points to the specific lines of code responsible for memory growth. It accomplishes this via an included specialized memory allocator.
- Scalene separates out the percentage of **memory consumed by Python code vs. native code**.
- Scalene produces **_per-line_ memory profiles**.
- Scalene **identifies lines with likely memory leaks**.
- Scalene **profiles _copying volume_**, making it easy to spot inadvertent copying, especially due to crossing Python/library boundaries (e.g., accidentally converting `numpy` arrays into Python arrays, and vice versa).

### Other features

- Scalene can produce **reduced profiles** (via `--reduced-profile`) that only report lines that consume more than 1% of CPU or perform at least 100 allocations.
- Scalene supports `@profile` decorators to profile only specific functions.
- When Scalene is profiling a program launched in the background (via `&`), you can **suspend and resume profiling**.

# Comparison to Other Profilers

## Performance and Features

Below is a table comparing the **performance and features** of various profilers to Scalene.

![Performance and feature comparison](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/images/profiler-comparison.png)

- **Slowdown**: the slowdown when running a benchmark from the Pyperformance suite. Green means less than 2x overhead. Scalene's overhead is just a 35% slowdown.

Scalene has all of the following features, many of which only Scalene supports:

- **Lines or functions**: does the profiler report information only for entire functions, or for every line -- Scalene does both.
- **Unmodified Code**: works on unmodified code.
- **Threads**: supports Python threads.
- **Multiprocessing**: supports use of the `multiprocessing` library -- _Scalene only_
- **Python vs. C time**: breaks out time spent in Python vs. native code (e.g., libraries) -- _Scalene only_
- **System time**: breaks out system time (e.g., sleeping or performing I/O) -- _Scalene only_
- **Profiles memory**: reports memory consumption per line / function
- **GPU**: reports time spent on an NVIDIA GPU (if present) -- _Scalene only_
- **Memory trends**: reports memory use over time per line / function -- _Scalene only_
- **Copy volume**: reports megabytes being copied per second -- _Scalene only_
- **Detects leaks**: automatically pinpoints lines responsible for likely memory leaks -- _Scalene only_

## Output

If you include the `--cli` option, Scalene prints annotated source code for the program being profiled
(as text, JSON (`--json`), or HTML (`--html`)) and any modules it
uses in the same directory or subdirectories (you can optionally have
it `--profile-all` and only include files with at least a
`--cpu-percent-threshold` of time).  Here is a snippet from
`pystone.py`.

![Example profile](https://raw.githubusercontent.com/plasma-umass/scalene/master/docs/images/sample-profile-pystone.png)

* **Memory usage at the top**: Visualized by "sparklines", memory consumption over the runtime of the profiled code.
* **"Time Python"**: How much time was spent in Python code.
* **"native"**: How much time was spent in non-Python code (e.g., libraries written in C/C++).
* **"system"**: How much time was spent in the system (e.g., I/O).
* **"GPU"**: (not shown here) How much time spent on the GPU, if your system has an NVIDIA GPU installed.
* **"Memory Python"**: How much of the memory allocation happened on the Python side of the code, as opposed to in non-Python code (e.g., libraries written in C/C++).
* **"net"**: Positive net memory numbers indicate total memory allocation in megabytes; negative net memory numbers indicate memory reclamation.
* **"timeline / %"**: Visualized by "sparklines", memory consumption generated by this line over the program runtime, and the percentages of total memory activity this line represents.
* **"Copy (MB/s)"**: The amount of megabytes being copied per second (see "About Scalene").

##  Scalene

The following command runs Scalene on a provided example program.

```console
scalene test/testme.py
```

  Click to see all Scalene's options (available by running with --help)

```console
    % scalene --help
     usage: scalene [-h] [--outfile OUTFILE] [--html] [--reduced-profile]
                    [--profile-interval PROFILE_INTERVAL] [--cpu-only]
                    [--profile-all] [--profile-only PROFILE_ONLY]
                    [--use-virtual-time]
                    [--cpu-percent-threshold CPU_PERCENT_THRESHOLD]
                    [--cpu-sampling-rate CPU_SAMPLING_RATE]
                    [--malloc-threshold MALLOC_THRESHOLD]

     Scalene: a high-precision CPU and memory profiler.
     https://github.com/plasma-umass/scalene

     command-line:
        % scalene [options] yourprogram.py
     or
        % python3 -m scalene [options] yourprogram.py

     in Jupyter, line mode:
        %scrun [options] statement

     in Jupyter, cell mode:
        %%scalene [options]
        code...
        code...

     optional arguments:
       -h, --help            show this help message and exit
       --outfile OUTFILE     file to hold profiler output (default: stdout)
       --html                output as HTML (default: text)
       --reduced-profile     generate a reduced profile, with non-zero lines only (default: False)
       --profile-interval PROFILE_INTERVAL
                             output profiles every so many seconds (default: inf)
       --cpu-only            only profile CPU time (default: profile CPU, memory, and copying)
       --profile-all         profile all executed code, not just the target program (default: only the target program)
       --profile-only PROFILE_ONLY
                             profile only code in filenames that contain the given strings, separated by commas (default: no restrictions)
       --use-virtual-time    measure only CPU time, not time spent in I/O or blocking (default: False)
       --cpu-percent-threshold CPU_PERCENT_THRESHOLD
                             only report profiles with at least this percent of CPU time (default: 1%)
       --cpu-sampling-rate CPU_SAMPLING_RATE
                             CPU sampling rate (default: every 0.01s)
       --malloc-threshold MALLOC_THRESHOLD
                             only report profiles with at least this many allocations (default: 100)

     When running Scalene in the background, you can suspend/resume profiling
     for the process ID that Scalene reports. For example:

        % python3 -m scalene [options] yourprogram.py &
      Scalene now profiling process 12345
        to suspend profiling: python3 -m scalene.profile --off --pid 12345
        to resume profiling:  python3 -m scalene.profile --on  --pid 12345
```

### Scalene with Jupyter

Instructions for installing and using Scalene with Jupyter notebooks

[This notebook](https://nbviewer.jupyter.org/github/plasma-umass/scalene/blob/master/docs/scalene-demo.ipynb) illustrates the use of Scalene in Jupyter.

Installation:

```console
!pip install scalene
%load_ext scalene
```

Line mode:

```console
%scrun [options] statement
```

Cell mode:

```console
%%scalene [options]
code...
code...
```

## Installation

Using pip (Mac OS X, Linux, Windows, and WSL2)

Scalene is distributed as a `pip` package and works on Mac OS X, Linux (including Ubuntu in [Windows WSL2](https://docs.microsoft.com/en-us/windows/wsl/wsl2-index)) and (with limitations) Windows platforms.

> **Note**
>
> The Windows version currently only supports CPU and GPU profiling, but not memory or copy profiling.
>

You can install it as follows:
```console
  % pip install -U scalene
```

or
```console
  % python3 -m pip install -U scalene
```

You may need to install some packages first.

See https://stackoverflow.com/a/19344978/4954434 for full instructions for all Linux flavors.

For Ubuntu/Debian:

```console
  % sudo apt install git python3-all-dev
```

Using conda (Mac OS X, Linux, Windows, and WSL2)

```console
  % conda install -c conda-forge scalene
```

Scalene is distributed as a `conda` package and works on Mac OS X, Linux (including Ubuntu in [Windows WSL2](https://docs.microsoft.com/en-us/windows/wsl/wsl2-index)) and (with limitations) Windows platforms.

> **Note**
>
> The Windows version currently only supports CPU and GPU profiling, but not memory or copy profiling.
>

On ArchLinux

You can install Scalene on Arch Linux via the [AUR
package](https://aur.archlinux.org/packages/python-scalene-git/). Use your favorite AUR helper, or
manually download the `PKGBUILD` and run `makepkg -cirs` to build. Note that this will place
`libscalene.so` in `/usr/lib`; modify the below usage instructions accordingly.

# Frequently Asked Questions

Can I use Scalene with PyTest?

**A:** Yes! You can run it as follows (for example):

`python3 -m scalene --- -m pytest your_test.py`

Is there any way to get shorter profiles or do more targeted profiling?

**A:** Yes! There are several options:

1. Use `--reduced-profile` to include only lines and files with memory/CPU/GPU activity.
2. Use `--profile-only` to include only filenames containing specific strings (as in, `--profile-only foo,bar,baz`).
3. Decorate functions of interest with `@profile` to have Scalene report _only_ those functions.
4. Turn profiling on and off programmatically by importing Scalene profiler (`from scalene import scalene_profiler`) and then turning profiling on and off via `scalene_profiler.start()` and `scalene_profiler.stop()`. By default, Scalene runs with profiling on, so to delay profiling until desired, use the `--off` command-line option (`python3 -m scalene --off yourprogram.py`).

How do I run Scalene in PyCharm?

**A:**  In PyCharm, you can run Scalene at the command line by opening the terminal at the bottom of the IDE and running a Scalene command (e.g., `python -m scalene <your program>`). Use the options `--cli`, `--html`, and `--outfile <your output.html>` to generate an HTML file that you can then view in the IDE.

How do I use Scalene with Django?

**A:** Pass in the `--noreload` option (see https://github.com/plasma-umass/scalene/issues/178).

Does Scalene work with gevent/Greenlets?

**A:** Yes! Put the following code in the beginning of your program, or modify the call to `monkey.patch_all` as below:

```python
from gevent import monkey
monkey.patch_all(thread=False)
```

How do I use Scalene with PyTorch on the Mac?

**A:** Scalene works with PyTorch version 1.5.1 on Mac OS X. There's a bug in newer versions of PyTorch (https://github.com/pytorch/pytorch/issues/57185) that interferes with Scalene (discussion here: https://github.com/plasma-umass/scalene/issues/110), but only on Macs.

# Technical Information

For details about how Scalene works, please see the following paper, which won the Jay Lepreau Best Paper Award at [OSDI 2023](https://www.usenix.org/conference/osdi23/presentation/berger): [Triangulating Python Performance Issues with Scalene](https://arxiv.org/pdf/2212.07597). (Note that this paper does not include information about the AI-driven proposed optimizations.)

To cite Scalene in an academic paper, please use the following:

```latex
@inproceedings{288540,
author = {Emery D. Berger and Sam Stern and Juan Altmayer Pizzorno},
title = {Triangulating Python Performance Issues with {S}calene},
booktitle = {{17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)}},
year = {2023},
isbn = {978-1-939133-34-2},
address = {Boston, MA},
pages = {51--64},
url = {https://www.usenix.org/conference/osdi23/presentation/berger},
publisher = {USENIX Association},
month = jul
}
```

# Success Stories

If you use Scalene to successfully debug a performance problem, please [add a comment to this issue](https://github.com/plasma-umass/scalene/issues/58)!

# Acknowledgements

Logo created by [Sophia Berger](https://www.linkedin.com/in/sophia-berger/).

This material is based upon work supported by the National Science
Foundation under Grant No. 1955610. Any opinions, findings, and
conclusions or recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of the National
Science Foundation.


# Presidio - Data Protection and De-identification SDK

**Context aware, pluggable and customizable PII de-identification service for text and images.**

---

[![Build Status](https://dev.azure.com/csedevil/Presidio/_apis/build/status/Presidio-CI%20V2?branchName=main)](https://dev.azure.com/csedevil/Presidio/_build/latest?definitionId=212&branchName=main)
[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT)
![Release](https://img.shields.io/github/release/Microsoft/presidio.svg)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6076/badge)](https://bestpractices.coreinfrastructure.org/projects/6076)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/presidio-analyzer.svg)](https://pypi.python.org/pypi/presidio-analyzer/)

- Presidio Analyzer [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-analyzer.svg)](https://img.shields.io/pypi/dm/presidio-analyzer.svg)
- Presidio Anonymizer [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-anonymizer.svg)](https://img.shields.io/pypi/dm/presidio-anonymizer.svg)
- Presidio Image-Redactor [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-image-redactor.svg)](https://img.shields.io/pypi/dm/presidio-image-redactor.svg)
- Presidio Structured [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-structured.svg)](https://img.shields.io/pypi/dm/presidio-structured.svg)
## What is Presidio

Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to ensure sensitive data is properly managed and governed. It provides fast **_identification_** and **_anonymization_** modules for private entities in text such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more.

![Presidio demo gif](docs/assets/changing_text.gif)

---

### :blue_book: [Full documentation](https://microsoft.github.io/presidio)

### :question: [Frequently Asked Questions](docs/faq.md)

### :thought_balloon: [Demo](https://aka.ms/presidio-demo)

### :flight_departure: [Examples](https://microsoft.github.io/presidio/samples/)

---

## Are you using Presidio? We'd love to know how

Please help us improve by taking [this short anonymous survey](https://forms.office.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR9LagCGNW01LpMix2pnFWFJUQjJDTVkwSlJYRkFPSUNNVlVRRVRWVDVNSy4u).

---

### Goals

- Allow organizations to preserve privacy in a simpler way by democratizing de-identification technologies and introducing transparency in decisions.
- Embrace extensibility and customizability to a specific business need.
- Facilitate both fully automated and semi-automated PII de-identification flows on multiple platforms.

### Main features

1. **Predefined** or **custom PII recognizers** leveraging _Named Entity Recognition_, _regular expressions_, _rule based logic_ and _checksum_ with relevant context in multiple languages.
2. Options for connecting to external PII detection models.
3. Multiple usage options, **from Python or PySpark workloads through Docker to Kubernetes**.
4. **Customizability** in PII identification and de-identification.
5. Module for **redacting PII text in images** (standard image types and DICOM medical images).

:warning: Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed.

## Installing Presidio

1. [Using pip](https://microsoft.github.io/presidio/installation/#using-pip)
2. [Using Docker](https://microsoft.github.io/presidio/installation/#using-docker)
3. [From source](https://microsoft.github.io/presidio/installation/#install-from-source)
4. [Migrating from V1 to V2](./docs/presidio_V2.md)

## Running Presidio

1. [Getting started](https://microsoft.github.io/presidio/getting_started)
2. [Setting up a development environment](https://microsoft.github.io/presidio/development)
3. [PII de-identification in text](https://microsoft.github.io/presidio/text_anonymization)
4. [PII de-identification in images](https://microsoft.github.io/presidio/image-redactor)
5. [Usage samples and example deployments](https://microsoft.github.io/presidio/samples)

---

## Support

- Before you submit an issue, please go over the [documentation](https://microsoft.github.io/presidio/).
- For general discussions, please use the [Github repo's discussion board](https://github.com/microsoft/presidio/discussions).
- If you have a usage question, found a bug or have a suggestion for improvement, please file a [Github issue](https://github.com/microsoft/presidio/issues).
- For other matters, please email [presidio@microsoft.com](mailto:presidio@microsoft.com).

## Contributing

For details on contributing to this repository, see the [contributing guide](CONTRIBUTING.md).

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit [https://cla.microsoft.com](https://cla.microsoft.com).

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

## Contributors


# APEL client and server source code

[![Build Status](https://travis-ci.org/apel/apel.svg?branch=dev)](https://travis-ci.org/apel/apel)
[![Coverage Status](https://coveralls.io/repos/github/apel/apel/badge.svg?branch=dev)](https://coveralls.io/github/apel/apel?branch=dev)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/a8cdc36c46b241e6a77428fbfb6f23bd)](https://www.codacy.com/gh/apel/apel/dashboard)
[![Maintainability](https://api.codeclimate.com/v1/badges/03094b74f5fc4f728bc7/maintainability)](https://codeclimate.com/github/apel/apel/maintainability)

## Project overview

The APEL project provides grid accounting for EGI. It is written in
Python and uses MySQL. It has the following components:

### apel-parsers

These extract data from the following batch systems:
* LSF
* PBS
* SGE/OGE
* SLURM
* HTCondor

and place it in the client database.

### apel-client

This processes the data and sends it to the APEL server using SSM.

### apel-server

This processes data from all sites and sends it on to the accounting
portal using SSM.

### apel-lib

This contains all the library python code for apel-parsers, apel-client
and apel-server.

### apel-ssm

This is a messaging system, which has its own GitHub repository at
[apel/ssm](https://github.com/apel/ssm)

## Acknowledgements

APEL is provided by [STFC](https://stfc.ukri.org/), a part of [UK Research and Innovation](www.ukri.org), and is co-funded by the [EOSC-hub](https://www.eosc-hub.eu/) project (Horizon 2020) under Grant number 777536. Licensed under the [Apache 2 License](http://www.apache.org/licenses/LICENSE-2.0).


viur-core

    This is the core component of the ViUR framework.

## About

ViUR is an application development framework for the Google App Engine™.

ViUR was developed to meet the needs and requirements of both designers and developers. It provides a clear concept for the implementation of agile data management software systems. It is written in Python™ and has already attracted a growing community that is constantly supporting and improving ViUR.

## Getting started

To get started with ViUR, check out [viur-base](https://github.com/viur-framework/viur-base). It comes with a pre-configured and well documented project template to immediately start with.

## Migration

### from `<=v3.5` to `v3.6`
In [#833](https://github.com/viur-framework/viur-core/pull/833) the config has
changed from a dict to an object.
To migrate the access expressions like `conf["option"]` in your project
to `conf.option` the viur-core provides a migration script.
Install the _viur-core_ in your project, open a (virtual) environment shell
and `viur-core-migrate-config` will be available.
After checking the result with `viur-core-migrate-config ./deploy/ -d`
you can apply the changes with `viur-core-migrate-config ./deploy/ -x`.

## Contributing

Help of any kind to extend and improve or enhance this project in any kind or way is always appreciated.

We take great interest in your opinion about ViUR. We appreciate your feedback and are looking forward to hear about your ideas. Share your vision or questions with us and participate in ongoing discussions.

See our [contribution guidelines](CONTRIBUTING.md) for details.

## License

Copyright © 2024 by Mausbrand Informationssysteme GmbH.

Mausbrand and ViUR are registered trademarks of Mausbrand Informationssysteme GmbH.

Licensed under the MIT license. See LICENSE for more information.


# BIG-bench :chair:

The Beyond the Imitation Game Benchmark (BIG-bench) is a *collaborative*
benchmark intended to probe large language models and extrapolate their future
capabilities.
The *more than 200* tasks included in BIG-bench are summarized by keyword [here](bigbench/benchmark_tasks/keywords_to_tasks.md#summary-table), and by task name [here](bigbench/benchmark_tasks/README.md). A paper introducing the benchmark, including evaluation results on large language models, is currently under review, and is available as a [preprint](https://arxiv.org/abs/2206.04615).

The benchmark organizers can be contacted at bigbench@googlegroups.com.

**Table of contents**

* [BIG-bench Lite leaderboard](#big-bench-lite-leaderboard)
* [Quick start](#quick-start-colab-notebooks)
* [Installation](#installation-of-big-bench)
* [How do I create a task?](#how-do-i-create-a-task)
* [Creating a programmatic task](#creating-a-programmatic-task)
* [Submitting a model evaluation](#submitting-a-model-evaluation)
* [Frequently asked questions](#frequently-asked-questions)
* [Alan Turing sitting on a bench](#alan-turing-sitting-on-a-bench)

For more details about the benchmark, see our [detailed instructions](docs/doc.md).

## BIG-bench Lite leaderboard

BIG-bench Lite (BBL) is a small subset of [24 diverse JSON tasks](bigbench/benchmark_tasks/keywords_to_tasks.md#big-bench-lite) from BIG-bench.
It is designed to provide a canonical measure of model performance, while being far
cheaper to evaluate than the [full set](bigbench/benchmark_tasks/README.md) of more than 200 programmatic and JSON tasks in BIG-bench.
A leaderboard of current model performance on BBL is shown below.
To add new model results to the full BIG-bench leaderboard, to the BBL leaderboard, and to individual task performance plots, open a PR which includes the score files generated when you [evaluate your model](docs/doc.md#testing-and-submitting) on BIG-bench tasks.

## Citation

If you use BIG-bench in a publication, please cite our analysis paper:

```bibtex
# TMLR
@article{srivastava2023beyond,
  title={Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models},
  author={BIG-bench authors},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2023},
  url={https://openreview.net/forum?id=uyTL5Bvosj},
  note={}
}
```

## Quick start Colab notebooks

**Using [SeqIO](https://github.com/google/seqio) to inspect and evaluate BIG-bench json tasks**:
- [load BIG-bench json tasks and inspect examples](https://colab.research.google.com/github/google/BIG-bench/blob/main/bigbench/bbseqio/docs/quick_start.ipynb)

- [evaluate a t5x model on a BIG-bench json task](https://colab.research.google.com/github/google/BIG-bench/blob/main/bigbench/bbseqio/docs/t5x_eval.ipynb)

**Creating new BIG-bench tasks**

- [lightweight task creation and evaluation](https://colab.research.google.com/github/google/BIG-bench/blob/main/notebooks/colab_examples.ipynb)
- [manually perform BIG-bench tasks](https://colab.research.google.com/github/google/BIG-bench/blob/main/notebooks/TaskTestingNotebook.ipynb)  (after creating a task in your own branch, use this notebook to manually evaluate and verify that it is behaving correctly)

## Quick start instructions to load BIG-bench json tasks using [SeqIO](https://github.com/google/seqio) :chair:+:telescope:

```
!pip install git+https://github.com/google/BIG-bench.git # This may take a few minutes

import seqio
from bigbench.bbseqio import tasks

# See all subtasks
bb_mix = seqio.get_mixture_or_task("bigbench:all_json.mix.t5_default_vocab.0_shot.all_examples")
all_subtasks = [t.name for t in bb_mix.tasks]
print("\n".join(all_subtasks[:7]) + "\n...")

# Inspect an example task
task = seqio.get_mixture_or_task("bigbench:simple_arithmetic_json.gen.t5_default_vocab.0_shot.all_examples")
ds = task.get_dataset(split="all", sequence_length={"inputs": 32, "targets": 32})
print(next(iter(ds)))
```

Note: SeqIO currently only supports loading BIG-bench tasks defined via JSON, and not the programmatic tasks.

See the BIG-bench SeqIO [README](https://github.com/google/BIG-bench/blob/main/bigbench/bbseqio/README.md) for more details.

## Installation of BIG-bench

**Requirements**

* Python 3.5-3.8
* pytest (for running the automated tests)

**Instructions**

First, clone the repository and set up the environment.
```bash
# When creating a new task, replace this with your forked repository (see below)
git clone https://github.com/google/BIG-bench.git
cd BIG-bench
python setup.py sdist
pip install -e .
```

Next, run the automatic tests to ensure everything works.
```bash
pytest -s
```

## How do I create a task?

The benchmark contains two types of tasks: simplified tasks which are defined by a JSON file, and programmatic tasks that can interact with language models in more sophisticated ways. In this section and the next we will walk through how to create both types of tasks. More details are available in [docs/doc.md](docs/doc.md#How-to-submit-a-task-to-BIG-Bench). Before creating a task, we strongly recommend reading the [review criteria for accepting task submissions](docs/doc.md#review-criteria-for-submissions).

Each task is contained in a folder `bigbench/benchmark_tasks/TASK_NAME`, with a `README.md` (or `README.pdf`) file describing the task, and a `task.json` or `task.py` file defining the task. In this section we walk through creating a JSON-based task, for which the task data is specified in a `task.json` file.

### Setup

First, [fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo) in GitHub! :fork_and_knife:

Your fork will have its own location, which we will call `PATH_TO_YOUR_FORK`.
Next, [clone the forked repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) and create a branch for your new task, which here we will call **my_awesome_json_task**:
```bash
git clone $PATH_TO_YOUR_FORK
cd BIG-bench
git checkout -b my_awesome_json_task
```
We will base our task on an existing example.
Create a new task directory by copying over the example task:
```bash
cd bigbench/benchmark_tasks
cp -r simple_arithmetic_json my_awesome_json_task
cd my_awesome_json_task
```

### Creating a JSON task

We recommend starting with one of the JSON template tasks. As described in the previous section, [simple_arithmetic_json](bigbench/benchmark_tasks/simple_arithmetic_json) is a good choice.

The task consists of two files: `README.md` describes the task, and `task.json` contains the task data.
We will now edit both files with the new task information.

:warning: All task files (including `README.md` and `task.json`) contain a "canary" string, which should not be edited. This is to prevent benchmark tasks from leaking into web-scraped training data. :warning:

First, edit `README.md` to include the correct author and task information.
Second, edit `task.json` and update the following fields. See the [JSON schema](docs/doc.md#json-schema) for more details.

| Field             | Description                                                                                |
| ----------        | -----------                                                                                |
| `name`            | A short, human-readable task name. In this example we use `"My awesome task"`.                                                                                |
| `description`     | A short human-readable description for your task.                                                                                |
| `keywords`        | Keywords describing the task. See [here](keywords.md) for suggested keywords.                                                                                |
| `metrics`         | The metrics to be used in evaluation, such as `"exact_str_match"`. See [here](docs/doc.md#available-metrics) for a description of available metrics. |
| `preferred_score` | Which metric from `metrics` to prefer when reporting the task evaluation results.                                                                                |
| `examples`        | Add your task's input/output examples here!                                                                                |

Task examples can be specified in one of two ways:

* As a single input/target-output pair, where the target output will be compared against the model output.
For example: `{"input": "1 + 1 = ", "target": "2"}`. Multiple valid targets (specified as a list) are allowed for all metrics, for example `{"input": "1 + 1 = ", "target": ["two","2"]}`.

* As an input and multiple-choice targets, where the target scores will be compared against the model's predicted probabilities.
For example: `{"input": "1 + 1 = ", "target_scores": { "1": 0, "2": 1, "3": 0, "4": 0 }}`

The task this example is based on uses the input/output format.
For an example of a multiple choice task, please see [simple_arithmetic_json_multiple_choice](bigbench/benchmark_tasks/simple_arithmetic_json_multiple_choice/task.json).
For a full description of the supported formats and metrics for json tasks see [here](docs/doc.md#available-metrics).

**Standardizing styles**
We use the [black](https://github.com/psf/black) code formatter to enforce standardized styles in your task submission via a pre-commit hook. To use this hook, install `pre-commit` with `pip install pre-commit` (installed by default if you've followed the above instructions). Then run `pre-commit install` to install the hook. On future commits, you should see the `black` code formatter is run on all python files you've staged for commit.

**Testing and evaluating**

Once the task data is ready, test it:
```bash
cd ../../..  # get to BIG-bench/
pytest -s bigbench/api/test_tasks.py --tasks my_awesome_json_task
```

To see how a few publicly-available models perform on your task, run the following evaluation script. Depending on your hardware and the size of your task, this may take a long time!
```bash
scripts/evaluate_task.sh --task my_awesome_json_task --output_dir results
```

### Submitting

Once the tests pass and you are happy with the task, submit your task for review.
First, commit and push your changes:
```bash
git add bigbench/benchmark_tasks/my_awesome_json_task/*
git commit -m "Added my_awesome_json_task"
git push --set-upstream origin my_awesome_json_task
```
Finally, [submit a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).
The last `git push` command prints a URL that can be copied into a browser to initiate such a pull request.
Alternatively, you can do so from the GitHub website.

:sparkles: Congratulations, you've submitted a task to BIG-bench! :sparkles:

## Creating a programmatic task

Programmatic tasks are defined in code, and allow for more sophisticated interaction with the evaluated language models.
For example, using programmatic tasks one can query a model over several rounds, using each of the model's responses to inform the next query.

For programmatic tasks, we recommend starting from the template task [simple_arithmetic](bigbench/benchmark_tasks/simple_arithmetic).

Similarly to JSON tasks, programmatic tasks are contained in a folder `bigbench/benchmark_tasks/TASK_NAME`, with a `README.md` or `README.pdf` describing the task. Unlike JSON tasks, programmatic tasks are defined by a `task.py` Python file rather than a `task.json` file. In this section we will create a programmatic task called **my_awesome_task**, based on the example task in `benchmark_tasks/simple_arithmetic`.

Start by forking the repository :fork_and_knife:, cloning it :dancers:, and creating a branch :herb: as described [above](#creating-a-json-task).
Then, from the directory `bigbench/benchmark_tasks`, copy the example task:
```bash
cp -r simple_arithmetic my_awesome_task
cd my_awesome_task
```

### Creating the task

Edit `README.md` to include the correct author and task information.
Then, edit `task.py` as described below. See the [API documentation](https://google.github.io/BIG-bench/docs/html/bigbench/index.html) for more details.
⚠⚠⚠ NOTE: There is an incoming change to the function signature of programmatic tasks.  [See this pull request](https://github.com/google/BIG-bench/pull/288) for more details.  If you are developing a programmatic task, be sure to include support for the new functionality in this pull request in your task. ⚠⚠⚠

* Rename the main task class from `ArithmeticTask` to `MyAwesomeTask`.

* The `get_task_details` method returns task metadata. Change the returned values including `name`, `description`, `keywords`, `max_input_length_per_query`, and `max_queries` as appropriate for your task. See [here](keywords.md) for a list of suggested keywords.

* The meat of the task is the `evaluate_model` method, which evaluates a given model on your task. The model can be queried using two methods of the `model` argument passed to `evaluate_model`:
  * `model.generate_text()`: Generate text for given inputs.
  * `model.cond_log_prob()`: Compute the probabilities of provided model outputs for given inputs.

* Implement `evaluate_model` for your task, returning a `ScoreData` object (or list of such objects) containing the evaluation scores. If you require a standard NLP metric, please use the T5 implementations found [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/evaluation/metrics.py). The T5 package will be available during runtime.

:warning: Do not edit the `canary` comments. These are to prevent BIG-bench tasks from leaking into web-scraped training data. :warning:

### Testing and evaluating

Test your new task:
```bash
cd ../../..  # get to BIG-bench/
pytest -s bigbench/api/test_tasks.py --tasks my_awesome_task
```

You can use the following script to evaluate your task on a few publicly-available models.
The evaluation results are written to a JSON file.
```bash
scripts/evaluate_task.sh --task my_awesome_task --output_dir results
```
:warning: Depending on your hardware and the size of your task, this may take a long time! :warning:

### Submitting

Once the tests pass and you are ready to submit your task, commit and push your changes:
```bash
git add bigbench/benchmark_tasks/my_awesome_task/*
git commit -m "Added my_awesome_task"
git push --set-upstream origin my_awesome_task
```

Finally, submit your task for review by creating pull request as described [above](#creating-a-json-task).

:sparkles: Congratulations, you've submitted a task to BIG-bench! :sparkles:

## Submitting a Model Evaluation

You can submit a BIG-bench evaluation for your model to the repository. Your model results will then appear
in the performance plots and be available for other researchers. To submit your evaluation results, follow
the detailed instructions [here](docs/model_eval_contribution.md).

## Frequently asked questions

### Can I submit non-English tasks?

Yes! Challenging language models to perform well on different languages is well within the scope of this benchmark.

### Can I use external package X?

The packages we already support are listed [here](docs/doc.md#external-dependencies). We are open to supporting additional packages that enable new types of tasks, but we will need to evaluate on a case-by-case basis to ensure compatibility with our internal tooling. If you have a package not listed above that you would like us to support, [shoot us an email](mailto:bigbench@googlegroups.com).

### I signed the CLA but the Google bot keeps complaining!

A common problem happens if the CLA is signed using one email address, while changes are committed from another email address that is associated with the same user. Please see [these instructions](https://opensource.google/docs/cla/#wrong-email) for a way to resolve this issue.

### Who should I contact regarding copyright inquiries?

For questions related to copyright and licensing, please contact [big-bench-copyright@google.com](mailto:big-bench-copyright@google.com).

## NL-augmenter 🦎 → 🐍

If you are interested in BIG-bench, you may also be interested in [NL-Augmenter](https://github.com/GEM-benchmark/NL-Augmenter), a large scale collaborative effort to develop transformations of datasets dealing with natural language.

## Alan Turing sitting on a bench

Photograph by Hamish MacPherson at the Alan Turing memorial in Sackville Park, https://commons.wikimedia.org/wiki/File:Alan_Turing_Memorial_Sackville_Park.jpg