<img src="https://raw.githubusercontent.com/opensearch-project/opensearch-build/main/opensearch_build_image.png" height="64px" alt="OpenSearch Build">

[![python](https://github.com/opensearch-project/opensearch-build/actions/workflows/python-tests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/python-tests.yml)
[![groovy](https://github.com/opensearch-project/opensearch-build/actions/workflows/groovy-tests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/groovy-tests.yml)
[![manifests](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml)
[![codecov](https://codecov.io/gh/opensearch-project/opensearch-build/branch/main/graph/badge.svg?token=03S5XZ80UI)](https://codecov.io/gh/opensearch-project/opensearch-build)

- [Releasing OpenSearch](#releasing-opensearch)
  - [Releases and Versions](#releases-and-versions)
  - [Release labels](#release-labels)
- [Onboarding a New Plugin](#onboarding-a-new-plugin)
- [Building and Testing an OpenSearch Distribution](#building-and-testing-an-opensearch-distribution)
  - [Building from Source](#building-from-source)
  - [Assembling a Distribution](#assembling-a-distribution)
  - [Building Patches](#building-patches)
  - [Min snapshots](#min-snapshots)
  - [CI/CD Environment](#cicd-environment)
  - [Build Numbers](#build-numbers)
  - [Latest Distribution URL](#latest-distribution-url)
  - [Testing the Distribution](#testing-the-distribution)
  - [Checking Release Notes](#checking-release-notes)
  - [Signing Artifacts](#signing-artifacts)
    - [PGP](#pgp)
    - [Windows](#windows)
  - [Signing RPM artifacts](#signing-rpm-artifacts)
- [Making a Release](#making-a-release)
  - [Releasing for Linux](#releasing-for-linux)
  - [Releasing for FreeBSD](#releasing-for-freebsd)
  - [Releasing for Windows](#releasing-for-windows)
  - [Releasing for macOS](#releasing-for-macos)
- [Utilities](#utilities)
  - [Checking Out Source](#checking-out-source)
  - [Cross-Platform Builds](#cross-platform-builds)
  - [Sanity Checking the Bundle](#sanity-checking-the-bundle)
  - [Auto-Generating Manifests](#auto-generating-manifests)
- [Deploying Infrastructure](#deploying-infrastructure)
- [Contributing](#contributing)
- [Getting Help](#getting-help)
- [Code of Conduct](#code-of-conduct)
- [Security](#security)
- [License](#license)
- [Copyright](#copyright)

### Releasing OpenSearch

Please refer to the [release process document](https://github.com/opensearch-project/opensearch-build/wiki/Releasing-the-Distribution) for detailed information on how to release the OpenSearch and OpenSearch Dashboards software.

#### Releases and Versions

The OpenSearch project releases as versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. It [follows semantic versioning](https://opensearch.org/blog/technical-post/2021/08/what-is-semver/). Software, such as Data Prepper, clients, and the Logstash output plugin, are versioned independently of the OpenSearch Project. They also may have independent releases from the main project distributions. The OpenSearch Project may also release software under alpha, beta, release candidate, and generally available labels. The definition of when to use these labels is derived from [the Wikipedia page on Software release lifecycle](https://en.wikipedia.org/wiki/Software_release_life_cycle). Below is the definition of when to use each label.

#### Release labels:

* **Alpha** - The code is released with instructions to build. Built distributions of the software may not be available. Some features many not be complete. Additional testing and development work is planned. Distributions will be postfixed with `-alphaX` where "X" is the number of the alpha version  (e.g., "2.0-alpha1").
* **Beta** - Built distributions of the software are available. All features are completed. Additional testing and development work is planned. Distributions will be postfixed with `-betaX` where "X" is the number of the beta version  (e.g., "2.0.0-beta1").
* **Release Candidate** - Built distributions of the software are available. All features are completed. Code is tested and minimal validation remains. At this stage the software is potentially stable and will release unless significant bugs emerge. Distributions will be postfixed with `-rcX` where "X" is the number of the release candidate version (e.g., "2.0.0-rc1").
* **Generally Available** - Built distributions of the software are available. All features are completed and documented. All testing is completed. Distributions for generally available versions are not postfixed with an additional label (e.g., "2.0.0").

### Onboarding a New Plugin

Plugin owners can follow the [Onboarding Process](ONBOARDING.md) to onboard their plugins to the release process.

### Building and Testing an OpenSearch Distribution

The distribution workflow builds a complete OpenSearch and OpenSearch Dashboards distribution from source. You can currently build 1.0, 1.1, 1.1-SNAPSHOT and 1.2 versions. This system performs a top-down [build](src/build_workflow) of all components required for a specific OpenSearch and OpenSearch Dashboards release, then [assembles](src/assemble_workflow/) a distribution. The input to the system is a manifest that defines the order in which components should be built. All manifests for our current releases are [here](manifests).

#### Building from Source

```bash
./build.sh manifests/1.3.0/opensearch-1.3.0.yml
```

This builds OpenSearch 1.3.0 from source, placing the output into `./builds/opensearch`.

See [build workflow](src/build_workflow) for more information.

#### Assembling a Distribution

```bash
./assemble.sh builds/opensearch/manifest.yml
```

The assembling step takes output from the build step, installs plugins, and assembles a full distribution into the `dist` folder.

See [assemble workflow](src/assemble_workflow) for more information.

#### Building Patches

A patch release contains output from previous versions mixed with new source code. Manifests can mix such references. See [opensearch-1.3.1.yml](https://github.com/opensearch-project/opensearch-build/blob/opensearch-1.3.1/manifests/1.3.1/opensearch-1.3.1.yml) for an example.

OpenSearch is often released with changes in `opensearch-min`, and no changes to plugins other than a version bump. This can be performed by a solo engineer following [a cookbook](https://github.com/opensearch-project/opensearch-plugins/blob/main/META.md#increment-a-version-in-every-plugin). See also [opensearch-build#1375](https://github.com/opensearch-project/opensearch-build/issues/1375) which aims to automate incrementing versions for the next development iteration.

#### Min Snapshots

Snapshots for OpenSearch core/min can be downloaded and used in CI's, local development, etc using below links:

Linux:
```
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-linux-x64-latest.tar.gz
```
macOS:
```
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-darwin-x64-latest.tar.gz
```

Windows:
```
https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-windows-x64-latest.zip
```

#### CI/CD Environment

We build, assemble, and test our artifacts on docker containers. We provide docker files in [docker/ci](docker/ci) folder, and images on [staging docker hub repositories](https://hub.docker.com/r/opensearchstaging/ci-runner/). All Jenkins pipelines can be found in [jenkins](./jenkins/). Jenkins itself is in the process of being made public and its CDK open-sourced.

See [jenkins](./jenkins) and [docker](./docker) for more information.

#### Build Numbers

The distribution URL and the build output manifest include a Jenkins auto-incremented build number. For example, the [manifest](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/5905/linux/x64/rpm/dist/opensearch/manifest.yml) from [OpenSearch build 5905](https://build.ci.opensearch.org/job/distribution-build-opensearch/5905/) contains the following.

```yml
build:
  name: OpenSearch
  version: 2.2.0
  platform: linux
  architecture: x64
  distribution: rpm
  id: '5905'
```

#### Latest Distribution URL

Use the `latest` keyword in the URL to obtain the latest build for a given version. For example `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/rpm/dist/opensearch/manifest.yml` redirects to [build 5905](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/5905/linux/x64/rpm/dist/opensearch/manifest.yml) at the time of writing this.

The `latest` keyword is resolved to a specific build number by checking the `index.json` [file](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/index.json). This file has contents such as this.

```
{"latest":"5905"}
```

The file is updated when a distribution build job is completed for the given product and version (or is created when such distribution job succeeds for the first time). Since one distribution build job consists of multiple stages for different combinations of distribution type, platform and architecture, the `index.json` is only modified once all stages succeed. With this said, the `latest` URL only works when the distribution build job succeeds at least once for the given product and version.

The resolution logic is implemented in the [CloudFront URL rewriter](https://github.com/opensearch-project/opensearch-ci/tree/main/resources/cf-url-rewriter).
The TTL (time to live) is set to `5 mins` which means that the `latest` URL may need up to 5 mins to get new contents after `index.json` is updated.

All the artifacts accessible through the regular distribution URL can be accessed by the `latest` URL. This includes both OpenSearch Core, OpenSearch Dashboards Core and their plugins.

For example, you can download the latest .tar.gz distribution build of OpenSearch 2.2.0 directly at `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.2.0-linux-x64.tar.gz`, without having to first download and parse the [complete build manifest](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/manifest.yml).

For plugin artifacts, you can also use the `latest` keyword to get the latest plugin artifacts with a known version. E.g. in order to get performance-analyzer x64 tarball artifacts for 2.1.0, you can obtain it with link `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip`, which will redirect you to `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip`.

For bundled artifacts, here are some examples for LINUX and Windows:
* Linux Tar: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz`
* Windows Zip: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/windows/x64/zip/dist/opensearch/opensearch-2.4.0-windows-x64.zip`

#### Testing the Distribution

Tests the OpenSearch distribution, including integration, backwards-compatibility and performance tests.

```bash
./test.sh <test-type> <test-manifest-path> <path>
```

See [src/test_workflow](./src/test_workflow) for more information.

#### Checking Release Notes

Workflow to check if the release notes exist or not and shows the latest commit for OpenSearch and Dashboard distributions.

To run:
```bash
./release_notes.sh check manifests/2.2.0/opensearch-2.2.0.yml --date 2022-07-26
```

See [src/release_notes_workflow](./src/release_notes_workflow) for more information.
#### Signing Artifacts

For all types of signing within OpenSearch project we use `opensearch-signer-client` (in progress of being open-sourced) which is a wrapper around internal signing system and is only available for authenticated users. The input requires a path to the build manifest or directory containing all the artifacts or a single artifact.

Usage:

```bash
./sign.sh builds/opensearch/manifest.yml
```

The tool currently supports following platforms for signing.

##### PGP

Anything can be signed using PGP signing eg: tarball, any type of file, etc. A `.sig` file will be returned containing the signature. OpenSearch and OpenSearch dashboards distributions, components such as data prepper, etc. as well as maven artifacts are signed using PGP signing. See [this page](https://opensearch.org/verify-signatures.html) for how to verify signatures.

##### Windows

Windows signing can be used to sign windows executables such as `.msi, .msp, .msm, .cab, .dll, .exe, .appx, .appxbundle, .msix, .msixbundle, .sys, .vxd, .ps1, .psm1`, and any PE file that is supported by [Signtool.exe](https://docs.microsoft.com/en-us/dotnet/framework/tools/signtool-exe). Various windows artifacts such as SQL OBDC, opensearch-cli, etc are signed using this method.
Windows code signing uses EV (Extended Validated) code signing certificates.

|  Types of signing/Details   | Digest           | Cipher  | Key Size|
| ------------- |:-------------| :-----| :-----|
| PGP      | SHA1 | AES-128 | 2048 |
| Windows      | SHA256      |    RSA | |
| RPM | SHA512      |    RSA | 4096 |

#### Signing RPM artifacts

RPM artifacts are signed via a legacy shell script which uses a [macros template](scripts/pkg/sign_templates/rpmmacros). See [this commit](https://github.com/opensearch-project/opensearch-build/commit/950d55c1ed3f82e98120541fa40ff506338c1059) for more information and [this issue](https://github.com/opensearch-project/opensearch-build/issues/1547) to add RPM artifact signing functionality to the above signing system. Currently we are only signing OpenSearch and OpenSearch dashboards RPM distributions using this method.

See [src/sign_workflow](./src/sign_workflow) for more information.

### Making a Release

#### Releasing for Linux and Windows

The Linux / Windows release is managed by a team at Amazon following [this release template](.github/ISSUE_TEMPLATE/release_template.md) (e.g. [opensearch-build#2649](https://github.com/opensearch-project/opensearch-build/issues/2649)).

#### Releasing for FreeBSD

The FreeBSD ports and packages for OpenSearch are managed by a community [OpenSearch Team](https://wiki.freebsd.org/OpenSearch) at FreeBSD.  When a new release is rolled out, this team will update the port and commit it to the FreeBSD ports tree. Anybody is welcome to help the team by providing patches for [upgrading the ports](https://docs.freebsd.org/en/books/porters-handbook/book/#port-upgrading) following the [FreeBSD Porter's Handbook](https://docs.freebsd.org/en/books/porters-handbook/book/) instructions.

#### Releasing for macOS

At this moment there's no official macOS distribution. However, this project does support building and assembling OpenSearch for macOS. See [opensearch-build#37](https://github.com/opensearch-project/opensearch-build/issues/37) and [#38](https://github.com/opensearch-project/opensearch-build/issues/38) for more details.

### Utilities

#### Checking Out Source

The [checkout workflow](src/checkout_workflow) checks out source code for a given manifest for further examination.

```bash
./checkout.sh manifests/1.3.0/opensearch-1.3.0.yml
```

See [src/checkout_workflow](./src/checkout_workflow) for more information.

#### Cross-Platform Builds

You can perform cross-platform builds. For example, build and assemble a Windows distribution on macOS.

```bash
export JAVA_HOME=$(/usr/libexec/java_home) # required by OpenSearch install-plugin during assemble
./build.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot --platform windows
./assemble.sh builds/opensearch/manifest.yml
```

This will produce `dist/opensearch-1.3.0-SNAPSHOT-windows-x64.zip` on Linux and macOS.

#### Sanity Checking the Bundle

This workflow runs sanity checks on every component present in the bundle, executed as part of the [manifests workflow](.github/workflows/manifests.yml) in this repository. It ensures that the component GitHub repositories are correct and versions in those components match the OpenSearch version.

The following example sanity-checks components in the OpenSearch 1.3.0 manifest.

```bash
./ci.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot
```

See [src/ci_workflow](./src/ci_workflow) for more information.

#### Auto-Generating Manifests

The [manifests workflow](src/manifests_workflow) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is found, a new input manifest is added to [manifests](manifests), and [a pull request is opened](.github/workflows/manifests.yml) (e.g. [opensearch-build#491](https://github.com/opensearch-project/opensearch-build/pull/491)).

Show information about existing manifests.

```bash
./manifests.sh list
```
Check for updates and create any new manifests.

```bash
./manifests.sh update
```

See [src/manifests_workflow](./src/manifests_workflow) for more information.

### Deploying Infrastructure

This project uses jenkins as the build infrastructure for building, testing and releasing the artifacts. The infrastructure is deployed using CDK and code can be found in [opensearch-ci](https://github.com/opensearch-project/opensearch-ci/tree/main) repository.

## Contributing

See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md).

## Getting Help

If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.

For more information, see [project website](https://opensearch.org/) and [documentation](https://opensearch.org/docs/). If you need help and are unsure where to open an issue, try [forums](https://discuss.opendistrocommunity.dev/).

## Code of Conduct

This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments.

## Security

If you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to security@opensearch.org. Please do **not** create a public GitHub issue.

## License

This project is licensed under the [Apache v2.0 License](LICENSE.txt).

## Copyright

Copyright OpenSearch Contributors. See [NOTICE](NOTICE) for details.


# <img src="https://cdn.rawgit.com/theupdateframework/artwork/3a649fa6/tuf-logo.svg" height="100" valign="middle" alt="TUF"/> A Framework for Securing Software Update Systems

![Build](https://github.com/theupdateframework/python-tuf/actions/workflows/ci.yml/badge.svg)
[![Coveralls](https://coveralls.io/repos/theupdateframework/python-tuf/badge.svg?branch=develop)](https://coveralls.io/r/theupdateframework/python-tuf?branch=develop)
[![Docs](https://readthedocs.org/projects/theupdateframework/badge/)](https://theupdateframework.readthedocs.io/)
[![CII](https://bestpractices.coreinfrastructure.org/projects/1351/badge)](https://bestpractices.coreinfrastructure.org/projects/1351)
[![PyPI](https://img.shields.io/pypi/v/tuf)](https://pypi.org/project/tuf/)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/theupdateframework/python-tuf/badge)](https://api.securityscorecards.dev/projects/github.com/theupdateframework/python-tuf)

----------------------------
[The Update Framework (TUF)](https://theupdateframework.io/) is a framework for
secure content delivery and updates. It protects against various types of
supply chain attacks and provides resilience to compromise. This repository is a
**reference implementation** written in Python. It is intended to conform to
version 1.0 of the [TUF
specification](https://theupdateframework.github.io/specification/latest/).

Python-TUF provides the following APIs:
  * [`tuf.api.metadata`](https://theupdateframework.readthedocs.io/en/latest/api/tuf.api.html),
    a "low-level" API, designed to provide easy and safe access to TUF
    metadata and to handle (de)serialization from/to files.
  * [`tuf.ngclient`](https://theupdateframework.readthedocs.io/en/latest/api/tuf.ngclient.html),
    a client implementation built on top of the metadata API.
  * `tuf.repository`, a repository library also built on top of the metadata
    API. This module is currently not considered part of python-tuf stable API.

The reference implementation strives to be a readable guide and demonstration
for those working on implementing TUF in their own languages, environments, or
update systems.

About The Update Framework
--------------------------
The Update Framework (TUF) design helps developers maintain the security of a
software update system, even against attackers that compromise the repository
or signing keys.
TUF provides a flexible
[specification](https://github.com/theupdateframework/specification/blob/master/tuf-spec.md)
defining functionality that developers can use in any software update system or
re-implement to fit their needs.

TUF is hosted by the [Linux Foundation](https://www.linuxfoundation.org/) as
part of the [Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF)
and its design is [used in production](https://theupdateframework.io/adoptions/)
by various tech companies and open source organizations. A variant of TUF
called [Uptane](https://uptane.github.io/) is used to secure over-the-air
updates in automobiles.

Please see [TUF's website](https://theupdateframework.com/) for more information about TUF!

Documentation
-------------
* [Introduction to TUF's Design](https://theupdateframework.io/overview/)
* [The TUF Specification](https://theupdateframework.github.io/specification/latest/)
* [Developer documentation](https://theupdateframework.readthedocs.io/), including
  [API reference](
    https://theupdateframework.readthedocs.io/en/latest/api/api-reference.html) and [instructions for contributors](https://theupdateframework.readthedocs.io/en/latest/CONTRIBUTING.html)
* [Usage examples](https://github.com/theupdateframework/python-tuf/tree/develop/examples/)
* [Governance](https://github.com/theupdateframework/python-tuf/blob/develop/docs/GOVERNANCE.md)
and [Maintainers](https://github.com/theupdateframework/python-tuf/blob/develop/docs/MAINTAINERS.txt)
for the reference implementation
* [Miscellaneous Docs](https://github.com/theupdateframework/python-tuf/tree/develop/docs)
* [Python-TUF development blog](https://theupdateframework.github.io/python-tuf/)

Contact
-------
Questions, feedback, and suggestions are welcomed on our low volume [mailing
list](https://groups.google.com/forum/?fromgroups#!forum/theupdateframework) or
the [#tuf](https://cloud-native.slack.com/archives/C8NMD3QJ3) channel on [CNCF
Slack](https://slack.cncf.io/).

We strive to make the specification easy to implement, so if you come across
any inconsistencies or experience any difficulty, do let us know by sending an
email, or by reporting an issue in the GitHub [specification
repo](https://github.com/theupdateframework/specification/issues).

Security Issues and Bugs
------------------------

See [SECURITY.md](docs/SECURITY.md)

License
-------

This work is [dual-licensed](https://en.wikipedia.org/wiki/Multi-licensing) and
distributed under the (1) MIT License and (2) Apache License, Version 2.0.
Please see [LICENSE-MIT](https://github.com/theupdateframework/python-tuf/blob/develop/LICENSE-MIT)
and [LICENSE](https://github.com/theupdateframework/python-tuf/blob/develop/LICENSE).

Acknowledgements
----------------

This project is hosted by the Linux Foundation under the Cloud Native Computing
Foundation.  TUF's early development was managed by members of the [Secure
Systems Lab](https://ssl.engineering.nyu.edu/) at [New York
University](https://engineering.nyu.edu/). We appreciate the efforts of all
[maintainers and emeritus
maintainers](https://github.com/theupdateframework/python-tuf/blob/develop/docs/MAINTAINERS.txt),
as well as the contributors Konstantin Andrianov, Kairo de Araujo, Ivana
Atanasova, Geremy Condra, Zane Fisher, Pankhuri Goyal, Justin Samuel, Tian
Tian, Martin Vrachev and Yuyu Zheng who are among those who helped
significantly with TUF's reference implementation. Maintainers and Contributors
are governed by the [CNCF Community Code of
Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

This material is based upon work supported by the National Science Foundation
under Grant Nos. CNS-1345049 and CNS-0959138. Any opinions, findings, and
conclusions or recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the National Science
Foundation.


.. image:: https://raw.githubusercontent.com/nextcord/nextcord/master/assets/repo-banner.svg
   :alt: Nextcord

.. image:: https://img.shields.io/discord/881118111967883295?color=blue&label=discord
   :target: https://discord.gg/nextcord
   :alt: Discord server invite
.. image:: https://img.shields.io/pypi/v/nextcord.svg
   :target: https://pypi.org/project/nextcord/
   :alt: PyPI version info
.. image:: https://img.shields.io/pypi/dm/nextcord?color=informational&label=pypi%20downloads
   :target: https://pypi.org/project/nextcord/
   :alt: PyPI version info
.. image:: https://img.shields.io/pypi/pyversions/nextcord.svg
   :target: https://pypi.org/project/nextcord/
   :alt: PyPI supported Python versions
.. image:: https://img.shields.io/readthedocs/nextcord
   :target: https://docs.nextcord.dev/
   :alt: Nextcord documentation

Nextcord
--------

A modern, easy-to-use, feature-rich, and async-ready API wrapper for Discord written in Python.

Key Features
-------------

- Modern Pythonic API using ``async`` and ``await``
- Proper rate limit handling
- Optimised in both speed and memory

Installing
----------

**Python 3.8 or higher is required**

To install the library without full voice support, you can just run the following command:

.. code:: sh

    # Linux/macOS
    python3 -m pip install -U nextcord

    # Windows
    py -3 -m pip install -U nextcord

Otherwise to get voice support you should run the following command:

.. code:: sh

    # Linux/macOS
    python3 -m pip install -U "nextcord[voice]"

    # Windows
    py -3 -m pip install -U nextcord[voice]

To install additional packages for speedup, run the following command:

.. code:: sh

    # Linux/macOS
    python3 -m pip install -U "nextcord[speed]"

    # Windows
    py -3 -m pip install -U nextcord[speed]

To install the development version, do the following:

.. code:: sh

    $ git clone https://github.com/nextcord/nextcord/
    $ cd nextcord
    $ python3 -m pip install -U .[voice]

Optional Packages
~~~~~~~~~~~~~~~~~~

* `PyNaCl <https://pypi.org/project/PyNaCl/>`__ (for voice support)
* `aiodns <https://pypi.org/project/aiodns/>`__, `Brotli <https://pypi.org/project/Brotli/>`__, `cchardet <https://pypi.org/project/cchardet/>`__ (for aiohttp speedup)
* `orjson <https://pypi.org/project/orjson/>`__ (for json speedup)

Please note that on Linux installing voice you must install the following packages via your favourite package manager (e.g. ``apt``, ``dnf``, etc) before running the above commands:

* libffi-dev (or ``libffi-devel`` on some systems)
* python-dev (e.g. ``python3.8-dev`` for Python 3.8)

Quick Example
~~~~~~~~~~~~~

.. code:: py

    import nextcord
    from nextcord.ext import commands

    bot = commands.Bot()

    @bot.slash_command(description="Replies with pong!")
    async def ping(interaction: nextcord.Interaction):
        await interaction.send("Pong!", ephemeral=True)

    bot.run("token")

You can find more examples in the `examples directory <https://github.com/nextcord/nextcord/blob/master/examples/>`_.

**NOTE:** It is not advised to leave your token directly in your code, as it allows anyone with it to access your bot. If you intend to make your code public you should `store it securely <https://github.com/nextcord/nextcord/blob/master/examples/secure_token_storage.py/>`_.

Links
------

- `Documentation <https://docs.nextcord.dev/>`_
- `Official Discord Server <https://discord.gg/nextcord>`_
- `Discord Developers Server <https://discord.gg/discord-developers>`_


# Notifications-admin

Notifications admin application.

## Upstream

This repo is a clone / modifed version of:
https://github.com/alphagov/notifications-admin

## Features of this application

 - Register and manage users
 - Create and manage services
 - Send batch emails and SMS by uploading a CSV
 - Show history of notifications

## Functional constraints

- We currently do not support sending of letters
- We currently do not receive a response if text messages were delivered or not

## First-time setup

[Brew is a package manager](https://brew.sh/) for OSX. The following command installs brew:
```shell
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

Languages needed
- Python 3.10
- [Node](https://nodejs.org/) 10.15.3 or greater
- [npm](https://www.npmjs.com/) 6.4.1 or greater

```shell
    brew install node
```

[NPM](npmjs.org) is Node's package management tool. `n` is a tool for managing
different versions of Node. The following installs `n` and uses the long term support (LTS)
version of Node.

```shell
    npm install -g n
    n lts
    npm rebuild node-sass
```

### Local installation instruction

On OS X:

1. Install PyEnv with Homebrew. This will preserve your sanity.

`brew install pyenv`

2. Install Python 3.10.8 or whatever is the latest

`pyenv install 3.10.8`

3. If you expect no conflicts, set `3.10.8` as you default

`pyenv global 3.10.8`

4. Ensure that version `3.10.8` is now the default by running

`python --version`

If it did not, add to your shell rc file. ex: `.bashrc` or `.zshrc`
```
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
```
and open a new terminal.

If you are still not running Python 3.10.8 take a look here: https://github.com/pyenv/pyenv/issues/660

5. Install `poetry`:

`pip install poetry==1.3.2`

6. Restart your terminal and make your virtual environtment:

`mkvirtualenv -p ~/.pyenv/versions/3.10.8/bin/python notifications-admin`

7. You can now return to your environment any time by entering

`workon notifications-admin`

8. Find the appropriate env variables and copy them into the .env file. A sane set of defaults exists in `.env.example` in the root folder. If you are working for CDS you should use the ones in the LastPass folder. If using from lastPass and running the API locally, change API_HOST_NAME to point to your local machine

9. Install all dependencies

`pip3 install -r requirements.txt`

10. Generate the version file ?!?

`make generate-version-file`

11. Generate the translations

`make babel`

12. Install npm and build the assets

`npm install` followed by `npm run build`

13.  Run the service

`flask run -p 6012 --host=localhost`

14. To test

`pip3 install -r requirements_for_test.txt`

`make test`

## Rebuilding the frontend assets

If you want the front end assets to re-compile on changes, leave this running
in a separate terminal from the app

```shell
    npm run watch
```

## Updating application dependencies

`poetry.lock` file is generated from the `pyproject.toml` in order to pin
versions of all nested dependencies. If `pyproject.toml` has been changed (or
we want to update the unpinned nested dependencies) `poetry.lock` should be
regenerated with

```
poetry lock --no-update
```

`poetry.lock` should be committed alongside `pyproject.toml` changes.

## Working with static assets

When running locally static assets are served by Flask at http://localhost:6012/static/…

When running on preview, staging and production there’s a bit more to it:

![notify-static-after](https://user-images.githubusercontent.com/355079/50343595-6ea5de80-051f-11e9-85cf-2c20eb3cdefa.png)

## Translations

- Wrap your template text

```
<h1>{{ _('Hello') }}</h1>
```

- For form hints

Set a variable

```
 <div class="extra-tracking">
  {% set hint_txt = _('We’ll send you a security code by text message') %}
  {{textbox(form.mobile_number, width='3-4', hint=hint_txt) }}
 </div>
```

For forms

```
from flask_babel import _
```

Wrap your text

```
_('Your text here')
```

For JavaScript

```
// add your text to main_template
window.APP_PHRASES = {
  now: "{{ _('Now') }}",
}
```

```
// in your JS file
let now_txt = window.polyglot.t("now");
```

- Extract

Currently this is a manual step. Add a row to `fr.csv` in `app/translations/csv/` for each new string you have wrapped. The format is: `"wrapped string","translation"`. Make sure the wrapped string you are adding is unique.

- Compile

```bash
make babel
```

- Testing

Some typos in the `fr.csv` file might not be caught by `babel` but will lead to incorrect or missing French text in the app. To test for the kinds of typos we’ve encountered before, run:
```bash
make test-translations
```
Note that this make target is always run during our CI process and will fail if any problems are detected when pushing changes.

## Using Local Jinja for testing template changes

See the [notification-api](https://github.com/cds-snc/notification-api) README for detailed instructions.

Template files used in this repo: `sms_preview_template.jinja2, email_preview_template.jinja2`

Note: Tests may break if `USE_LOCAL_JINJA_TEMPLATES` is set to `True` in your .env

## Redis

You need a [redis](https://redis.io/) server running to use certain parts of Notify, such as the "go live" flow. To use redis, add `REDIS_ENABLED=1` to your .env file and run the following command:

```bash
redis-server
```

## Testing

There are testing utilities available through the project.

### Trigger an exception on purpose

It is sometimes useful to trigger an exception for testing purposes (logger format,
multiline log behavior, etc.). To do so, you can hit the `_debug?key=DEBUG_KEY` endpoint.
The `DEBUG_KEY` should be defined in the environment's configuration, sourced at the
application' startup. When that endpoint is reached with the proper `DEBUG_KEY` secret,
a `500` HTTP error will be generated. When an incorrect or empty secret is provided, a classic '404'
not found error will get returned.

=======

# Notifications-admin

Application d'administration des notifications.

## Branche amont (_upstream_)

Ce dépôt Git est une version modifiée de :
https://github.com/alphagov/notifications-admin

## Caractéristiques de cette application

- Enregistrer et gérer les utilisateurs
- Créer et gérer des services
- Envoyer des courriels et des SMS par lots en téléversant un CSV
- Afficher l'historique des notifications

## Contraintes fonctionnelles

- Nous ne pouvons pas actuellement envoyer des lettres
- Nous ne pouvons pas savoir si les SMS ont été délivrés ou non

## Première mise en place

[Brew est un gestionnaire de paquets]((https://brew.sh/)) pour OSX. La commande suivante permet d'installer brew :
```shell
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

Langages nécessaires
- Python 3.10
- [Node](https://nodejs.org/) 10.15.3 ou supérieur
- [npm](https://www.npmjs.com/) 6.4.1 ou plus
```shell
brew install node
```

[NPM](npmjs.org) est l'outil de gestion des paquets de Node. `n` est un outil de gestion des
différentes versions de Node. Ce qui suit installe `n` et utilise le support à long terme (LTS)
version de Node.
```shell
npm install -g n
n lts
npm rebuild node-sass
```

### Instruction d'installation locale

Sur macOS :

1. Installer PyEnv avec Homebrew. Cela vous permettra de préserver votre santé mentale.

`brew install pyenv`

2. Installez Python 3.10.8 ou la dernière version

`pyenv install 3.10.8`

3. Si vous n'attendez aucun conflit, mettez `3.10.8` comme valeur par défaut

`pyenv global 3.10.8`

4. Assurez-vous que la version 3.10.8 est maintenant la version par défaut en exécutant

`python --version`

Si ce n’est pas le cas, ajoutez les lignes suivantes à votre fichier shell rc. ex : `.bashrc` ou `.zshrc`
```
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
```
et ouvrez un nouveau terminal.
Si vous n’utilisez toujours pas Python 3.10.8, jetez un coup d’œil ici : https://github.com/pyenv/pyenv/issues/660

5. Installez `virtualenv` :

`pip install virtualenvwrapper`

6. Ajoutez ce qui suit à votre `.bashrc` ou `.zshrc`

```
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source ~/.pyenv/versions/3.10.8/bin/virtualenvwrapper.sh
```

7. Redémarrez votre terminal et créez votre environnement virtuel :

`mkvirtualenv -p ~/.pyenv/versions/3.10.8/bin/python notifications-admin`

8. Vous pouvez maintenant retourner dans votre environnement à tout moment en entrant

`workon notifications-admin`

9. Trouvez les variables env appropriées et copiez-les dans le fichier .env. Un ensemble de valeurs par défaut existe dans le fichier `.env.example` à la racine ou vous pouvez utiliser celles du dossier LastPass. Si vous utilisez celles de LastPass et que vous exécutez l'API localement, modifiez `API_HOST_NAME` pour qu'elle pointe vers votre machine locale

10. Installer toutes les dépendances

`pip3 install -r requirements.txt`

11. Générer le fichier de version

`make generate-version-file`

12. Générer les traductions

`make babel`

13. Installer les dépendances npm et construire les actifs

`npm install` suivi de `npm run build`.

14.  Démarrer le service

`flask run -p 6012 --host=localhost`.

15. Pour tester

`pip3 install -r requirements_for_test.txt`

`make test`

## Reconstruire les fichiers CSS et JS du frontend

Si vous souhaitez que les fichier JS et CSS soient recompilés en fonction des changements, laissez tourner cette fonction dans un terminal séparé de l'application
```shell
npm run watch
```

## Mise à jour des dépendances des applications

Le fichier `requirements.txt` est généré à partir du fichier `requirements-app.txt` afin d'épingler des versions de toutes les dépendances imbriquées. Si `requirements-app.txt` a été modifié (ou nous voulons mettre à jour les dépendances imbriquées non épinglées) `requirements.txt` devrait être régénérée avec

```
make freeze-requirements
```

Le fichier `requirements.txt` doit être ajouté en même temps que les modifications du fichier `requirements-app.txt`.

## Travailler avec des fichier statiques

Lorsque utilisé localement, les fichiers statiques sont servis par Flask à http://localhost:6012/static/...

Lorsque en production ou sur staging, c'est un peu plus compliqué:

![notify-static-after](https://user-images.githubusercontent.com/355079/50343595-6ea5de80-051f-11e9-85cf-2c20eb3cdefa.png)

## Traductions

- Le texte dans le code est en anglais
- Enveloppez votre texte avec `{{ }}`
- Les traductions sont dans `app/translations/csv/fr.csv`

```
<h1>{{ _('Hello') }}</h1>
```

- Pour des conseils sur les formulaires

Crée une variable

```
 <div class="extra-tracking">
  {% set hint_txt = _('We’ll send you a security code by text message') %}
  {{textbox(form.mobile_number, width='3-4', hint=hint_txt) }}
 </div>
```

Pour les formulaires

```
from flask_babel import _
```

Enveloppez votre texte
```
_("Votre texte ici")
```

Pour JavaScript

```
// ajoutez votre texte au main_template
window.APP_PHRASES = {
  now: "{{ _('Now') }}",
}
```

```
// dans vos fichier JS
let now_txt = window.polyglot.t("now") ;
```

- Extrait

Actuellement, il s'agit d'une étape manuelle. Ajoutez une ligne à `fr.csv` dans `app/translations/csv/` pour chaque nouvelle  de charactère que vous avez enveloppée. Le format est le suivant : `"Texte Anglais", "traduction"`. Assurez-vous que la chaîne enveloppée que vous ajoutez est unique.

- Compiler

```Bash
make babel
```

- Tester

Certaines erreurs de frappe dans le fichier `fr.csv` pourraient ne pas être détectées par `babel` mais entraîneraient des changements incorrects ou du texte manquant en français dans l’application. Pour tester contre ces types d’erreurs de frappe qu’on a vu dans le passé, exécutez :
```bash
make test-translations
```
Cette cible make est toujours exécutée pendant le processus d’intégration continue et échouera si des problèmes sont détectés lorsqu’on pousse le code.

## Utiliser Jinja localement pour tester les changements de modèles

Voir le dépôt [notification-api](https://github.com/cds-snc/notification-api) README pour des instructions détaillées.

Fichiers de modèles utilisés dans ce dépôt : `sms_preview_template.jinja2, email_preview_template.jinja2`

Note : les tests peuvent échouer si `USE_LOCAL_JINJA_TEMPLATES` est réglé sur `True` dans votre `.env`

## Tests

Il y a quelques utilitaires de tests qui sont disponibles dans le projet.

### Déclencher une erreur de façon intentionnelle

Il est quelques fois pratique de déclencher une erreur pour des buts de tests (format
du log, comportement multiligne du log, etc.). Pour se faire, vous pouvez accéder à l'URL
`_debug?key=DEBUG_KEY`. Le secret `DEBUG_KEY` devrait être défini dans la configuration
de l'environnement lors du démarrage de l'application. Quand cet URL sera accédé avec
le bon secret `DEBUG_KEY`, une erreur HTTP `500` sera générée. Sans ce secret ou avec
une valeur invalide, une erreur `404` de page non-trouvée sera alors produite.


[![PyPI version](https://badge.fury.io/py/cmind.svg)](https://pepy.tech/project/cmind)
[![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind)
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md)
[![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind)

[![CM test](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml)
[![CM script automation features test](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml)

### About

Collective Knowledge (CK) in a community project to develop open-source tools, platforms and automation recipes
that can help researchers and engineers automate their repetitive, tedious and time-consuming tasks
to build, run, benchmark and optimize AI, ML and other applications and systems
across diverse and continuously changing models, data, software and hardware.

CK consists of several ongoing sub-projects:

* [Collective Mind framework (CM)](cm) (*~1MB*) - a very light-weight Python-based framework with minimal dependencies
  to help users implement, share and reuse cross-platform automation recipes to
  build, benchmark and optimize applications on any platform
  with any software and hardware. CM attempts to extends the `cmake` concept
  with reusable automation recipes and workflows written in plain Python or native OS scripts,
  accessible via a human readable interface with simple tags,
  and shareable in public and private repositories in a decentralized way.
  Furthermore, in comparison with cmake, these automation recipes can not only detect missing code
  but also download artifacts (models, data sets), preprocess them, build missing
  dependencies, install them and run the final code on diverse platforms in a unified and automated way.
  You can read more about the CM concept in this [presentation](https://doi.org/10.5281/zenodo.8105339).

  * [CM automation recipes for MLOps and DevOps](https://github.com/mlcommons/cm4mlops) (*~6MB*) - a small collection of portable, extensible and technology-agnostic automation recipes
    with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications
    on diverse platforms with any software and hardware: see [online catalog](https://access.cknowledge.org/playground/?action=scripts)
    and [source code](https://github.com/mlcommons/cm4mlops/blob/master/script).

  * [CM automation recipes to reproduce research projects](https://github.com/ctuning/cm4research) (*~1MB*) - a unified CM interface to help researchers
    and engineers access, prepare and run diverse research projects and make it easier to validate them in the real world
    across rapidly evolving models, data, software and hardware
    (see [our reproducibility initatives](https://cTuning.org/ae)
    and [motivation](https://www.youtube.com/watch?v=7zpeIVwICa4) behind this project).

  * [CM automation recipes for ABTF](https://github.com/mlcommons/cm4abtf) (*~1MB*) - a unified CM interface and automation recipes
    to run automotive benchmark across different models, data sets, software and hardware from different vendors.

  * [Modular C++ harness for MLPerf loadgen](https://github.com/mlcommons/cm4mlops/tree/main/script/app-mlperf-inference-mlcommons-cpp)

  * [Modular Python harness for MLPerf loadgen](https://github.com/mlcommons/cm4mlops/tree/main/script/app-mlperf-inference-mlcommons-python)

* [Collective Knowledge Playground](https://access.cKnowledge.org) - an external platform being developed by [cKnowledge](https://cKnowledge.org)
  to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows,
  and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges)
  to find the most performance and cost-effective AI/ML Systems.

  * [CK GUI to run modular benchmarks](https://access.cknowledge.org/playground/?action=howtorun) - such benchmarks
    are composed from [CM scripts](https://access.cknowledge.org/playground/?action=scripts)
    and can run via a unified CM interface.

### License

[Apache 2.0](LICENSE.md)

### Documentation

**MLCommons is updating the CM documentation based on user feedback - please stay tuned for more details**.

* [CM Getting Started Guide and FAQ](docs/getting-started.md)
  * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference)
  * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md)
  * [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts)
  * [Other CM tutorials](docs/tutorials)
* [Full documentation](docs/README.md)
* [CM development tasks](docs/taskforce.md#current-tasks)
* [CM and CK history](docs/history.md)

### Citing this project

Please use this [BibTex file](citation.bib).

### Acknowledgments

[Collective Knowledge automation framework (deprecated CK v1 and v2)](https://github.com/mlcommons/ck/tree/master/ck),
[Collective Mind automation framework (CM)](https://github.com/mlcommons/ck/tree/master/cm),
[CM4MLOPS](https://github.com/mlcommons/cm4mlops) and
[CM4ABTF](https://github.com/mlcommons/cm4abtf) were originally developed by [Grigori Fursin](https://cKnowledge.org/gfursin)
and donated to MLCommons to benefit everyone. You can learn more about the motivation behind these projects from the following presentations:

* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ]
* ACM TechTalk'21 about automating research projects: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]

We would like to thank all our great
[volunteers, collaborators and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
for their support, fruitful discussions, and useful feedback!
We thank the [cTuning foundation](https://cTuning.org), [cKnowledge.org](https://cKnowledge.org)
and [MLCommons](https://mlcommons.org) for sponsoring this project!


## EO Datasets

[![Linting](https://github.com/GeoscienceAustralia/eo-datasets/actions/workflows/lint.yml/badge.svg)](https://github.com/GeoscienceAustralia/eo-datasets/actions/workflows/lint.yml)
[![Tests](https://github.com/GeoscienceAustralia/eo-datasets/actions/workflows/test.yml/badge.svg)](https://github.com/GeoscienceAustralia/eo-datasets/actions/workflows/test.yml)
[![Coverage Status](https://img.shields.io/codecov/c/github/GeoscienceAustralia/eo-datasets)](https://app.codecov.io/gh/GeoscienceAustralia/eo-datasets)

A tool to easily write, validate and convert [ODC](https://github.com/opendatacube/datacube-core)
datasets and metadata.

## Installation

    pip install eodatasets3

Python 3.8+ is supported.

## Dataset assembly

The assembler api aims to make it easy to write datasets.

```python
from datetime import datetime
from pathlib import Path

from eodatasets3 import DatasetAssembler

with DatasetAssembler(
    Path("/some/output/collection/path"), naming_conventions="default"
) as p:
    # Add some common metadata fields.
    p.platform = "landsat-7"
    p.instrument = "ETM"
    p.datetime = datetime(2019, 7, 4, 13, 7, 5)
    p.processed_now()

    # Support for custom metadata fields
    p.properties["fmask:cloud_shadow"] = 42.0

    # If you have a source dataset, you can include it as provenance.
    # Assembler can also copy common metadata properties from it.
    # (... so we didn't need to set the "platform" above!)
    p.add_source_path(source_dataset, auto_inherit_properties=True)

    # Write measurements. They can be from numpy arrays, open rasterio datasets,
    # file paths, ODC Datasets...
    p.write_measurement("red", red_path)
    ...  # now write more measurements

    # Create a jpg thumbnail image using the measurements we've written
    p.write_thumbnail(red="swir1", green="swir2", blue="red")

    # Validate the dataset and write it to the destination folder atomically.
    p.done()
```

The Assembler will write a folder of [COG](https://www.cogeo.org/) imagery, an [eo3](#open-data-cube-compatibility)
metadata doc for Open Data Cube, and create appropriate file and folder structures for the chosen naming conventions.

If you already have existing imagery, you can use DatasetAssembler to create a matching metadata document.

See [the documentation guide for more features and examples](https://eodatasets.readthedocs.io/en/latest/).

## Open Data Cube compatibility

The assembler writes a format called "eo3", which will be the native metadata format for Open Data Cube
2.0. We recommend new products are written with this format, even if targeting Open Data Cube 1.
Datacube versions from 1.8 onwards are compatible natively with eo3.

eo3 adds information about the native grid of the data, and aims to be more easily interoperable
with the upcoming [Stac Item metadata](https://github.com/radiantearth/stac-spec/tree/master/item-spec).

# Other Tools Included

## Validator

`eo3-validate` a lint-like checker to check ODC documents.

Give it ODC documents for your products, types and/or datasets to find
errors quickly.

    ❯ eo3-validate my-product.odc-product.yaml /tmp/path/to/dataset.odc-metadata.yaml
    ❯ eo3-validate https://explorer.dea.ga.gov.au/products/ga_ls_fc_3.odc-product.yaml

Note that documents are processed in order. A product should be specified before its dataset,
so that the product is known by the validator before the dataset is checked.

Similarly, a Metadata Type should ideally be specified before its product, so that the type is known.
But all are optional for basic checks

- **Note** You can use `--odc` or `--explorer-url` to automatically load a list of types and products.

### Help text

    ❯ eo3-validate --help
	Usage: eo3-validate [OPTIONS] [PATHS]...

	  Validate ODC dataset documents

	  Paths can be products, dataset documents, or directories to scan (for files
	  matching names '*.odc-metadata.yaml' etc), either local or URLs.

	  Datasets are validated against matching products that have been scanned
	  already, so specify products first, and datasets later, to ensure they can
	  be matched.

	Options:
	  --version                       Show the version and exit.
	  -W, --warnings-as-errors        Fail if any warnings are produced
	  -f, --output-format [plain|quiet|github]
					  Output format  [default: plain]
	  --thorough                      Attempt to read the data/measurements, and
					  check their properties match
	  --explorer-url TEXT             Use product definitions from the given
					  Explorer URL to validate datasets. Eg:
					  "https://explorer.dea.ga.gov.au/"
	  --odc                           Use product definitions from datacube to
					  validate datasets
	  -q, --quiet                     Only print problems, one per line
	  --help                          Show this message and exit.

### Eo3-validate github action (market place)
#### Inputs

`command`: **Required** The command to run. Default `"eo3-validate"`.
`filepath`: **Required** The filepath to odc documents. Default `""`.

#### Example usage

```
uses: actions/eo3-validate@v1
with:
  command: 'eo3-validate'
  filepath: '/'
```

### Eo3-validate github action (docker)
#### Example usage

```
name: Run eo3-validate
run: |
    docker run -v $PWD/:/code/odc-files opendatacube/eo-datasets:latest eo3-validate ./odc-files

```

### Disabling warnings

ODC is very configurable, and sometimes the validator will be too strict
for you.

You can ease some restrictions for a product by adding
a `default_allowances` section to the end of your Product definition.

(The same section can also be included in any Metadata Type, and will apply
to all products of that type)

Example supported fields:

```yaml

# Possible restriction to ease.
# All fields can be omitted (please don't include them if you don't need them).
default_allowances:

  # "It's okay if some datasets to have null geometry."
  require_geometry: false

  # Allow some metadata fields (from the metadata type) to be null
  # (but they must still be in each dataset document)
  allow_nullable_fields:
    - dataset_maturity

  # Allow some metadata fields to be entirely missing from datasets
  allow_missing_fields:
    - sentinel_product_name
    - s2cloudless_clear

  # Allow datasets to have extra measurements that weren't in the product definition.
  allow_extra_measurements: [nbar_blue, nbar_green, nbar_red]
```

### 'Thorough' mode

By default, the validator will only look at the given metadata document(s).

But when the `--thorough` flag is given, the validator will attempt to read any
referenced imagery, checking their properties match the definitions (nodata, dtype etc).

## Stac metadata conversion

`eo3-to-stac`: Convert an EO3 metadata doc to a Stac Item

	❯ eo3-to-stac --help
	Usage: eo3-to-stac [OPTIONS] [ODC_METADATA_FILES]...

	  Convert an EO3 metadata doc to a Stac Item.

	Options:
	  -v, --verbose
	  -u, --stac-base-url TEXT      Base URL of the STAC file
	  -e, --explorer-base-url TEXT  Base URL of the ODC Explorer
	  --validate / --no-validate    Validate output STAC Item against online
					schemas

	  --help                        Show this message and exit.

Example usage:

	❯ eo3-to-stac LT05_L1TP_113081_19880330_20170209_01_T1.odc-metadata.yaml
	❯ ls
	LT05_L1TP_113081_19880330_20170209_01_T1.odc-metadata.yaml
	LT05_L1TP_113081_19880330_20170209_01_T1.stac-item.json

## Prep Scripts

Some scripts are included for preparing common metadata documents,
such as landsat scenes.

`eo3-prepare`: Prepare ODC metadata from the commandline.

Some sub-commands need the ancillary dependencies, for reading from
exotic formats: `pip install .[ancillary]`

	❯ eo3-prepare --help
	Usage: eo3-prepare [OPTIONS] COMMAND [ARGS]...

	Options:
	  --version  Show the version and exit.
	  --help     Show this message and exit.

	Commands:
	  landsat-l1     Prepare eo3 metadata for USGS Landsat Level 1 data.
	  modis-mcd43a1  Prepare MODIS MCD43A1 tiles for indexing into a Data...
	  noaa-prwtr     Prepare NCEP/NCAR reanalysis 1 water pressure datasets...
	  sentinel-l1   Prepare eo3 metadata for Sentinel-2 Level 1C data produced...

Prep scripts have their own options, for example Sentinel L1 generation can filter by time
or region, if the inputs follow a common directory structure:

```
❯ eo3-prepare sentinel-l1 --help
Usage: eo3-prepare sentinel-l1 [OPTIONS] [DATASETS]...

  Prepare eo3 metadata for Sentinel-2 Level 1C data produced by Sinergise or
  ESA.

  Takes ESA zipped datasets or Sinergise dataset directories

Options:
  -v, --verbose
  -f, --datasets-path FILE        A file to read input dataset paths from, one
                                  per line
  -j, --jobs INTEGER              Number of workers to run in parallel
  --overwrite-existing / --skip-existing
                                  Overwrite if exists (otherwise skip)
  --embed-location / --no-embed-location
                                  Embed the location of the dataset in the
                                  metadata? (if you wish to store them
                                  separately. default: auto)
  --always-granule-id / --never-granule-id
                                  Include the granule id in metadata
                                  filenames? (default: auto -- include only
                                  for multi-granule files). Beware that multi-
                                  granule datasets without a granule id in the
                                  filename will overwrite each-other
  --throughly-check-existing / --cheaply-check-existing
                                  Should we open every dataset to check if
                                  *all* inner granules have been produced?
                                  Default: false.
  --provider [sinergise.com|esa.int]
                                  Restrict scanning to only packages of the
                                  given provider. (ESA assumes a zip file,
                                  sinergise a directory)
  --output-base DIRECTORY         Write metadata files into a directory
                                  instead of alongside each dataset
  --input-relative-to DIRECTORY   Input root folder that should be used for
                                  the subfolder hierarchy in the output-base
  --only-regions-in-file FILE     Only process datasets in the given regions.
                                  Expects a file with one region code per
                                  line. (Note that some older ESA datasets
                                  have no region code, and will not match any
                                  region here.)
  --after-month YEAR-MONTH        Limit the scan to datasets newer than a
                                  given month (expressed as {year}-{month}, eg
                                  '2010-01')
  --before-month YEAR-MONTH       Limit the scan to datasets older than the
                                  given month (expressed as {year}-{month}, eg
                                  '2010-01')
  --index                         Index newly-generated metadata into the
                                  configured datacube
  --dry-run                       Show what would be created, but don't create
                                  anything
  -E, --env TEXT
  -C, --config, --config_file TEXT
  --help                          Show this message and exit.
```

An example of preparing metadata in a separate directory (not alongside the datasets) at NCI
can be as follows:

```bash
module use -a /g/data/v10/private/modules/modulefiles /g/data/v10/public/modules/modulefiles
module load eodatasets3

# With a folder of input paths, 4 workers, and separate output directory:
eo3-prepare sentinel-l1 -j 4 --output-base /output/metadata/directory \
  /g/data/fj7/Copernicus/Sentinel-2/MSI/L1C/2021

# Using a file for input paths. Filter them to a certain region list and recent months:
eo3-prepare sentinel-l1 \
    --output-base /g/data/v10/agdc/jez/c3/L1C  \
    --only-regions-in-file test-regions.txt \
    --after-month 2022-04 \
    -f l1cs-2022-05-02.txt

```

`eo3-package-wagl`: Convert and package WAGL HDF5 outputs.

 Needs the wagl dependencies group: `pip install .[wagl]`

	❯ eo3-package-wagl --help
	Usage: eo3-package-wagl [OPTIONS] H5_FILE

	  Package WAGL HDF5 Outputs

	  This will convert the HDF5 file (and sibling fmask/gqa files) into
	  GeoTIFFS (COGs) with datacube metadata using the DEA naming conventions
	  for files.

	Options:
	  --level1 FILE                   Optional path to the input level1 metadata
					  doc (otherwise it will be loaded from the
					  level1 path in the HDF5)

	  --output DIRECTORY              Put the output package into this directory
					  [required]

	  -p, --product [nbar|nbart|lambertian|sbt]
					  Package only the given products (can specify
					  multiple times)

	  --with-oa / --no-oa             Include observation attributes (default:
					  true)

	  --with-oa / --no-oa             Include observation attributes (default:
					  true)

	  --oa-resolution FLOAT           Resolution choice for observation attributes
					  (default: automatic based on sensor)

	  --help                          Show this message and exit.

# Development Setup

Run the tests using [pytest](http://pytest.org/).

	❯ pytest

You may need to install test dependencies first:

	❯ pip install -e .[test]

Dependencies such as gdal can be tricky to install on some systems. You
may prefer to use the included Docker file for development: run `make
build` to create a container, and `make test` to run tests.

We have strict linting and formatting checks on this reposistory, so
please run pre-commit (below) after checkout.

## Pre-commit setup

	❯ pip install pre-commit
	❯ pre-commit install

(if you are using Conda, you need to `conda install pre_commit` instead of using pip)

Your code will now be formatted and validated before each commit. You can also invoke it manually by running `pre-commit run`

This allows you to immediately catch and fix issues before you raise a pull request that fails.

Most notably, all code is formatted using
[black](https://github.com/ambv/black), and checked with
[pyflakes](https://github.com/PyCQA/pyflakes).

## Docker dependencies

To update the set of frozen dependencies inside docker:

1) Ensure you have an existing, working image. ie, run `make build`. If the current image is broken,
you may need to `git checkout` the last working version first.
2) Run `make dependency-update`. This will recalculate the list of all dependencies from the definitions in setup.py.

Note that this will run pip-compile _inside_ the docker container for maximum compatibility, hence the need for an existing container.

## Creating Releases

First, draft [some release notes](https://github.com/GeoscienceAustralia/eo-datasets/releases)
for users of the library.

Now tag and upload:

```
# Be up-to-date.
git fetch origin

# Create a tag for the new version
# (using semantic versioning https://semver.org/)
git tag eodatasets3-<version> origin/eodatasets3

# Create package
python3 setup.py sdist bdist_wheel

# Upload it (Jeremy, Damien, Kirill have pypi ownership)
python3 -m twine upload  dist/*

# Push tag to main repository
git push origin --tags
```


Novus
==========

.. image:: https://discord.com/api/guilds/208895639164026880/embed.png
   :target: https://discord.gg/vfl
   :alt: Discord server invite
.. image:: https://img.shields.io/pypi/v/novus.svg
   :target: https://pypi.python.org/pypi/novus
   :alt: PyPI version info
.. image:: https://img.shields.io/pypi/pyversions/novus.svg
   :target: https://pypi.python.org/pypi/novus
   :alt: PyPI supported Python versions

A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python.

A full fork of Rapptz's Discord.py library, with new features added on top.

Key Features
-------------

- Modern Pythonic API using ``async`` and ``await``.
- Proper rate limit handling.
- Optimised in both speed and memory.

Installing
----------

**Python 3.8 or higher is required**

To install the library without full voice support, you can just run the following command:

.. code:: sh

    # Linux/macOS
    python3 -m pip install -U novus

    # Windows
    py -3 -m pip install -U novus

Otherwise to get voice support you should run the following command:

.. code:: sh

    # Linux/macOS
    python3 -m pip install -U "novus[voice]"

    # Windows
    py -3 -m pip install -U novus[voice]

To install the development version, do the following:

.. code:: sh

    $ git clone https://github.com/Voxel-Fox-Ltd/novus
    $ cd novus
    $ python3 -m pip install -U .[voice]

Optional Packages
~~~~~~~~~~~~~~~~~~

* `PyNaCl <https://pypi.org/project/PyNaCl/>`__ (for voice support)

Please note that on Linux installing voice you must install the following packages via your favourite package manager (e.g. ``apt``, ``dnf``, etc) before running the above commands:

* libffi-dev (or ``libffi-devel`` on some systems)
* python-dev (e.g. ``python3.6-dev`` for Python 3.6)

Quick Example
--------------

.. code:: py

    import discord

    class MyClient(discord.Client):
        async def on_ready(self):
            print('Logged on as', self.user)

        async def on_message(self, message):
            # don't respond to ourselves
            if message.author == self.user:
                return

            if message.content == 'ping':
                await message.channel.send('pong')

    client = MyClient()
    client.run('token')

Bot Example
~~~~~~~~~~~~~

.. code:: py

    import discord
    from discord.ext import commands

    bot = commands.Bot(command_prefix='>')

    @bot.command()
    async def ping(ctx):
        await ctx.send('pong')

    bot.run('token')

You can find more examples in the examples directory.

Links
------

- `Documentation <https://novus.readthedocs.io/en/latest/index.html>`_
- `Official Discord Server <https://discord.gg/vfl>`_


# Cactus-Blockchain

**Cactus** is a modern community-centric green cryptocurrency based on a proof-of-space-and-time consensus algorithm. It is a community-supported fork of the [Cactus Network](https://github.com/Cactus-Network/cactus-blockchain) codebase.

For more information, see our website and downloads at Cactus Network.

You can learn more in the Cactus Wiki: Quick Start Guide

Please check out the Cactus Discord Channel: (https://discord.gg/qfHBqZnXTj).

Full Node List Here: Full Node IP's

Introducer Port: 11444

<svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg>Windows Installer

Download the Windows installer (exe) or Zip file - Cactus Blockchain Windows

As the Cactus code signing certificate is new you will likely have to ask to keep the download and when you run the installer, you will have to choose "More Info" and "Run Anyway" to be able to run the installer. There is no need to use the command line. Some Windows anti-virus applications are seeing the download as a false positive. You can see the entire source code and build method here so we think it's safe for you to ask those tools to ignore it. Running the installer while plotting on a previous version will stop your plotting process, so be careful.

You can learn more through the Cactus Quick Start Guide: Quick Start Guide

<svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg>Ubuntu/Debian Install

sudo apt-get update

sudo apt-get upgrade -y

# Install Git

sudo apt install git -y

# Checkout the source and install

git clone https://github.com/Cactus-Network/cactus-blockchain.git --recurse-submodules

      cd cactus-blockchain
sh install.sh

. ./activate
cactus init

#add your plot directories the /.cactus/main/config/config.yaml

cactus start famer

#add your harvesters - https://github.com/Cactus-Network/cactus-blockchain/wiki/Farming-on-many-machines

# The GUI requires you have Ubuntu Desktop or a similar windowing system installed.

# You can not install and run the GUI as root

cd /home/'yourusername'/cactus-blockchain/

. ./activate
sh install-gui.sh

cd cactus-blockchain-gui
npm run electron &

 <svg class="octicon octicon-link" viewbox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg>Ubuntu/Debian Update to 1.3.4

  **Stop all cactus processes (cactus stop -d all) then,

Delete ('rm -rf) your 'cactus-blockchain' folder (**not your '.cactus' folder**). Then clone 'cactus-blockchain'.

   git clone https://github.com/Cactus-Network/cactus-blockchain.git --recurse-submodules

cd cactus-blockchain

sh install.sh

 . ./activate

 cactus init

 cactus start farmer

 *Note, if you would like to run the Timelord to support the Cactus ecosystem, run 'sh install-timelord.sh', then cactus start timelord. If you receive errors during the installation process, your system is missing dependencies. In this case, thank you for trying :)


# Milvus Documentation

Welcome to Milvus documentation!

This repository contains technical documentation for [Milvus](https://github.com/milvus-io/milvus), the world's most advanced open-source vector database. Visit [Milvus.io](https://milvus.io/docs) or [Web content repo](https://github.com/milvus-io/web-content) for fully rendered documents.

Each branch corresponds to a Milvus release by name. We've set the branch of the latest Milvus release as the default branch. For documentation of a different Milvus release, switch to the corresponding branch.

## Index

- [Suggest changes](#suggest-changes)

- [Contribute to Milvus docs](#contribute-to-milvus-docs)

- [Active Milvus releases](#active-milvus-releases)

- [Other contributing channels](#other-contributing-channels)

- [Reference](#reference)

- [Get in touch](#get-in-touch)

## Suggest changes

Have a new topic that you want to include in the docs? Need additional instructions or clarifications? Spot a broken link? Please [open an issue](https://github.com/milvus-io/milvus-docs/issues/new/choose).

## Contribute to Milvus docs

Want to be a contributor to the docs? See [CONTRIBUTING](CONTRIBUTING.md) for details about setting yourself up and getting started.

## Active Milvus releases

Currently, we are maintaining the following versions of Milvus documentation in different branches:

| `v2.1.x` branch | 2.1.x version (the default and latest branch) |
| --------------- | --------------------------------------------- |
| `v2.0.x` branch | 2.0.x version                                 |
| `v1.1.1` branch | 1.1.1 version                                 |
| `v1.1.0` branch | 1.1.0 version                                 |
| `v1.0.0` branch | 1.0.0 version (first with long-term support)  |

> **Note:**
> The latest branch is set as the default working branch.
> Milvus releases prior to v1.0.0 are no longer supported, but you can still find their documentation in the corresponding branch.

## Other contributing channels

Not all content rendered on [milvus.io](https://milvus.io/docs) is hosted in this repository.

### Contribute to API docs

To contribute to the Milvus SDK documentation, visit the language-specific repositories listed below:

- [PyMilvus](https://github.com/milvus-io/pymilvus)
- [GO](https://github.com/milvus-io/milvus-sdk-go)
- [Java](https://github.com/milvus-io/milvus-sdk-java)
- [Node.js](https://github.com/milvus-io/milvus-sdk-node)

### Contribute to community content

If you would like to write an article for our [blog](https://medium.com/unstructured-data-service), please [file an issue](https://github.com/milvus-io/community/issues/new) in the [Milvus community repository](https://github.com/milvus-io/community) or contact community@zilliz.com. Feel free to submit either a completed draft or article ideas. All submissions will be reviewed as quickly as possible. If your article or idea seems like a good fit for our blog, we will reach out to you directly.

> In your title/subject line, please put in [blog submission] to make sure your email does not get buried.

## Reference

Reference to cite when you use Milvus in a research paper:

```

@inproceedings{2021milvus,

  title={Milvus: A Purpose-Built Vector Data Management System},

  author={Wang, Jianguo and Yi, Xiaomeng and Guo, Rentong and Jin, Hai and Xu, Peng and Li, Shengjun and Wang, Xiangyu and Guo, Xiangzhou and Li, Chengming and Xu, Xiaohai and others},

  booktitle={Proceedings of the 2021 International Conference on Management of Data},

  pages={2614--2627},

  year={2021}

}

```

## Get in touch

Join the Milvus community on [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ) to share your suggestions, advice, or questions with our engineering team.

Subscribe to Milvus's mailing lists:

- [Technical Steering Committee](https://lists.lfai.foundation/g/milvus-tsc)

- [Technical Discussions](https://lists.lfai.foundation/g/milvus-technical-discuss)

- [Announcement](https://lists.lfai.foundation/g/milvus-announce)

Follow Milvus on social media:

- [Medium](https://medium.com/@milvusio)

- [Twitter](https://twitter.com/milvusio)

- [LinkedIn](https://www.linkedin.com/company/the-milvus-project)


# The Legend of Zelda - Skyward Sword Randomizer

This is a randomizer for the game _The Legend of Zelda: Skyward Sword_.

You can access most resources on the [Skyward Sword Randomizer website](https://ssrando.com); you can also come discuss or ask for help on the [Skyward Sword Randomizer & Modding Discord](https://discord.gg/evpNKkaaw6).

## Requirements
- A computer (to run the randomizer)
- A clean NTSC-U 1.00 ISO (MD5 hash: e7c39bb46cf938a5a030a01a677ef7d1)
- The randomizer (see below for installation)

- Something to run the randomized game:
    + An emulator (Dolphin is the most widely used one)
    + A homebrewed Wii that can launch games from an external medium, such as a SD card, a USB drive or an external hard drive. Visit [https://wii.guide](https://wii.guide) for homebrewing information

## Installation

You can either run from a compiled binary (recommended for Windows, required for tournaments) or directly from source ([jump here](#Installing-from-source), required for Linux). Currently, only the command-line interface works for OSX; it needs an installation from source.

## Installing from a compiled binary
1. Choose the version of the randomizer on the [website](https://ssrando.com) and download it
2. Extract the ZIP file where you want to install the randomizer
    + if you already have another version installed, you can use the same folder for multiple instances and rename the executables to avoid conflicts
3. [Run](#Running-the-randomizer) the extracted executable (`ssrando.exe`)

## Running the randomizer

1. As indicated, select a clean NTSC-U 1.00 ISO. The randomizer will then check its integrity, which may take a few minutes
2. Choose the folder where the randomized file will be created; by default, it is the randomizer installation folder
3. Customize the settings to your liking. You can use permalinks to store and share selected settings
4. Hit randomize; this may also take a few minutes

## Playing the randomized game

The randomized game will only work if the game language is set to **English**, other languages **will not work**.

Do **not** use the game's hero-mode files, it **will not work**.
If you only want it to skip cutscenes, they are made skippable even in normal mode.
If you really wish to play on hero mode, there are options to enable its features.
When these options selected, even if they don't appear as such in the menu, all files will be set to hero mode.

If you didn't follow these rules, your progress cannot be restored; delete the save from the Wii menu / Dolphin's toolbar and correct the settings to play the game.

### For Dolphin
1. Just open the generated `SOUE01.iso` in Dolphin

### For console
1. Move the generated `SOUE01.iso` to your external medium in this folder (the folder name may change depending on the game loader you are using):

    `/wbfs/The Legend of Zelda Skyward Sword [SOUE01]/`
2. Launch the USB loader, you should see the game as "The Legend of Zelda Skyward Sword"

## Installing from source

You will need Python (at least version 3.9) and pip (which should come with Python)

1. In a terminal, navigate to the directory where you want to install the randomizer (using `cd`)
2. Clone the repository with git and enter the directory

        git clone https://github.com/ssrando/ssrando
        cd ssrando

3. Choose the branch you want to run (`main` is the most up-to-date), or the version you want to run (`v1.x.x` for stable releases, `async-[month]-[yy]` for async races; you can check the releases in Github):

        git checkout [BRANCH OR VERSION]
    Replace `[BRANCH OR VERSION]` with the desired branch or version in the command
4. Install `poetry`, which is used to install the dependencies
        pip install poetry
    If your system uses old versions of Python, you may need to replace `pip` with `python3 -mpip`

5. Install the dependencies:

        poetry install

6. [Run](#Running-the-randomizer) the randomizer:

        poetry run python randoscript.py

## The command-line interface

If you installed from source, you can run the randomizer without using the GUI by using `--noui` when running `randoscript.py`.

You will need to select the settings using the command-line options, use `--help` to list them.

For now, you have to run the GUI once so the ISO can be extracted (ask for a workaround in Discord if needed)

You can also pass options when launching the GUI, they will be pre-entered (this can be useful when creating a script to run the randomizer)

## Model Customization
After running the randomizer once, a folder `oarc` will be created, which contains Link's model (`Alink.arc`) and the Loftwing's model (`Bird_Link.arc`) as well as several other arc files. All arcs can also be found in the `actual-extract` folder. These folders can be used to get unmodified models from your clean ISO and can be used for creating custom models.

To modify them, you need one or more external programs to edit, save, and convert the existing game models (more information can be found in the [Skyward Sword Randomizer & Modding Discord](https://discord.gg/evpNKkaaw6)).

Modified Link and Loftwing models should be placed in a model pack using `models/[model pack name]/Player` and `models/[model pack name]/Loftwing` folders respectively. Other modifed arcs should be placed in the `arc-replacements` folder (or alternatively in `models/[model pack name]/Player/AdditionalArcs` for easier model pack creation).

Modified arcs **must** have the same same name as the original model they replace.

## Tests
Tests need a source installation and an extracted ISO:

    python -mpytest test
If your system uses old versions of Python, you may need to replace `python` with `python3`

### Logic Dump
The `dump.yaml` file contains a dump of the static logic data for downstream consumers (such as trackers).
If you make changes to logic, you may have to re-generate the dump

    poetry run python randoscript.py --dump dump.yaml
and commit the result, otherwise a GitHub action will fail and flag the outdated dump.

## Contributing
Contributions are always welcome! Discussion happens on Discord.

We are using `black` to format code; you can run `black .` to format all files.

To install developing dependencies (including `black`):

    poetry install --only=dev
If your system uses old versions of Python, you may need to replace `pip` with `python3 -mpip`

### Executable
To build the executable, you need `PyInstaller` installed:

    poetry install --only=build

Then, build the executable using

    pyinstaller ssrando.spec

### Contributors
- lepelog: Main developer
- Peppernicus2000: Logic, fixes
- Azer67: Logic
- MrCheeze: Reverse engineering file formats
- LagoLunatic: For implementing the logic for [TWWR](https://github.com/LagoLunatic/wwrando), which is also used here
- DGod63: title screen logo
- cjs07: GUI


[![ONNX Runtime](https://github.com/huggingface/optimum/actions/workflows/test_onnxruntime.yml/badge.svg)](https://github.com/huggingface/optimum/actions/workflows/test_onnxruntime.yml)

# Hugging Face Optimum

🤗 Optimum is an extension of 🤗 Transformers and Diffusers, providing a set of optimization tools enabling maximum efficiency to train and run models on targeted hardware, while keeping things easy to use.

## Installation

🤗 Optimum can be installed using `pip` as follows:

```bash
python -m pip install optimum
```

If you'd like to use the accelerator-specific features of 🤗 Optimum, you can install the required dependencies according to the table below:

| Accelerator                                                                                | Installation                                                      |
|:-----------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------|
| [ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/overview)                                               | `pip install --upgrade --upgrade-strategy eager optimum[onnxruntime]`      |
| [Intel Neural Compressor](https://huggingface.co/docs/optimum/intel/index)                                             | `pip install --upgrade --upgrade-strategy eager optimum[neural-compressor]`|
| [OpenVINO](https://huggingface.co/docs/optimum/intel/index)                                                            | `pip install --upgrade --upgrade-strategy eager optimum[openvino]`         |
| [NVIDIA TensorRT-LLM](https://huggingface.co/docs/optimum/main/en/nvidia_overview)                                     | `docker run -it --gpus all --ipc host huggingface/optimum-nvidia`          |
| [AMD Instinct GPUs and Ryzen AI NPU](https://huggingface.co/docs/optimum/amd/index)                                    | `pip install --upgrade --upgrade-strategy eager optimum[amd]`              |
| [AWS Trainum & Inferentia](https://huggingface.co/docs/optimum-neuron/index)                                           | `pip install --upgrade --upgrade-strategy eager optimum[neuronx]`          |
| [Habana Gaudi Processor (HPU)](https://huggingface.co/docs/optimum/habana/index)                                       | `pip install --upgrade --upgrade-strategy eager optimum[habana]`           |
| [FuriosaAI](https://huggingface.co/docs/optimum/furiosa/index)                                                         | `pip install --upgrade --upgrade-strategy eager optimum[furiosa]`          |

The `--upgrade --upgrade-strategy eager` option is needed to ensure the different packages are upgraded to the latest possible version.

To install from source:

```bash
python -m pip install git+https://github.com/huggingface/optimum.git
```

For the accelerator-specific features, append `optimum[accelerator_type]` to the above command:

```bash
python -m pip install optimum[onnxruntime]@git+https://github.com/huggingface/optimum.git
```

## Accelerated Inference

🤗 Optimum provides multiple tools to export and run optimized models on various ecosystems:

- [ONNX](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model) / [ONNX Runtime](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models)
- TensorFlow Lite
- [OpenVINO](https://huggingface.co/docs/optimum/intel/inference)
- Habana first-gen Gaudi / Gaudi2, more details [here](https://huggingface.co/docs/optimum/main/en/habana/usage_guides/accelerate_inference)
- AWS Inferentia 2 / Inferentia 1, more details [here](https://huggingface.co/docs/optimum-neuron/en/guides/models)
- NVIDIA TensorRT-LLM , more details [here](https://huggingface.co/blog/optimum-nvidia)

The [export](https://huggingface.co/docs/optimum/exporters/overview) and optimizations can be done both programmatically and with a command line.

### Features summary

| Features                           | [ONNX Runtime](https://huggingface.co/docs/optimum/main/en/onnxruntime/overview)| [Neural Compressor](https://huggingface.co/docs/optimum/main/en/intel/optimization_inc)| [OpenVINO](https://huggingface.co/docs/optimum/main/en/intel/inference)| [TensorFlow Lite](https://huggingface.co/docs/optimum/main/en/exporters/tflite/overview)|
|:----------------------------------:|:------------------:|:------------------:|:------------------:|:------------------:|
| Graph optimization                 | :heavy_check_mark: | N/A                | :heavy_check_mark: | N/A                |
| Post-training dynamic quantization | :heavy_check_mark: | :heavy_check_mark: | N/A                | :heavy_check_mark: |
| Post-training static quantization  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Quantization Aware Training (QAT)  | N/A                | :heavy_check_mark: | :heavy_check_mark: | N/A                |
| FP16 (half precision)              | :heavy_check_mark: | N/A                | :heavy_check_mark: | :heavy_check_mark: |
| Pruning                            | N/A                | :heavy_check_mark: | :heavy_check_mark: | N/A                |
| Knowledge Distillation             | N/A                | :heavy_check_mark: | :heavy_check_mark: | N/A                |

### OpenVINO

Before you begin, make sure you have all the necessary libraries installed :

```bash
pip install --upgrade --upgrade-strategy eager optimum[openvino]
```

It is possible to export 🤗 Transformers and Diffusers models to the OpenVINO format easily:

```bash
optimum-cli export openvino --model distilbert-base-uncased-finetuned-sst-2-english distilbert_sst2_ov
```

If you add `--weight-format int8`, the weights will be quantized to `int8`, check out our [documentation](https://huggingface.co/docs/optimum/main/en/intel/optimization_ov#weight-only-quantization) for more detail on weight only quantization. To apply quantization on both weights and activations, you can find more information [here](https://huggingface.co/docs/optimum/main/en/intel/optimization_ov#static-quantization).

To load a model and run inference with OpenVINO Runtime, you can just replace your `AutoModelForXxx` class with the corresponding `OVModelForXxx` class. To load a PyTorch checkpoint and convert it to the OpenVINO format on-the-fly, you can set `export=True` when loading your model.

```diff
- from transformers import AutoModelForSequenceClassification
+ from optimum.intel import OVModelForSequenceClassification
  from transformers import AutoTokenizer, pipeline

  model_id = "distilbert-base-uncased-finetuned-sst-2-english"
  tokenizer = AutoTokenizer.from_pretrained(model_id)
- model = AutoModelForSequenceClassification.from_pretrained(model_id)
+ model = OVModelForSequenceClassification.from_pretrained("distilbert_sst2_ov")

  classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)
  results = classifier("He's a dreadful magician.")
```

You can find more examples in the [documentation](https://huggingface.co/docs/optimum/intel/inference) and in the [examples](https://github.com/huggingface/optimum-intel/tree/main/examples/openvino).

### Neural Compressor

Before you begin, make sure you have all the necessary libraries installed :

```bash
pip install --upgrade --upgrade-strategy eager optimum[neural-compressor]
```

Dynamic quantization can be applied on your model:

```bash
optimum-cli inc quantize --model distilbert-base-cased-distilled-squad --output ./quantized_distilbert
```

To load a model quantized with Intel Neural Compressor, hosted locally or on the 🤗 hub, you can do as follows :
```python
from optimum.intel import INCModelForSequenceClassification

model_id = "Intel/distilbert-base-uncased-finetuned-sst-2-english-int8-dynamic"
model = INCModelForSequenceClassification.from_pretrained(model_id)
```

You can find more examples in the [documentation](https://huggingface.co/docs/optimum/intel/optimization_inc) and in the [examples](https://github.com/huggingface/optimum-intel/tree/main/examples/neural_compressor).

### ONNX + ONNX Runtime

Before you begin, make sure you have all the necessary libraries installed :

```bash
pip install optimum[exporters,onnxruntime]
```

It is possible to export 🤗 Transformers and Diffusers models to the [ONNX](https://onnx.ai/) format and perform graph optimization as well as quantization easily:

```plain
optimum-cli export onnx -m deepset/roberta-base-squad2 --optimize O2 roberta_base_qa_onnx
```

The model can then be quantized using `onnxruntime`:

```bash
optimum-cli onnxruntime quantize \
  --avx512 \
  --onnx_model roberta_base_qa_onnx \
  -o quantized_roberta_base_qa_onnx
```

These commands will export `deepset/roberta-base-squad2` and perform [O2 graph optimization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization#optimization-configuration) on the exported model, and finally quantize it with the [avx512 configuration](https://huggingface.co/docs/optimum/main/en/onnxruntime/package_reference/configuration#optimum.onnxruntime.AutoQuantizationConfig.avx512).

For more information on the ONNX export, please check the [documentation](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model).

#### Run the exported model using ONNX Runtime

Once the model is exported to the ONNX format, we provide Python classes enabling you to run the exported ONNX model in a seemless manner using [ONNX Runtime](https://onnxruntime.ai/) in the backend:

```diff
- from transformers import AutoModelForQuestionAnswering
+ from optimum.onnxruntime import ORTModelForQuestionAnswering
  from transformers import AutoTokenizer, pipeline

  model_id = "deepset/roberta-base-squad2"
  tokenizer = AutoTokenizer.from_pretrained(model_id)
- model = AutoModelForQuestionAnswering.from_pretrained(model_id)
+ model = ORTModelForQuestionAnswering.from_pretrained("roberta_base_qa_onnx")
  qa_pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)
  question = "What's Optimum?"
  context = "Optimum is an awesome library everyone should use!"
  results = qa_pipe(question=question, context=context)
```

More details on how to run ONNX models with `ORTModelForXXX` classes [here](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/models).

### TensorFlow Lite

Before you begin, make sure you have all the necessary libraries installed :

```bash
pip install optimum[exporters-tf]
```

Just as for ONNX, it is possible to export models to [TensorFlow Lite](https://www.tensorflow.org/lite) and quantize them:

```plain
optimum-cli export tflite \
  -m deepset/roberta-base-squad2 \
  --sequence_length 384  \
  --quantize int8-dynamic roberta_tflite_model
```

## Accelerated training

🤗 Optimum provides wrappers around the original 🤗 Transformers [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) to enable training on powerful hardware easily.
We support many providers:

- Habana's Gaudi processors
- AWS Trainium instances, check [here](https://huggingface.co/docs/optimum-neuron/en/guides/distributed_training)
- ONNX Runtime (optimized for GPUs)

### Habana

Before you begin, make sure you have all the necessary libraries installed :

```bash
pip install --upgrade --upgrade-strategy eager optimum[habana]
```

```diff
- from transformers import Trainer, TrainingArguments
+ from optimum.habana import GaudiTrainer, GaudiTrainingArguments

  # Download a pretrained model from the Hub
  model = AutoModelForXxx.from_pretrained("bert-base-uncased")

  # Define the training arguments
- training_args = TrainingArguments(
+ training_args = GaudiTrainingArguments(
      output_dir="path/to/save/folder/",
+     use_habana=True,
+     use_lazy_mode=True,
+     gaudi_config_name="Habana/bert-base-uncased",
      ...
  )

  # Initialize the trainer
- trainer = Trainer(
+ trainer = GaudiTrainer(
      model=model,
      args=training_args,
      train_dataset=train_dataset,
      ...
  )

  # Use Habana Gaudi processor for training!
  trainer.train()
```

You can find more examples in the [documentation](https://huggingface.co/docs/optimum/habana/quickstart) and in the [examples](https://github.com/huggingface/optimum-habana/tree/main/examples).

### ONNX Runtime

```diff
- from transformers import Trainer, TrainingArguments
+ from optimum.onnxruntime import ORTTrainer, ORTTrainingArguments

  # Download a pretrained model from the Hub
  model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")

  # Define the training arguments
- training_args = TrainingArguments(
+ training_args = ORTTrainingArguments(
      output_dir="path/to/save/folder/",
      optim="adamw_ort_fused",
      ...
  )

  # Create a ONNX Runtime Trainer
- trainer = Trainer(
+ trainer = ORTTrainer(
      model=model,
      args=training_args,
      train_dataset=train_dataset,
      ...
  )

  # Use ONNX Runtime for training!
  trainer.train()
```

You can find more examples in the [documentation](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/trainer) and in the [examples](https://github.com/huggingface/optimum/tree/main/examples/onnxruntime/training).


# Backend for [tough-dev.school](https://tough-dev.school/)

![CI](https://github.com/tough-dev-school/education-backend/actions/workflows/ci.yml/badge.svg) [![Maintainability](https://api.codeclimate.com/v1/badges/fe9fb0b64052a426f355/maintainability)](https://codeclimate.com/github/f213/education-backend/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/fe9fb0b64052a426f355/test_coverage)](https://codeclimate.com/github/f213/education-backend/test_coverage)

Django-based production project, integrated with Tinkoff, Dashamail, Postmark, S3 and telegram. Frontend is built on vue.js in the [separate repo](https://github.com/tough-dev-school/lms-frontend-v2).

## Configuration

Configuration is stored in `src/core/.env`, for examples see `src/core/.env.ci`

## Installing on a local machine

This project requires python 3.11. Deps are managed by [Poetry](https://python-poetry.org/).

Install requirements:

```bash
poetry install --no-root
```

Configure postgres and redis. It's convenient to use docker and docker-compose:

```bash
docker compose up -d
```

If you don't have access to de-anonymized db image use `postgres:13.6-alpine` in `docker-compose.yml` instead:

```yaml
postgres:
    image: postgres:13.6-alpine
    ...
```

Run the server:

```bash
cp src/core/.env.ci src/core/.env

poetry run python src/manage.py migrate
poetry run python src/manage.py createsuperuser

make server
```

Testing:

```bash
# run lint
make lint

# run unit tests
make test
```

## Backend Code requirements

### Style

* Obey [django's style guide](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style).
* Configure your IDE to use [flake8](https://pypi.python.org/pypi/flake8) for checking your python code. For running flake8 manualy, do `cd src && flake8`
* Prefer English over your native language in comments and commit messages.
* Commit messages should contain the unique id of issue they are linked to (refs #100500)
* Every model and a model method should have a docstring.

### Code organisation

* KISS and DRY.
* Obey [django best practices](http://django-best-practices.readthedocs.io/en/latest/index.html)
* If you want to implement some business logic — make a service for that. Service examples: [UserCreator](https://github.com/tough-dev-school/education-backend/blob/master/src/users/services/user_creator.py#L22), [OrderCreator](https://github.com/tough-dev-school/education-backend/blob/master/src/orders/services/order_creator.py#L11)
* **No logic is allowed within the views or templates**. Only services and models.
* Use PEP-484 [type hints](https://www.python.org/dev/peps/pep-0484/) when possible.
* Prefer [Manager](https://docs.djangoproject.com/en/1.10/topics/db/managers/) methods over static methods.
* Do not use [signals](https://docs.djangoproject.com/en/1.10/topics/signals/) for business logic. Signals are good only for notification purposes.
* No l10n is allowed in python code, use [django translation](https://docs.djangoproject.com/en/1.10/topics/i18n/translation/).