# README for cardano-node-tests

[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

System and end-to-end (E2E) tests for cardano-node.

Check this [documentation](https://tests.cardano.intersectmbo.org) for more details.

## Running tests using Github Actions

The easiest way to run the tests is by using Github Actions.

1. fork this repository
1. enable Github Actions in your fork ("Settings" / "Actions" / "General" / "Actions permissions", check "Allow all actions and reusable workflows")
1. go to "Actions", select "01 Regression tests" (or "02 Regression tests with db-sync")
1. select "Run workflow"

![Run workflow](https://user-images.githubusercontent.com/2352619/209117914-ef3afb38-2b8b-4a4f-a03f-c9b52bccc5ba.png)

## Running tests using Nix

1. install and configure nix, follow [cardano-node documentation](https://github.com/IntersectMBO/cardano-node/blob/master/doc/getting-started/building-the-node-using-nix.md)

1. clone this repo

1. run the tests

    ```sh
    ./.github/regression.sh
    ```

---
**NOTE** <!-- markdownlint-disable-line MD036 -->

When using `CI_BYRON_CLUSTER`, it takes ~ 30 minutes for local cluster instance to get from Byron to Babbage. If it seems that tests are stuck, they are likely just waiting for local cluster instances to be fully started.

---

## Running individual tests on persistent local cluster using Nix

Sometimes it is useful to run individual tests and keep the local cluster running in between test runs.

1. run nix shell that has all the needed dependencies

    ```sh
    nix flake update --accept-flake-config --override-input cardano-node "github:IntersectMBO/cardano-node/master"  # change `master` to rev you want
    nix develop --accept-flake-config .#venv
    ```

1. prepare testing environment

    ```sh
    source ./prepare_test_env.sh babbage  # 'conway' is also supported
    ```

1. start the cluster instance

    ```sh
    ./dev_workdir/babbage_fast/start-cluster
    ```

1. run some test

    ```sh
    pytest -s -k test_minting_one_token cardano_node_tests/tests/tests_plutus
    # or run some tests and see all the executed `cardano-cli` commands
    pytest -s --log-level=debug -k test_minting_one_token cardano_node_tests/tests/tests_plutus
    ```

1. stop the cluster instance

    ```sh
    ./dev_workdir/babbage_fast/stop-cluster
    ```

## Variables for configuring testrun

Tests execution can be configured using env variables.

* `SCHEDULING_LOG` – specifies the path to the file where log messages for tests and cluster instance scheduler are stored
* `PYTEST_ARGS` – specifies additional arguments for pytest (default: unset)
* `MARKEXPR` – specifies marker expression for pytest (default: unset)
* `TEST_THREADS` – specifies the number of pytest workers (default: 20)
* `CLUSTERS_COUNT` – number of cluster instances that will be started (default: 9)
* `CLUSTER_ERA` – cluster era for Cardano node – used for selecting the correct cluster start script (default: babbage)
* `TX_ERA` (deprecated) – era for transactions – can be used for creating Shelley-era (Allegra-era, ...) transactions (default: unset)
* `COMMAND_ERA` – era for cardano-cli commands – can be used for creating Shelley-era (Allegra-era, ...) transactions (default: unset)
* `NUM_POOLS` – number of stake pools created in each cluster instance (default: 3)
* `ENABLE_P2P` – use P2P networking instead of the default legacy networking (default: unset)
* `MIXED_P2P` – use mix of P2P and legacy networking; half of stake pools using legacy and the other half P2P (default: unset)
* `DB_BACKEND` – 'mem' or 'lmdb', default is 'mem' (or legacy) backend if unset (default: unset)
* `SCRIPTS_DIRNAME` – path to a dir with local cluster start/stop scripts and configuration files (default: unset)
* `BOOTSTRAP_DIR` – path to a bootstrap dir for the given testnet (genesis files, config files, faucet data) (default: unset)
* `KEEP_CLUSTERS_RUNNING` – don't stop cluster instances after testrun is finished
  (WARNING: this implies interactive behavior when running tests using the `./.github/regression.sh` script)

When running tests using the `./.github/regression.sh` script, you can also use

* `CI_BYRON_CLUSTER` - start local cluster in Byron era, and progress to later eras by HFs (same effect as `SCRIPTS_DIRNAME=babbage`)
* `NODE_REV` - revison of `cardano-node` (default: 'master')
* `DBSYNC_REV` - revison of `cardano-db-sync` (default: unset; db-sync is not used by default)
* `CARDANO_CLI_REV` - revison of `cardano-cli` (default: unset; cardano-cli bundled in cardano-node repo is used by default)
* `PLUTUS_APPS_REV` - revison of `plutus-apps` (default: 'main')

For example:

* running tests on local cluster intances, each with 6 stake pools and mix of P2P and legacy networking

    ```sh
    NUM_POOLS=6 MIXED_P2P=1 ./.github/regression.sh
    ```

* running tests on local cluster intances using 15 pytest workers, Babbage cluster era, Alonzo transaction era, cluster scripts that start a cluster directly in Babbage era, and selecting only tests without 'long' marker that also match the given `-k` pytest argument

    ```sh
    TEST_THREADS=15 CLUSTER_ERA=babbage TX_ERA=alonzo SCRIPTS_DIRNAME=babbage_fast PYTEST_ARGS="-k 'test_stake_pool_low_cost or test_reward_amount'" MARKEXPR="not long" ./.github/regression.sh
    ```

* running tests on Shelley-qa testnet with '8.0.0' release of `cardano-node`

    ```sh
    NODE_REV=8.0.0 BOOTSTRAP_DIR=~/tmp/shelley_qa_config/ ./.github/regression.sh
    ```

## Local usage for tests development (useful only for tests developers)

Install and configure nix, follow [cardano-node documentation](https://github.com/IntersectMBO/cardano-node/blob/master/doc/getting-started/building-the-node-using-nix.md).
Install and configure poetry, follow [Poetry documentation](https://python-poetry.org/docs/#installation).

### Preparing Python virtual environment

Create a Python virtual environment (requires Python v3.8 or newer) and install this package together with development requirements:

```sh
./setup_dev_venv.sh
```

### Running development cluster

When running tests, the testing framework starts and stops cluster instances as needed. That is not ideal for test development, as starting a cluster instance can take up to several epochs (to get from Byron to Babbage). To keep the Cardano cluster running in between test runs, one needs to start it in 'development mode':

1. cd to 'cardano-node' repo

    ```sh
    cd ../cardano-node
    ```

1. update and checkout the desired commit/tag

    ```sh
    git checkout master
    git pull origin master
    git fetch --all --tags
    git checkout tags/<tag>
    ```

1. launch devops shell

    ```sh
    nix develop .#devops
    ```

1. cd back to 'cardano-node-tests' repo

    ```sh
    cd ../cardano-node-tests
    ```

1. activate virtual env

    ```sh
    poetry shell
    ```

1. add virtual env to `PYTHONPATH`

    ```sh
    export PYTHONPATH="$(echo $VIRTUAL_ENV/lib/python3*/site-packages)":$PYTHONPATH
    ```

1. set env variables

    ```sh
    export CARDANO_NODE_SOCKET_PATH="$PWD/dev_workdir/state-cluster0/bft1.socket" DEV_CLUSTER_RUNNING=1
    mkdir -p "${CARDANO_NODE_SOCKET_PATH%/*}"
    ```

1. prepare cluster scripts for starting local cluster directly in Babbage era

    ```sh
    mkdir -p dev_workdir
    prepare-cluster-scripts -c -d dev_workdir/babbage_fast -s cardano_node_tests/cluster_scripts/babbage_fast/
    ```

1. start the cluster instance in development mode

    ```sh
    ./dev_workdir/babbage_fast/start-cluster
    ```

After the cluster starts, keys and configuration files are available in the `<your path to cardano-node repo>/state-cluster0` directory. The pool-related files and keys are located in the `nodes` subdirectory, genesis keys in the `shelley` and `byron` subdirectories, and payment address with initial funds and related keys in the `byron` subdirectory. The local faucet address and related key files are stored in the `addrs_data` subdirectory.

### Restarting development cluster

To restart the running cluster (eg, after upgrading `cardano-node` and `cardano-cli` binaries), run:

```sh
./scripts/restart_dev_cluster.sh
```

---
**NOTE** <!-- markdownlint-disable-line MD036 -->

Restaring the running development cluster is useful mainly when using the "babbage" start scripts (not the "babbage_fast" version). It takes ~ 30 minutes for the local cluster instance to get from Byron to Babbage. Starting local cluster using the "babbage_fast" version takes less than 1 minute.

---

### Checking the development environment

To check that the development environment was correctly setup, run the `./check_dev_env.sh` script.

```text
$ ./check_dev_env.sh
'cardano-node' available: ✔
'cardano-cli' available: ✔
'python' available: ✔
'pytest' available: ✔
'nix-shell' available: ✔
'jq' available: ✔
'supervisord' available: ✔
'supervisorctl' available: ✔
'bech32' available: ✔
inside nix shell: ✔
in repo root: ✔
DEV cluster: ✔
python works: ✔
in python venv: ✔
venv in PYTHONPATH: ✔
cardano-node-tests installed: ✔
pytest works: ✔
same version of node and cli: ✔
socket path set: ✔
socket path correct: ✔
socket path exists: ✔
cluster era: babbage
transaction era: babbage
using dbsync (optional): ✔
dbsync available: ✔
P2P network (optional): -
```

### Running individual tests

Example:

```sh
pytest -k "test_name1 or test_name2" cardano_node_tests
pytest -m "not long" cardano_node_tests
pytest -m smoke cardano_node_tests/tests/test_governance.py
```

### Running linters

It is sufficient to activate the Python virtual environment before running linters, development cluster is not needed:

1. activate virtual env

    ```sh
    poetry shell
    ```

1. run linters

    ```sh
    make lint
    ```

### Installing `cardano-clusterlib` in development mode

Sometimes it is useful to test local changes made to [cardano-clusterlib](https://github.com/IntersectMBO/cardano-clusterlib-py).
To install cardano-clusterlib in development mode:

1. activate virtual env

    ```sh
    poetry shell
    ```

1. enable legacy behavior (needed by `mypy` and `pylint` linters)

    ```sh
    export SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
    ```

1. update virtual env (answer 'y' to question "Install into the current virtual env? [y/N]")

    ```sh
    ./setup_dev_venv.sh
    ```

1. uninstall `cardano-clusterlib` installed by poetry

    ```sh
    pip uninstall cardano-clusterlib
    ```

1. cd to 'cardano-clusterlib-py' repo

    ```sh
    cd ../cardano-clusterlib-py
    ```

1. install `cardano-clusterlib` in development mode

    ```sh
    pip install -e .
    ```

1. cd back to 'cardano-node-tests' repo

    ```sh
    cd -
    ```

1. check that you are really using cardano-clusterlib files from your local repo

    ```sh
    python -c 'from cardano_clusterlib import clusterlib_klass; print(clusterlib_klass.__file__)'
    ```

Note that after you run `poetry install` (eg through running `./setup_dev_venv.sh`), poetry will reinstall `cardano-clusterlib`. If you want to keep using cardano-clusterlib in development mode, you'll need to repeat the steps above.

### Updating dependencies using poetry

Edit `pyproject.toml` and run

```sh
./poetry_update_deps.sh
```

### Building documentation

Build and deploy documentation:

```sh
./deploy_doc.sh
```

## Contributing

Install this package and its dependencies as described above.

Run `pre-commit install` to set up the Git hook scripts that will check your changes before every commit. Alternatively, run `make lint` manually before pushing your changes.

Follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html), with the exception that formatting is handled automatically by [Black](https://github.com/psf/black) (through the `pre-commit` command).

See the [CONTRIBUTING](https://github.com/IntersectMBO/cardano-node-tests/blob/master/CONTRIBUTING.md) document for more details.


# TIAGo robot

This package contains the description, controllers and bringup for all possible TIAGo configurations (end effectors, force torque sensors..).

To make maintenance easier, the `tiago.urdf.xacro` takes arguments that specify whether the robot has an arm or not, the end effector type, force torque sensor, laser model and many other parameters.

The other configuration files that differ between robot configurations are generated from template files.

The templates are written using [empy](https://pypi.org/project/empy/) and have the extension `.em`.

To regenerate a group of files, you must execute `ros2 run tiago_bringup regen_em_file.py EM_FILE_NAME` from the directory where the `.em` file is.


# <img alt="Logo" width="50px" src="./images/compliance-trestle-800x800.png" style="vertical-align: middle;" /> Compliance-trestle (also known as `trestle`)

![[OS Compatibility](#prerequisites)](https://img.shields.io/badge/platform-osx%20%7C%20linux%20%7C%20windows-orange.svg)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/compliance-trestle)
![[Pre-commit](https://github.com/pre-commit/pre-commit)](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)
![[Code Coverage](https://sonarcloud.io/dashboard?id=compliance-trestle)](https://sonarcloud.io/api/project_badges/measure?project=compliance-trestle&metric=coverage)
![[Quality gate](https://sonarcloud.io/dashboard?id=compliance-trestle)](https://sonarcloud.io/api/project_badges/measure?project=compliance-trestle&metric=alert_status)
![[Pypi](https://pypi.org/project/compliance-trestle/)](https://img.shields.io/pypi/dm/compliance-trestle)
![GitHub Actions status](https://img.shields.io/github/workflow/status/oscal-compass/compliance-trestle/Trestle%20PR%20pipeline?event=push)

Trestle is an ensemble of tools that enable the creation, validation, and governance of documentation artifacts for compliance needs. It leverages NIST's [OSCAL](https://pages.nist.gov/OSCAL/) as a standard data format for interchange between tools and people, and provides an opinionated approach to OSCAL adoption.

Trestle is designed to operate as a CICD pipeline running on top of compliance artifacts in `git`, to provide transparency for the state of compliance across multiple stakeholders in an environment friendly to developers. Trestle passes the generated artifacts onto tools that orchestrate the enforcement, measurement, and reporting of compliance.

It also provides tooling to manage OSCAL documents in a more human-friendly manner. By splitting large OSCAL data structures into smaller and easier to edit sub-structures, creation and maintenance of these artifacts can follow normal `git` workflows including peer review via pull request, versioning, releases/tagging.

Trestle provides three separate but related functions in the compliance space:

- Manage OSCAL documents to allow editing and manipulation while making sure the schemas are enforced
- Transform documents from other formats to OSCAL
- Provide support and governance to author compliance content as markdown and drawio.

Trestle provides tooling to help orchestrate the compliance process across a number of dimensions:

- Help manage OSCAL documents in a more human-friendly manner by expanding the large OSCAL data structures into smaller and easier to edit sub-structures while making sure the schemas are enforced.
- Transform documents from other formats to OSCAL
- Provide governance for markdown documents and enforce consistency of format and content based on specified templates
- Tooling manage authoring and governance of markdown and drawio files within a repository.
- Support within trestle to streamline management within a managed git environment.
- An underlying object model that supports developers interacting with OSCAL artifacts.

## Important Note:

The current version of trestle supports NIST OSCAL 1.1.2 as well as previous versions 1.1.x and 1.0.x. All files created by trestle will be output as OSCAL version 1.1.2.

There was a breaking change in OSCAL moving from version 1.0.0 to 1.0.2 mainly due to `prop` becoming `props` in AssessmentResults.  Those who require strict OSCAL 1.0.0 please use trestle version 0.37.x.  That version is stable but will not have any features added, and we encourage all users to move to OSCAL 1.1.2. OSCAL version 1.0.0 files are still handled on import but any AssessmentResults must conform to the `props` in AssessmentResults OSCAL specification.

## Why Trestle

Compliance suffers from being a complex topic that is hard to articulate simply. It involves complete and accurate execution of multiple procedures across many disciplines (e.g. IT, HR, management) with periodic verification and audit of those procedures against controls.

While it is possible to manage the description of controls and how an organisation implements them in ad hoc ways with general tools (spreadsheets, documents), this is hard to maintain for multiple accreditations and, in the IT domain at least, creates a barrier between the compliance efforts and the people doing daily work (DevOps staff).

Trestle aims to reduce or remove this barrier by bringing the maintenance of control descriptions into the DevOps domain. The goal is to have changes to the system (for example, updates to configuration management) easily related to the controls impacted, and to enable modification of those controls as required in concert with the system changes.

Trestle implicitly provides a core opinionated workflow driven by its pipeline to allow standardized interlocks with other compliance tooling platforms.

## Machine readable compliance format

Compliance activities at scale, whether size of estate or number of accreditations, require automation to be successful and repeatable. OSCAL as a standard allows teams to bridge between the "Governance" layer and operational tools.

By building human managed artifacts into OSCAL, Trestle is not only able to validate the integrity of the artifacts that people generate - it also enables reuse and sharing of artifacts, and furthermore can provide suitable input into tools that automate operational compliance.

## Supported OSCAL elements and extensions

`trestle` implicitly supports all OSCAL schemas for use within the object model. The development roadmap for `trestle` includes adding workflow around specific elements / objects that is opinionated.

## Supported file formats for OSCAL objects.

OSCAL supports `xml`, `json` and `yaml` with their [metaschema](https://github.com/usnistgov/metaschema) tooling. Trestle
natively supports only `json` and `yaml` formats at this time.

Future roadmap anticipates that support for xml [import](https://github.com/oscal-compass/compliance-trestle/issues/177) and [upstream references](https://github.com/oscal-compass/compliance-trestle/issues/178) will be enabled. However, it is expected
that full support will remain only for `json` and  `yaml`.

Users needing to import XML OSCAL artifacts are recommended to look at NIST's XML to json conversion page [here](https://github.com/usnistgov/OSCAL/blob/main/build/README.md#converters).

## Python codebase, easy installation via pip

Trestle runs on almost all Python platforms (e.g. Linux, Mac, Windows), is available on PyPi and can be easily installed via pip. It is under active development and new releases are made available regularly.\
To install run: `pip install compliance-trestle`\
See [Install trestle in a python virtual environment](https://oscal-compass.github.io/compliance-trestle/python_trestle_setup/) for the full installation guide.

## Complete documentation and tutorials

Complete documentation, tutorials, and background on compliance can be found [here](https://oscal-compass.github.io/compliance-trestle).

## Agile Authoring

A trestle-based agile authoring repository setup tool, documentation and tutorial can be found [here](https://github.com/oscal-compass/compliance-trestle-agile-authoring).

Agile authoring comprises the following beneficial features:

- based on OSCAL documents behind-the-scenes
- employs GIT for document control and access
- exposes text (markdown) and spread sheets (csv) to ease management of compliance artifacts
- implements compliance digitization for improved audit readiness and cost effectiveness

## Demos

A collection of demos utilizing trestle can be found in the related project [compliance-trestle-demos](https://github.com/oscal-compass/compliance-trestle-demos).

## Development status

Compliance trestle is currently stable and is based on NIST OSCAL version 1.1.2, with active development continuing.

## Community meetings and communications

Please refer to the community [README](https://github.com/oscal-compass/community/blob/main/README.md) for communication details.

## Contributing to Trestle

Our project welcomes external contributions. Please consult [contributing](https://oscal-compass.github.io/compliance-trestle/contributing/mkdocs_contributing/) to get started.

## License & Authors

If you would like to see the detailed LICENSE click [here](LICENSE).
Consult [contributors](https://github.com/oscal-compass/compliance-trestle/graphs/contributors) for a list of authors and [maintainers](MAINTAINERS.md) for the core team.

```text
# Copyright (c) 2020 IBM Corp. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

```


[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/guardrails-ai)
[![CI](https://github.com/guardrails-ai/guardrails/actions/workflows/ci.yml/badge.svg)](https://github.com/guardrails-ai/guardrails/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/guardrails-ai/guardrails/graph/badge.svg?token=CPkjw91Ngo)](https://codecov.io/gh/guardrails-ai/guardrails)
[![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/)
[![X (formerly Twitter) Follow](https://img.shields.io/twitter/follow/guardrails_ai)](https://x.com/guardrails_ai)
[![Discord](https://img.shields.io/discord/1085077079697150023?logo=discord&label=support&link=https%3A%2F%2Fdiscord.gg%2Fgw4cR9QvYE)](https://discord.gg/U9RKkZSBgx)
[![Static Badge](https://img.shields.io/badge/Docs-blue?link=https%3A%2F%2Fwww.guardrailsai.com%2Fdocs)](https://www.guardrailsai.com/docs)
[![Static Badge](https://img.shields.io/badge/Blog-blue?link=https%3A%2F%2Fwww.guardrailsai.com%2Fblog)](https://www.guardrailsai.com/blog)

## What is Guardrails?

Guardrails is a Python framework that helps build reliable AI applications by performing two key functions:
1. Guardrails runs Input/Output Guards in your application that detect, quantify and mitigate the presence of specific types of risks. To look at the full suite of risks, check out [Guardrails Hub](https://hub.guardrailsai.com/).
2. Guardrails help you generate structured data from LLMs.

### Guardrails Hub

Guardrails Hub is a collection of pre-built measures of specific types of risks (called 'validators'). Multiple validators can be combined together into Input and Output Guards that intercept the inputs and outputs of LLMs. Visit [Guardrails Hub](https://hub.guardrailsai.com/) to see the full list of validators and their documentation.

## Installation

```python
pip install guardrails-ai
```

## Getting Started

### Create Input and Output Guards for LLM Validation

1. Download and configure the Guardrails Hub CLI.

    ```bash
    pip install guardrails-ai
    guardrails configure
    ```
2. Install a guardrail from Guardrails Hub.

    ```bash
    guardrails hub install hub://guardrails/regex_match
    ```
3. Create a Guard from the installed guardrail.

    ```python
    from guardrails import Guard, OnFailAction
    from guardrails.hub import RegexMatch

    guard = Guard().use(
        RegexMatch, regex="\(?\d{3}\)?-? *\d{3}-? *-?\d{4}", on_fail=OnFailAction.EXCEPTION
    )

    guard.validate("123-456-7890")  # Guardrail passes

    try:
        guard.validate("1234-789-0000")  # Guardrail fails
    except Exception as e:
        print(e)
    ```
    Output:
    ```console
    Validation failed for field with errors: Result must match \(?\d{3}\)?-? *\d{3}-? *-?\d{4}
    ```
4. Run multiple guardrails within a Guard.
    First, install the necessary guardrails from Guardrails Hub.

    ```bash
    guardrails hub install hub://guardrails/competitor_check
    guardrails hub install hub://guardrails/toxic_language
    ```

    Then, create a Guard from the installed guardrails.

    ```python
    from guardrails import Guard, OnFailAction
    from guardrails.hub import CompetitorCheck, ToxicLanguage

    guard = Guard().use_many(
        CompetitorCheck(["Apple", "Microsoft", "Google"], on_fail=OnFailAction.EXCEPTION),
        ToxicLanguage(threshold=0.5, validation_method="sentence", on_fail=OnFailAction.EXCEPTION),),
    )

    guard.validate(
        """An apple a day keeps a doctor away.
        This is good advice for keeping your health."""
    )  # Both the guardrails pass

    try:
        guard.validate(
            """Shut the hell up! Apple just released a new iPhone."""
        )  # Both the guardrails fail
    except Exception as e:
        print(e)
    ```
    Output:
    ```console
    Validation failed for field with errors: Found the following competitors: [['Apple']]. Please avoid naming those competitors next time, The following sentences in your response were found to be toxic:

    - Shut the hell up!
    ```

### Use Guardrails to generate structured data from LLMs

Let's go through an example where we ask an LLM to generate fake pet names. To do this, we'll create a Pydantic [BaseModel](https://docs.pydantic.dev/latest/api/base_model/) that represents the structure of the output we want.

```py
from pydantic import BaseModel, Field

class Pet(BaseModel):
    pet_type: str = Field(description="Species of pet")
    name: str = Field(description="a unique pet name")
```

Now, create a Guard from the `Pet` class. The Guard can be used to call the LLM in a manner so that the output is formatted to the `Pet` class. Under the hood, this is done by either of two methods:
1. Function calling: For LLMs that support function calling, we generate structured data using the function call syntax.
2. Prompt optimization: For LLMs that don't support function calling, we add the schema of the expected output to the prompt so that the LLM can generate structured data.

```py
from guardrails import Guard
import openai

prompt = """
    What kind of pet should I get and what should I name it?

    ${gr.complete_json_suffix_v2}
"""
guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)

raw_output, validated_output, *rest = guard(
    llm_api=openai.completions.create,
    engine="gpt-3.5-turbo-instruct"
)

print(validated_output)
```

This prints:
```
{
    "pet_type": "dog",
    "name": "Buddy
}
```

## FAQ

#### I'm running into issues with Guardrails. Where can I get help?

You can reach out to us on [Discord](https://discord.gg/gw4cR9QvYE) or [Twitter](https://twitter.com/guardrails_ai).

#### Can I use Guardrails with any LLM?

Yes, Guardrails can be used with proprietary and open-source LLMs. Check out this guide on [how to use Guardrails with any LLM](https://www.guardrailsai.com/docs/how_to_guides/llm_api_wrappers).

#### Can I create my own validators?

Yes, you can create your own validators and contribute them to Guardrails Hub. Check out this guide on [how to create your own validators](https://www.guardrailsai.com/docs/hub/how_to_guides/custom_validator).

#### Does Guardrails support other languages?

Guardrails can be used with Python and JavaScript. Check out the docs on how to use Guardrails from JavaScript. We are working on adding support for other languages. If you would like to contribute to Guardrails, please reach out to us on [Discord](https://discord.gg/gw4cR9QvYE) or [Twitter](https://twitter.com/guardrails_ai).

## Contributing

We welcome contributions to Guardrails!

Get started by checking out Github issues and check out the [Contributing Guide](CONTRIBUTING.md). Feel free to open an issue, or reach out if you would like to add to the project!


polars-upgrade
==============

[![PyPI version](https://badge.fury.io/py/polars-upgrade.svg)](https://badge.fury.io/py/polars-upgrade)

Automatically upgrade your Polars code so it's compatible with future versions.

## Installation

Easy:
```
pip install -U polars-upgrade
```

## Usage (command-line)

Run
```
polars-upgrade my_project --target-version=0.20.31
```
from the command line. Replace `0.20.31` and `my_project` with your Polars version,
and the name of your directory.

NOTE: this tool will modify your code!
You're advised to stage your files before running it.

## Usage (pre-commit hook)

```yaml
-   repo: https://github.com/MarcoGorelli/polars-upgrade
    rev: 0.3.6  # polars-upgrade version goes here
    hooks:
    -   id: polars-upgrade
        args: [--target-version=0.20.31]  # Polars version goes here
```

## Usage (Jupyter Notebooks)

Install [nbqa](https://github.com/nbQA-dev/nbQA) and then run
```
nbqa polars_upgrade my_project --target-version=0.20.31
```

## Usage (library)

In a Python script:
```python
from polars_upgrade import rewrite, Settings

src = """\
import polars as pl
df.select(pl.count())
"""
settings = Settings(target_version=(0, 20, 4))
output = rewrite(src, settings=settings)
print(output)
```
Output:
```
import polars as pl
df.select(pl.len())
```

If your snippet does _not_ include `import polars` or `import as pl`,
then you will also need to provide `pl` and/or `polars` to `aliases`, else `polars-upgrade` will
not perform the rewrite. Example:

```python
from polars_upgrade import rewrite, Settings

src = """\
df.select(pl.count())
"""
settings = Settings(target_version=(0, 20, 4))
output = rewrite(src, settings=settings, aliases={'pl'})
print(output)
```
Output:
```
df.select(pl.len())
```

## Supported rewrites

### Version 0.18.12+

```diff
- pl.avg
+ pl.mean
```

### Version 0.19.0+

```diff
- df.groupby_dynamic
+ df.group_by_dynamic
- df.groupby_rolling
+ df.rolling
- df.rolling('ts', period='3d').apply
+ df.rolling('ts', period='3d').map_groups
- pl.col('a').rolling_apply
+ pl.col('a').rolling_map
- pl.col('a').apply
+ pl.col('a').map_elements
- pl.col('a').map
+ pl.col('a').map_batches
- pl.map
+ pl.map_batches
- pl.apply
+ pl.map_groups
- pl.col('a').any(drop_nulls=True)
+ pl.col('a').any(ignore_nulls=True)
- pl.col('a').all(drop_nulls=True)
+ pl.col('a').all(ignore_nulls=True)
- pl.col('a').value_counts(multithreaded=True)
+ pl.col('a').value_counts(parallel=True)
```

### Version 0.19.2+
```diff
- pl.col('a').is_not
+ pl.col('a').not_
```

### Version 0.19.3+

```diff
- pl.enable_string_cache(True)
+ pl.enable_string_cache()
- pl.enable_string_cache(False)
+ pl.disable_string_cache()
- pl.col('a').list.count_match
+ pl.col('a').list.count_matches
- pl.col('a').is_last
+ pl.col('a').is_last_distinct
- pl.col('a').is_first
+ pl.col('a').is_first_distinct
- pl.col('a').str.strip
+ pl.col('a').str.strip_chars
- pl.col('a').str.lstrip
+ pl.col('a').str.strip_chars_start
- pl.col('a').str.rstrip
+ pl.col('a').str.strip_chars_end
- pl.col('a').str.count_match
+ pl.col('a').str.count_matches
- pl.col("dt").dt.offset_by("1mo_saturating")
+ pl.col("dt").dt.offset_by("1mo")
```

### Version 0.19.4+
```diff
- df.group_by_dynamic('ts', every='3d', truncate=True)
+ df.group_by_dynamic('ts', every='3d', label='left')
- df.group_by_dynamic('ts', every='3d', truncate=False)
+ df.group_by_dynamic('ts', every='3d', label='datapoint')
```

### Version 0.19.8+
```diff
- pl.col('a').list.lengths
+ pl.col('a').list.len
- pl.col('a').str.lengths
+ pl.col('a').str.len_bytes
- pl.col('a').str.n_chars
+ pl.col('a').str.len_chars
```

### Version 0.19.11+
```diff
- pl.col('a').shift(periods=4)
+ pl.col('a').shift(n=4)
- pl.col('a').shift_and_fill(periods=4)
+ pl.col('a').shift_and_fill(n=4)
- pl.col('a').list.shift(periods=4)
+ pl.col('a').list.shift(n=4)
- pl.col('a').map_dict(remapping={1: 2})
+ pl.col('a').map_dict(mapping={1: 2})
```

### Version 0.19.12+
```diff
- pl.col('a').keep_name
+ pl.col('a').name.keep
- pl.col('a').suffix
+ pl.col('a').name.suffix
- pl.col('a').prefix
+ pl.col('a').name.prefix
- pl.col('a').map_alias
+ pl.col('a').name.map
- pl.col('a').str.ljust
+ pl.col('a').str.pad_end
- pl.col('a').str.rjust
+ pl.col('a').str.pad_start
- pl.col('a').zfill(alignment=3)
+ pl.col('a').zfill(length=3)
- pl.col('a').ljust(width=3)
+ pl.col('a').ljust(length=3)
- pl.col('a').rjust(width=3)
+ pl.col('a').rjust(length=3)
```

### Version 0.19.13+
```diff
- pl.col('a').dt.milliseconds
+ pl.col('a').dt.total_milliseconds
- pl.col('a').dt.microseconds
+ pl.col('a').dt.total_microseconds
- pl.col('a').dt.nanoseconds
+ pl.col('a').dt.total_nanoseconds
```
(and so on for other units)

### Version 0.19.14+
```diff
- pl.col('a').list.take
+ pl.col('a').list.gather
- pl.col('a').cumcount
+ pl.col('a').cum_count
- pl.col('a').cummax
+ pl.col('a').cum_max
- pl.col('a').cummin
+ pl.col('a').cum_min
- pl.col('a').cumprod
+ pl.col('a').cum_prod
- pl.col('a').cumsum
+ pl.col('a').cum_sum
- pl.col('a').cumcount
+ pl.col('a').cum_count
- pl.col('a').take
+ pl.col('a').gather
- pl.col('a').take_every
+ pl.col('a').gather_every
- pl.cumsum
+ pl.cum_sum
- pl.cumfold
+ pl.cum_fold
- pl.cumreduce
+ pl.cum_reduce
- pl.cumsum_horizontal
+ pl.cum_sum_horizontal
- pl.col('a').list.take(index=[1, 2])
+ pl.col('a').list.take(indices=[1, 2])
- pl.col('a').str.parse_int(radix=1)
+ pl.col('a').str.parse_int(base=1)
```

### Version 0.19.15+
```diff
- pl.col('a').str.json_extract
+ pl.col('a').str.json_decode
```

### Version 0.19.16+
```diff
- pl.col('a').map_dict({'a': 'b'})
+ pl.col('a').replace({'a': 'b'}, default=None)
- pl.col('a').map_dict({'a': 'b'}, default='c')
+ pl.col('a').replace({'a': 'b'}, default='c')
```

### Version 0.20.0+
```diff
- df.write_database(table_name='foo', if_exists="append")
+ df.write_database(table_name='foo', if_table_exists="append")
```

### Version 0.20.4+
```diff
- pl.col('a').where
+ pl.col('a').filter
- pl.count()
+ pl.len()
- df.with_row_count('row_number')
+ df.with_row_index('row_number')
- pl.scan_ndjson(source, row_count_name='foo', row_count_offset=3)
+ pl.scan_ndjson(source, row_index_name='foo', row_index_offset=3)
[...and similarly for `read_csv`, `read_csv_batched`, `scan_csv`, `read_ipc`, `read_ipc_stream`, `scan_ipc`, `read_parquet`, `scan_parquet`]
```

### Version 0.20.5+
```diff
- df.pivot(index=index, values=values, columns=columns, aggregate_function='count')
+ df.pivot(index=index, values=values, columns=columns, aggregate_function='len')
```

### Version 0.20.6+
```diff
- pl.read_excel(source, xlsx2csv_options=options, read_csv_options=read_options)
+ pl.read_excel(source, engine_options=options, read_options=read_options)
```

### Version 0.20.7+
```diff
- pl.threadpool_size
+ pl.thread_pool_size
```

### Version 0.20.8+
```diff
- df.pivot(a, b, c)
+ df.pivot(values=a, index=b, columns=c)
```

### Version 0.20.11+
```diff
- pl.col('a').meta.write_json
+ pl.col('a').meta.serialize
```

### Version 0.20.14+
```diff
- df.group_by_dynamic('time', every='2d', by='symbol')
+ df.group_by_dynamic('time', every='2d', group_by='symbol')
- df.rolling('time', period='2d', by='symbol')
+ df.rolling('time', period='2d', group_by='symbol')
- df.upsample('time', every='2d', by='symbol')
+ df.upsample('time', every='2d', group_by='symbol')
```

### Version 0.20.17+
```diff
- pl.from_repr(tbl=data)
+ pl.from_repr(data=data)
```

### Version 0.20.24+
```diff
- pl.col('a').rolling_min('2d', by='time')
+ pl.col('a').rolling_min_by(window_size='2d', by='time')
- pl.col('a').rolling_max('2d', by='time')
+ pl.col('a').rolling_max_by(window_size='2d', by='time')
- pl.col('a').rolling_mean('2d', by='time')
+ pl.col('a').rolling_mean_by(window_size='2d', by='time')
- pl.col('a').rolling_std('2d', by='time')
+ pl.col('a').rolling_std_by(window_size='2d', by='time')
- pl.col('a').rolling_var('2d', by='time')
+ pl.col('a').rolling_var_by(window_size='2d', by='time')
- pl.col('a').rolling_prod('2d', by='time')
+ pl.col('a').rolling_prod_by(window_size='2d', by='time')
- pl.col('a').rolling_sum('2d', by='time')
+ pl.col('a').rolling_sum_by(window_size='2d', by='time')
```

### Version 0.20.29+
```diff
- df.join(df_right, how='outer')
+ df.join(df_right, how='full')
- df.join(df_right, how='outer_coalesce')
+ df.join(df_right, how='full', coalesce=True)
```

### Version 0.20.31+
```diff
- pl.read_csv(file, dtypes=schema)
+ pl.read_csv(file, schema=schema)
- pl.SQLContext(eager_execution=True)
+ pl.SQLContext(eager=True)
- pl.col('a').top_k(k=2, maintain_order=True)
+ pl.col('a').top_k(k=2)
```

## Notes

This work is derivative of [pyupgrade](https://github.com/asottile/pyupgrade) - many parts
have been lifted verbatim. As required, I've included pyupgrade's license.


========================================
SOCS - Simons Observatory Control System
========================================

| |pypi| |versions| |docker| |license|
| |tests| |pre-commit| |coverage| |docs|

Overview
--------

This repository, `SOCS`_, contains hardware control code for the
Simons Observatory.  This code operates within the framework provided
by `OCS`_.

.. _`OCS`: https://github.com/simonsobs/ocs/
.. _SOCS: https://github.com/simonsobs/socs/

Installation
------------

Install and update with pip::

    $ pip3 install -U socs

You may install optional dependencies by including one or more agent group
names on installation, for example::

    $ pip3 install -U socs[labjack,synacc]

For a complete list of agent groups see the `Installation Documentation`_.

If you would like to install all optional dependencies use the special varient
"all"::

    $ pip3 install -U socs[all]

**Note:** Not all optional dependencies can be installed this way. See the
`Installation Documentation`_ for more info on specific agent dependencies.

.. _`Installation Documentation`: https://socs.readthedocs.io/en/main/user/installation.html

Installing from Source
``````````````````````

If you are considering contributing to SOCS, or would like to use an unreleased
feature, you will want to install from source. To do so, clone this repository
and install using pip:

.. code-block:: bash

    git clone https://github.com/simonsobs/socs.git
    cd socs/
    pip3 install -r requirements.txt
    pip3 install .

In order for OCS tools to find these agents, you must add the full
path to the agents directory, e.g. ``/home/simons/code/socs/agents/``,
to your OCS site config file. Do so by adding the following under your
configured host if it does not already exist:

.. code-block:: yaml

  # List of additional paths to Agent plugin modules.
  'agent-paths': [
    '/path/to/socs/agents/',
  ],

See the `ocs docs`_ for more details.

.. _`ocs docs`: https://ocs.readthedocs.io/en/main/developer/site_config.html

Docker Images
-------------
Docker images for SOCS and each Agent are available on `Docker Hub`_. Official
releases will be tagged with their release version, i.e. ``v0.1.0``. These are
only built on release, and the ``latest`` tag will point to the latest of these
released tags. These should be considered stable.

Test images will be tagged with the latest released version tag, the number of
commits ahead of that release, the latest commit hash, i.e.
``v0.0.2-81-g9c10ba6-dev``. These get built on each commit to the ``main``
branch, and are useful for testing and development, but should be considered
unstable.

.. _Docker Hub: https://hub.docker.com/u/simonsobs

Documentation
-------------
The SOCS documentation can be built using Sphinx. There is a separate
``requirements.txt`` file in the ``docs/`` directory to install Sphinx and any
additional documentation dependencies::

  cd docs/
  pip3 install -r requirements.txt
  make html

You can then open ``docs/_build/html/index.html`` in your preferred web
browser. You can also find a copy hosted on `Read the Docs`_.

.. _Read the Docs: https://socs.readthedocs.io/en/latest/

Tests
-----
The tests for SOCS are run using pytest, and should be run from the
``tests/`` directory::

  $ cd tests/
  $ python3 -m pytest --cov

For more details see `tests/README.rst <tests_>`_.

.. _tests: https://github.com/simonsobs/socs/blob/main/tests/README.rst

Contributing
------------
For guidelines on how to contribute to OCS see `CONTRIBUTING.rst`_.

.. _CONTRIBUTING.rst: https://github.com/simonsobs/socs/blob/main/CONTRIBUTING.rst

License
--------
This project is licensed under the BSD 2-Clause License - see the
`LICENSE.txt`_ file for details.

.. _LICENSE.txt: https://github.com/simonsobs/socs/blob/main/LICENSE.txt

.. |coverage| image:: https://coveralls.io/repos/github/simonsobs/socs/badge.svg
    :target: https://coveralls.io/github/simonsobs/socs

.. |docker| image:: https://img.shields.io/badge/dockerhub-latest-blue
    :target: https://hub.docker.com/r/simonsobs/socs

.. |docs| image:: https://readthedocs.org/projects/socs/badge/?version=main
    :target: https://socs.readthedocs.io/en/main/?badge=main
    :alt: Documentation Status

.. |license| image:: https://img.shields.io/pypi/l/socs
    :target: LICENSE.txt
    :alt: PyPI - License

.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/simonsobs/socs/main.svg
   :target: https://results.pre-commit.ci/latest/github/simonsobs/socs/main
   :alt: pre-commit.ci status

.. |pypi| image:: https://img.shields.io/pypi/v/socs
   :target: https://pypi.org/project/socs/
   :alt: PyPI Package

.. |tests| image:: https://github.com/simonsobs/socs/actions/workflows/develop.yml/badge.svg?branch=main
    :target: https://github.com/simonsobs/socs/actions/workflows/develop.yml
    :alt: GitHub Workflow Status

.. |versions| image:: https://img.shields.io/pypi/pyversions/socs
    :alt: PyPI - Python Version


|license| |build| |coverage| |bench| |release| |pypi| |conda| |gitter|

.. image:: logos/logo.svg
   :width: 600 px
   :target: https://scikit.bio
   :alt: scikit-bio logo

*scikit-bio is an open-source, BSD-licensed Python 3 package providing data structures, algorithms and educational resources for bioinformatics.*

🌟 Announcing the revitalization of scikit-bio! 🌟
--------------------------------------------------

We are excited to announce the resurgence of **scikit-bio** (`<https://scikit.bio>`_)! With a re-assembled developer team and funding from the DOE, we're back with renewed vigor in 2024!

Our vision is to expand scikit-bio into a more robust and versatile library, catering to the ever-growing demands of multi-omic data analysis. This resurgence marks a new chapter in which we will focus on:

- Streamlining the analysis of diverse, massive omic data, emphasizing efficiency and versatility.
- Integrating advanced techniques for multi-omic integration to unravel the complex interplay between biological systems and environments.
- Implementing methods for modeling and annotating biological features utilizing community ecology and phylogenetics.

We invite the scientific community to join us in shaping the future of scikit-bio. Your expertise, feedback, and contributions will be the driving force behind this exciting phase.

Stay `tuned for updates <https://github.com/scikit-bio/scikit-bio/discussions/categories/announcements>`_, and let's innovate together for a deeper understanding of bio-complexities!

Get involved with scikit-bio
----------------------------

Your questions, ideas, and contributions matter!

Join our community on `GitHub Discussions <https://github.com/scikit-bio/scikit-bio/discussions>`_: This is your go-to place for asking questions, sharing insights, and participating in discussions about scikit-bio. Engage with both the developers and fellow users here.

Report issues and bugs: If you encounter specific problems when using scikit-bio, let us know directly through the `GitHub Issues <https://github.com/scikit-bio/scikit-bio/issues>`_ page. Your reports are vital for the continuous improvement of scikit-bio.

Wanna contribute? We enthusiastically welcome community contributors! Whether it's adding new features, improving code, or enhancing documentation, your contributions drive scikit-bio and open-source bioinformatics forward. Start your journey by reading the `Contributor's guidelines <https://scikit.bio/contribute.html>`_.

----

Visit the new scikit-bio website: https://scikit.bio to learn more about this project.

Releases
--------

Latest release: `0.6.1 <https://github.com/scikit-bio/scikit-bio/releases/tag/0.6.1>`_ (`documentation <https://scikit.bio/docs/0.6.1/index.html>`_, `changelog <https://github.com/scikit-bio/scikit-bio/blob/main/CHANGELOG.md#version-061>`_). Compatible with Python 3.8 and above.

Installation
------------

Install the latest release of scikit-bio using ``conda``::

    conda install -c conda-forge scikit-bio

Or using ``pip``::

    pip install scikit-bio

Verify the installation::

    python -m skbio.test

See further `instructions on installing <https://scikit.bio/install.html>`_ scikit-bio on various platforms.

Adoption
--------

Some of the projects that we know of that are using scikit-bio are:

- `QIIME 2 <https://qiime2.org/>`_, `Qiita <https://qiita.ucsd.edu/>`_, `Emperor <https://biocore.github.io/emperor/>`_, `tax2tree <https://github.com/biocore/tax2tree>`_, `ghost-tree <https://github.com/JTFouquier/ghost-tree>`_, `Platypus-Conquistador <https://github.com/biocore/Platypus-Conquistador>`_, `An Introduction to Applied Bioinformatics <https://readiab.org>`_.

License
-------

scikit-bio is available under the new BSD license. See `LICENSE.txt <LICENSE.txt>`_ for scikit-bio's license, and the `licenses directory <licenses>`_ for the licenses of third-party software that is (either partially or entirely) distributed with scikit-bio.

Team
----

Our core development team consists of three lead developers: **Dr. Qiyun Zhu** at Arizona State University (ASU) (@qiyunzhu), **Dr. James Morton** at Gutz Analytics (@mortonjt), and **Dr. Daniel McDonald** at the University of California San Diego (UCSD) (@wasade), one software engineer: **Matthew Aton** (@mataton) and one bioinformatician: **Dr. Lars Hunger** (@LarsHunger). **Dr. Rob Knight** at UCSD (@rob-knight) provides guidance on the development and research. **Dr. Greg Caporaso** (@gregcaporaso) at Northern Arizona University (NAU), the former leader of the scikit-bio project, serves as an advisor on the current project.

Credits
-------

We thank the many contributors to scikit-bio. A complete `list of contributors <graphs/contributors>`_ to the scikit-bio codebase is available at GitHub. This however may miss the larger community who contributed by testing the software and providing valuable comments, who we hold equal appreciation to.

Wanna contribute? We enthusiastically welcome community contributors! Whether it's adding new features, improving code, or enhancing documentation, your contributions drive scikit-bio and open-source bioinformatics forward. Start your journey by reading the `Contributor's guidelines <https://scikit.bio/contribute.html>`_.

Funding
-------

The development of scikit-bio is currently supported by the U.S. Department of Energy, Office of Science under award number `DE-SC0024320 <https://genomicscience.energy.gov/compbioawards2023/#Expanding>`_, awarded to Dr. Qiyun Zhu at ASU (lead PI), Dr. James Morton at Gutz Analytics, and Dr. Rob Knight at UCSD.

Citation
--------

If you use scikit-bio for any published research, please see our `Zenodo page <https://zenodo.org/record/8209901>`_ for how to cite.

Collaboration
-------------

For collaboration inquiries and other formal communications, please reach out to **Dr. Qiyun Zhu** at `qiyun.zhu@asu.edu`. We welcome academic and industrial partnerships to advance our mission.

Branding
--------

The logo of scikit-bio was created by `Alina Prassas <https://cargocollective.com/alinaprassas>`_. Vector and bitmap image files are available at the `logos <logos>`_ directory.

Pre-history
-----------

scikit-bio began from code derived from `PyCogent <https://github.com/pycogent/pycogent>`_ and `QIIME <https://github.com/biocore/qiime>`_, and the contributors and/or copyright holders have agreed to make the code they wrote for PyCogent and/or QIIME available under the BSD license. The contributors to PyCogent and/or QIIME modules that have been ported to scikit-bio are listed below:

- Rob Knight (@rob-knight), Gavin Huttley (@gavinhuttley), Daniel McDonald (@wasade), Micah Hamady, Antonio Gonzalez (@antgonza), Sandra Smit, Greg Caporaso (@gregcaporaso), Jai Ram Rideout (@jairideout), Cathy Lozupone (@clozupone), Mike Robeson (@mikerobeson), Marcin Cieslik, Peter Maxwell, Jeremy Widmann, Zongzhi Liu, Michael Dwan, Logan Knecht (@loganknecht), Andrew Cochran, Jose Carlos Clemente (@cleme), Damien Coy, Levi McCracken, Andrew Butterfield, Will Van Treuren (@wdwvt1), Justin Kuczynski (@justin212k), Jose Antonio Navas Molina (@josenavas), Matthew Wakefield (@genomematt) and Jens Reeder (@jensreeder).

.. |license| image:: https://img.shields.io/badge/License-BSD%203--Clause-blue.svg
   :alt: License
   :target: https://opensource.org/licenses/BSD-3-Clause
.. |build| image:: https://github.com/scikit-bio/scikit-bio/actions/workflows/ci.yml/badge.svg
   :alt: Build Status
   :target: https://github.com/scikit-bio/scikit-bio/actions/workflows/ci.yml
.. |coverage| image:: https://codecov.io/gh/scikit-bio/scikit-bio/graph/badge.svg?token=1qbzC6d2F5
   :alt: Coverage Status
   :target: https://codecov.io/gh/scikit-bio/scikit-bio
.. |bench| image:: https://img.shields.io/badge/benchmarked%20by-asv-green.svg
   :alt: ASV Benchmarks
   :target: https://s3-us-west-2.amazonaws.com/scikit-bio.org/benchmarks/main/index.html
.. |release| image:: https://img.shields.io/github/v/release/scikit-bio/scikit-bio.svg
   :alt: Release
   :target: https://github.com/scikit-bio/scikit-bio/releases
.. |pypi| image:: https://img.shields.io/pypi/dm/scikit-bio.svg?label=PyPI%20downloads
   :alt: PyPI Downloads
   :target: https://pypi.org/project/scikit-bio/
.. |conda| image:: https://img.shields.io/conda/dn/conda-forge/scikit-bio.svg?label=Conda%20downloads
   :alt: Conda Downloads
   :target: https://anaconda.org/conda-forge/scikit-bio
.. |gitter| image:: https://badges.gitter.im/Join%20Chat.svg
   :alt: Gitter
   :target: https://gitter.im/biocore/scikit-bio


[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![codecov](https://codecov.io/gh/mne-tools/mne-lsl/graph/badge.svg?token=Xoeh6T13qi)](https://codecov.io/gh/mne-tools/mne-lsl)
[![tests](https://github.com/mne-tools/mne-lsl/actions/workflows/pytest.yaml/badge.svg?branch=main)](https://github.com/mne-tools/mne-lsl/actions/workflows/pytest.yaml)
[![doc](https://github.com/mne-tools/mne-lsl/actions/workflows/doc.yaml/badge.svg?branch=main)](https://github.com/mne-tools/mne-lsl/actions/workflows/doc.yaml)
[![PyPI version](https://badge.fury.io/py/mne-lsl.svg)](https://badge.fury.io/py/mne-lsl)
[![Downloads](https://static.pepy.tech/badge/mne-lsl)](https://pepy.tech/project/mne-lsl)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/mne-lsl.svg)](https://anaconda.org/conda-forge/mne-lsl)
[![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/mne-lsl.svg)](https://anaconda.org/conda-forge/mne-lsl)
[![Conda Platforms](https://img.shields.io/conda/pn/conda-forge/mne-lsl.svg)](https://anaconda.org/conda-forge/mne-lsl)

**MNE-LSL** [(Documentation website)](https://mne.tools/mne-lsl)
provides a real-time brain signal streaming framework.
**MNE-LSL** contains an improved python-binding for the Lab Streaming Layer C++ library,
`mne_lsl.lsl`, replacing `pylsl`. This low-level binding is used in high-level objects
to interact with LSL streams.

Any signal acquisition system supported by native LSL or OpenVibe is also
supported by MNE-LSL. Since the data communication is based on TCP, signals can be
transmitted wirelessly. For more information about LSL, please visit the
[LSL github](https://github.com/sccn/labstreaminglayer).

# Install

MNE-LSL supports `python ≥ 3.9` and is available on
[PyPI](https://pypi.org/project/mne-lsl/) and on
[conda-forge](https://anaconda.org/conda-forge/mne-lsl).
Install instruction can be found on the
[documentation website](https://mne.tools/mne-lsl/stable/resources/install.html).

# Acknowledgment

**MNE-LSL** is based on **BSL** and **NeuroDecode**. The original version developed by
[**Kyuhwa Lee**](https://github.com/dbdq) was recognised at
[Microsoft Brain Signal Decoding competition](https://github.com/dbdq/microsoft_decoding)
with the First Prize Award (2016).
**MNE-LSL** is based on the refactor version, **BSL** by
[**Mathieu Scheltienne**](https://github.com/mscheltienne) and
[**Arnaud Desvachez**](https://github.com/dnastars) for the
[Fondation Campus Biotech Geneva (FCBG)](https://github.com/fcbg-hnp-meeg) and
development is still supported by the
[Fondation Campus Biotech Geneva (FCBG)](https://hnp.fcbg.ch/).

# Copyright and license

The code is released under the
[BSD 3-Clause License](https://opensource.org/license/bsd-3-clause/).


## Trader service

Trader is an autonomous service that performs **bets on existing prediction markets**. The service interacts with an [AI Mech](https://github.com/valory-xyz/mech) (a service that executes AI tasks), and its workflow is as follows:

1. Retrieve information on existing prediction markets (for example, markets created by a given address).
2. Select one of these markets for betting.
3. Send a request to an [AI Mech](https://github.com/valory-xyz/mech) to estimate the probability of the event referenced by the prediction market question, and what confidence the AI Mech has on that prediction.
    - The service will typically bet higher amounts for higher confidence predictions coming from the AI Mech.
    - These parameters are configurable.
4. If the response from the [AI Mech](https://github.com/valory-xyz/mech) meets certain criteria indicating profitability, the service will place a bet on that market. The betting amount can be adjusted based on the confidence level provided by the AI Mech.
5. In case the bet is deemed unprofitable, the market will be blacklisted for a configurable duration.
6. Repeat these steps continuously.

The Trader service is an [agent service](https://docs.autonolas.network/open-autonomy/get_started/what_is_an_agent_service/) (or autonomous service) based on the [Open Autonomy framework](https://docs.autonolas.network/open-autonomy/). Below we show you how to prepare your environment, how to prepare the agent keys, and how to configure and run the service.

## Prepare the environment

- System requirements:

  - Python `== 3.10`
  - [Poetry](https://python-poetry.org/docs/) `>=1.4.0`
  - [Docker Engine](https://docs.docker.com/engine/install/)
  - [Docker Compose](https://docs.docker.com/compose/install/)

- Clone this repository:

      git clone https://github.com/valory-xyz/trader.git

- Create a development environment:

      poetry install && poetry shell

- Configure the Open Autonomy framework:

      autonomy init --reset --author valory --remote --ipfs --ipfs-node "/dns/registry.autonolas.tech/tcp/443/https"

- Pull packages required to run the service:

      autonomy packages sync --update-packages

## Prepare the keys and the Safe

You need a **Gnosis keypair** and a **[Safe](https://safe.global/) address** to run the service.

First, prepare the `keys.json` file with the Gnosis keypair of your agent. (Replace the uppercase placeholders below):

    cat > keys.json << EOF
    [
    {
        "address": "YOUR_AGENT_ADDRESS",
        "private_key": "YOUR_AGENT_PRIVATE_KEY"
    }
    ]
    EOF

Next, prepare the [Safe](https://safe.global/). The trader agent runs as part of a **trader service**,
which is an [autonomous service](https://docs.autonolas.network/open-autonomy/get_started/what_is_an_agent_service/)
represented on-chain in the [Autonolas Protocol](https://docs.autonolas.network/protocol/) by a [Safe](https://safe.global/) multisig. Follow the next steps to obtain a **Safe address** corresponding to your agent address:

1. Visit https://registry.olas.network/gnosis/services/mint and connect to the Gnosis network. We recommend connecting using a wallet with a Gnosis EOA account that you own.
2. Fill in the following fields:
    - *"Owner address"*: a Gnosis address for which you will be able to sign later using a supported wallet. If you want to use the address you are connected to, click on *"Prefill Address"*.
    - Click on *"Generate Hash & File"* and enter the value corresponding to the `service/valory/trader/0.1.0` key in [`packages.json`](https://github.com/valory-xyz/trader/blob/main/packages/packages.json)
    - *"Canonical agent Ids"*: enter the number `12`
    - *"No. of slots to canonical agent Ids"*: enter the number `1`
    - *"Cost of agent instance bond (wei)"*: enter the number `10000000000000000`
    - *"Threshold"*: enter the number `1`
3. Press the *"Submit"* button. Your wallet will ask you to approve the transaction. Once the transaction is settled, you should see a message indicating that the service NFT has been minted successfully. You should also see that the service is in _Pre-Registration_ state.
4. Next, you can navigate to https://registry.olas.network/services#my-services, select your service and go through the steps:
    1. Activate registration
    2. Register agents: **here, you must use your agent address**.
    3. This is the last step. A transaction for the Safe deployment is already prepared and needs to be executed.
5. After completing the process you should see that your service is **Deployed**, and you will be able to retrieve your **Safe contract address** as shown in the image below:

<img src="/img/safe_address_screenshot.png" alt="Safe address field" width="500"/>

**You need to provide some funds (XDAI) both to your agent address and to the Safe address in order to place bets on prediction markets.**

## Configure the service

Set up the following environment variables, which will modify the performance of the trading agent. **Please read their description below**. We provide some defaults, but feel free to experiment with different values. Note that you need to provide `YOUR_AGENT_ADDRESS` and `YOUR_SAFE_ADDRESS` from the section above.

```bash
export RPC_0=INSERT_YOUR_RPC
export CHAIN_ID=100

export ALL_PARTICIPANTS='["YOUR_AGENT_ADDRESS"]'
export SAFE_CONTRACT_ADDRESS="YOUR_SAFE_ADDRESS"
export OMEN_CREATORS='["0x89c5cc945dd550BcFfb72Fe42BfF002429F46Fec"]'

export BET_AMOUNT_PER_THRESHOLD_000=0
export BET_AMOUNT_PER_THRESHOLD_010=0
export BET_AMOUNT_PER_THRESHOLD_020=0
export BET_AMOUNT_PER_THRESHOLD_030=0
export BET_AMOUNT_PER_THRESHOLD_040=0
export BET_AMOUNT_PER_THRESHOLD_050=0
export BET_AMOUNT_PER_THRESHOLD_060=30000000000000000
export BET_AMOUNT_PER_THRESHOLD_070=40000000000000000
export BET_AMOUNT_PER_THRESHOLD_080=60000000000000000
export BET_AMOUNT_PER_THRESHOLD_090=80000000000000000
export BET_AMOUNT_PER_THRESHOLD_100=100000000000000000

export BET_THRESHOLD=5000000000000000

export PROMPT_TEMPLATE='With the given question "@{question}" and the `yes` option represented by `@{yes}` and the `no` option represented by `@{no}`, what are the respective probabilities of `p_yes` and `p_no` occurring?'
```

These are the description of the variables used by the Trader service:

- `RPC_0`: RPC endpoint for the agent (you can get an RPC endpoint, e.g. [here](https://getblock.io/)).
- `CHAIN_ID`: identifier of the chain on which the service is running (Gnosis=100).
- `ALL_PARTICIPANTS`: list of all the agent addresses participating in the service. In this example we only are using a single agent.
- `SAFE_CONTRACT_ADDRESS`: address of the agents multisig wallet created [in the previous section](#prepare-the-keys-and-the-safe).
- `OMEN_CREATORS`: addresses of the market creator(s) that the service will track
  for placing bets on Omen. The address `0x89c5cc945dd550BcFfb72Fe42BfF002429F46Fec` corresponds to the Market creator agent for the Hackathon.
- `BET_AMOUNT_PER_THRESHOLD_X`: amount (wei) to bet when the prediction returned by the AI Mech surpasses a threshold of `X`% confidence for a given prediction market. In the values provided above the amounts vary between 0.03 xDAI (60% confidence) and 0.1 xDAI (100% confidence).
- `BET_THRESHOLD`: threshold (wei) for placing a bet. A bet will only be placed if `potential_net_profit - BET_THRESHOLD >= 0`. [See below](#some-notes-on-the-service).
- `PROMPT_TEMPLATE`: prompt to be used with the prediction AI Mech. Please keep it as a single line including the placeholders `@{question}`, `@{yes}` and `@{no}`.

## Run the service
Once you have configured (exported) the environment variables, you are in position to run the service.

- Fetch the service:

    ```bash
    autonomy fetch --local --service valory/trader && cd trader
    ```

- Build the Docker image:

    ```bash
    autonomy build-image
    ```

- Copy your `keys.json` file prepared [in the previous section](#prepare-the-keys-and-the-safe) in the same directory:

    ```bash
    cp path/to/keys.json .
    ```

- Build the deployment with a single agent and run:

    ```bash
    autonomy deploy build --n 1 -ltm
    autonomy deploy run --build-dir abci_build/
    ```

## Some notes on the service

Please take into consideration the following:

- If the service does not have enough funds for placing a bet, you will see an `Event.INSUFICIENT_FUNDS` in the service logs.
- If the service determines that a bet is not profitable
 (i.e., `potential_net_profit - BET_THRESHOLD < 0`), you will see an `Event.UNPROFITABLE` in the service logs,
 and the service will transition into the blacklisting round.
 This round blacklists a bet for a predetermined amount of time.
 This can be adjusted by using the `BLACKLISTING_DURATION` environment variable.
- For simplicity,
 the current implementation considers `potential_net_profit = num_shares - net_bet_amount - mech_price - BET_THRESHOLD`,
 although this calculation might be refined.
 The `net_bet_amount` is the bet amount minus the FPMM's fees.
- When assigning `BET_THRESHOLD` take into consideration that fees (at the time of writing this guide) are in the range of 0.02 xDAI. See, for example, [here](https://api.thegraph.com/subgraphs/name/protofire/omen-xdai/graphql?query=%7B%0A++fixedProductMarketMakers%28%0A++++where%3A+%7B%0A++++++creator_in%3A+%5B%220x89c5cc945dd550BcFfb72Fe42BfF002429F46Fec%22%5D%2C%0A++++++outcomeSlotCount%3A+2%2C%0A++++++isPendingArbitration%3A+false%0A++++%7D%2C%0A++++orderBy%3A+creationTimestamp%0A++++orderDirection%3A+desc%0A++%29%7B%0A++++fee%0A++%7D%0A%7D). We urge you to keep an eye on these fees, as they might vary.

## For advanced users

The trader service can be run as a multi-agent system. If you want to explore this option,
you need, for the case of 4 agents:

  - One `keys.json` file containing 4 addresses and keys.
  - Register these 4 keys in your service Safe as explained [in this section](#prepare-the-keys-and-the-safe).
  - Prepare extra environment variables need to be defined, including
    ```bash
    export RPC_0=INSERT_YOUR_RPC
    export RPC_1=INSERT_YOUR_RPC
    export RPC_2=INSERT_YOUR_RPC
    export RPC_3=INSERT_YOUR_RPC
    export ALL_PARTICIPANTS='["AGENT_ADDRESS_0,AGENT_ADDRESS_1,AGENT_ADDRESS_2,AGENT_ADDRESS_3"]'
    ```

    where   `RPC_i` is the RPC endpoint for agent `AGENT_ADDRESS_i`.

You can also explore the [`service.yaml`](https://github.com/valory-xyz/trader/blob/main/packages/valory/services/trader/service.yaml) file, which contains all the possible configuration variables for the service.

Finally, if you are experienced with the [Open Autonomy](https://docs.autonolas.network/) framework, you can also modify the internal business logic of the service yourself.

## Included strategies

| Strategies |
|---|
| packages/jhehemann/customs/kelly_criterion |
| packages/valory/customs/bet_amount_per_threshold |
| packages/valory/customs/mike_strat |
| packages/w1kke/customs/always_blue |


# pyatmo

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
[![GitHub Actions](https://github.com/jabesq/pyatmo/workflows/Python%20package/badge.svg)](https://github.com/jabesq/pyatmo/actions?workflow=Python+package)
[![PyPi](https://img.shields.io/pypi/v/pyatmo.svg)](https://pypi.python.org/pypi/pyatmo)
[![license](https://img.shields.io/pypi/l/pyatmo.svg)](https://github.com/jabesq/pyatmo/blob/master/LICENSE.txt)

> **Warning:**
> Due to personal reasons, I am currently unable to dedicate sufficient time to effectively manage this repository. Consequently, no attention will be given to existing or forthcoming issues until further notice. **However**, I want to assure you that I will continue to merge any pull requests that are submitted, provided they successfully pass the continuous integration tests and do not exhibit any glaring issues.
>
> I apologize for any inconvenience this may cause, and I sincerely hope to have the capacity to allocate more time to this repository in the near future. Your understanding is greatly appreciated.

---

Simple API to access Netatmo devices and data like weather station or camera data from Python 3.
For more detailed information see [dev.netatmo.com](http://dev.netatmo.com)

This project has no relation with the Netatmo company.

## Install

To install pyatmo simply run:

    pip install pyatmo

Depending on your permissions you might be required to use sudo.
Once installed you can simply add `pyatmo` to your Python 3 scripts by including:

    import pyatmo

## Note

The module requires a valid user account and a registered application.
Be aware that the module may stop working if Netatmo decides to change their API.

## Development

Clone the repo and install dependencies:

    git clone
    cd pyatmo
    pipenv install --dev

To add the pre-commit hook to your environment run:

    pip install pre-commit
    pre-commit install

## Testing

To run the full suite simply run the following command from within the virtual environment:

    pytest

or

    python -m pytest tests/

To generate code coverage xml (e.g. for use in VSCode) run

    python -m pytest --cov-report xml:cov.xml --cov pyatmo --cov-append tests/

Another way to run the tests is by using `tox`. This runs the tests against the installed package and multiple versions of python.

    tox

or by specifying a python version

    tox -e py310


# HyP3 documentation

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4646138.svg)](https://doi.org/10.5281/zenodo.4646138)

HyP3 documentation is built using [MkDocs](https://www.mkdocs.org/) and the
[ASF Theme](https://github.com/ASFHyP3/mkdocs-asf-theme).

## How to

### Setting up a development environment

In order to automatically document some of our APIs, we use a `conda` environment
with our APIs installed. You can get Miniconda (recommended) here:

<https://docs.conda.io/en/latest/miniconda.html>

Once conda is installed, from the repository root, you can create and activate a
conda environment with all the necessary dependencies

```
conda env create -f environment.yml
conda activate hyp3-docs
```

Later, you can update the environment's dependencies with

```
conda env update -f environment.yml
```

### Build and view the documentation site

With the `hyp3-docs` conda environment activated, run

```
mkdocs serve
```

to generate the documentation. This will allow you to view it at <http://127.0.0.1:8000/>.
MkDocs will automatically watch for new/changed files in this directory and
rebuild the website so you can see your changes live (just refresh the webpage!).

*Note: `mkdocs serve` captures your terminal; use `crtl+c` to exit. It is recommended you
use a second/dedicated terminal so you can keep this command running.*

### Deploy

This documentation site is deployed as a GitHub Organization website with a CNAME
so that it's viewable at <https://hyp3-docs.asf.alaska.edu/>. The website is served
out of the special <https://github.com/ASFHyP3/ASFHyP3.github.io> repository. Deployment
is handled automatically with the `.github/workflows/deploy_to_github_io.yml` GitHub
Action for any merge to `main`.

There is also a test site deployed to <https://hyp3-docs.asf.alaska.edu/hyp3-docs>, which
tracks the `develop` branch of this repo and is served out of the `gh-pages` branch
of this repo.

## Markdown formatting

The way MkDocs and GitHub parse the markdown documents are slightly different. Some compatibility tips:

* Raw links should be wrapped in angle brackets: `<https://example.com>`
* MkDocs is pickier about whitespace between types (e.g., headers, paragraphs, lists) and seems to
expect indents to be 4 spaces. So to get a representation like:

    - A list item

         ##### A sub list heading
        - A sub-list item

    in MkDocs, you'll want to write it like:

    ### Good
    ```
    - A list item

        ##### A sub list heading
        - A sub-list item
    ```

    ### Bad
    ```
    - A list item
      ##### A sub list heading
      - A sub-list item
    ```

    ```
    - A list item
        ##### A sub list heading
        - A sub-list item
    ```

    ```
    - A list item

      ##### A sub list heading
      - A sub-list item
    ```


[![All Contributors](https://img.shields.io/badge/all_contributors-16-orange.svg?style=flat-square)](#contributors)

> **WARNING**
> The 2023.9.1 update changes `site.static_path` to `site.static_paths` any custom output paths will need to be updated to reflect this change. `'output'` is still the default value for `site.static_paths` and will be used if no value is provided.

# Render Engine

[![PyTest](https://github.com/kjaymiller/render_engine/actions/workflows/test.yml/badge.svg)](https://github.com/kjaymiller/render_engine/actions/workflows/test.yml)
[![Discord](https://img.shields.io/discord/1174377880118104156?label=Discord&color=purple)](https://discord.gg/2xMQ4j4d8m)

## Learn More

- [Check out the Documentation](https://render-engine.readthedocs.io/en/latest/)
- [Contributors and Builders, Check out the Wiki](https://github.com/render-engine/render-engine/wiki)
- [Join the community!](https://discord.gg/2xMQ4j4d8m)

## What is RenderEngine

## The _3 layer_ Architecture

- **[Page](.github/render_engine/page.html)** - A single webpage item built from content, a template, raw data, or a combination of those things.
- **[Collection](.github/render_engine/collection.html)** - A group of webpages built from the same template, organized in a single directory
- **[Site](.github/render_engine/site.html)** - The container that helps to render all Pages and Collections with uniform settings and variables

## Installing Render Engine

To use the render engine, you must have Python 3.10 installed. You can download Python from [python.org](https://python.org).

- Linux/MacOS: [python.org](https://python.org)
- Windows: [Microsoft Store](https://apps.microsoft.com/store/detail/python-311/9NRWMJP3717K)

Render Engine is available in PyPI and can be installed using pip:

```bash
pip install render-engine
```

## Getting Started

Check out the [Getting Started](https://render-engine.readthedocs.io/en/latest/page/) Section in the [Documentation](https://render-engine.readthedocs.io)

## Contributors

Akriti Sengar
🔧

Anthony Shaw
📖

Emmanuel Anebi
🔌

Giovanni Martins
🚇

Jeff Triplett
📖 🧑🏫

John Aziz
📖 🚧 📓

Jon Banafato
💻

Jorge L. Gimeno
📖

Laura Langdon
📖

Mahhheshh
📖 💻 🚧

Mario Munoz
💻

Mohit Kambli
🚇

Pamela Fox
📖

Viktorius Suwandi
💻

VinayakG311
🔌

izaan17
🔌

### Sponsors at the $20/month and Higher Level

- [Brian Douglas](https://github.com/bdougie)

Thank you to them and all of those who continue to support this project!


# DeFyes

DeFyes is a Python library designed to track DeFi portfolios across multiple protocols.
This library serves as a versatile toolkit to retrieve DeFi positions' underlying
assets and financial metrics.

The project is currently undergoing active development, so we expect some backward incompatibility changes until we stabilize the API. We encourage you to stay engaged with the project’s updates and contribute to its evolution.

Some protocols supported: Aave v2 and v3, Lido, Compound v2 and v3, Balancer, Curve, Maker and more.
While cross-chain functionalities are supported, currently most integrations are focused on Ethereum
and Gnosis Chain.

## Install

When using `defyes` as a library, you have two ways to install it:
```sh
pip install . 'defyes[all]'
```
or, if you experiment dependencies conflicts, you can take control over some dependencies version by removing `[all]`
but specifying each missing dependency.
```sh
pip install . 'defyes'
pip install "karpatkit @ git+https://github.com/karpatkey/karpatkit.git@another_specific_version"
```
where `another_specific_version` should be a git reference (tag, hash, branch).

Have a look at `all` in the [pyproject.toml](pyproject.toml) file, `[project.optional-dependencies]` section.

> Take into account that installing just `defyes` without `defyes[all]` is an incompleted instalation.

## Configuration

- First you will have to copy, rename and modify the [config.json](https://github.com/karpatkey/defyes/blob/main/example_config.json)

- You should provide `RPC` endpoints and `EXPLORER API KEYS`

- You should set the env `CONFIG_PATH` env with the config.json's absolute path.

## Cache

To reduce the number of calls to RPC endpoints, and thus significantly speed up the functions, defyes implements a cache where the result of some web3 calls are stored.
If the same web3 function is called, with the very same arguments, its result will be retrieved from the cache.

By default the cache is enabled and caches all calls to web3 'eth_call' when a block other than 'latest' is specified.
In practice this means that the following example calls will be cached:

    ```python
    chef_contract.functions.totalAllocPoint().call(block_identifier=1560000)

    chef_contract.functions.poolLength().call(block_identifier=block)

    token_contract.functions.decimals().call(block_identifier=block)
    ```

The following calls will not be cached:

    ```python
    chef_contract.functions.poolLength().call(block_identifier='latest')

    token_contract.functions.decimals().call()

    web3.eth.get_balance()

    contract.functions.updateValue(43).transact()

    ```

This functionality of the cache is very useful as long as the same block can be used, even if only for a while, since successive calls will bring the responses from the cache. This is very useful for example in tests or when developing.

There are some calls that are expected to always return the same value from the blockchain.
For example, `token.contract.decimals.call(block_identifier=block)`, will return the same value if the block is equal or greater than the block the contract was deployed.
For this type of calls, which return something constant in time, the automatic cache is  not useful because if a block is specified, the value will only be used for that block. And if 'latest' is specified, it is automatically excluded from the cache.
For these cases the const_call() helper function is used.

### `const_call()` helper

The `const_call()` function forces the caching of a `.call()` without defining a block. Thus subsequent calls to `const_call()` will return the same result even if the block is different from the block in which the result was cached.
Warning: if a call is made after caching using a block prior to the contract creation, `const_call()` will reuse the locally stored value instead of returning an error.

Example usage:

    ```
    const_call(token_contract.functions.decimals())

    const_call(pool_contract.functions.token0())
    ```

### Cache config

By default the cache is stored in a non persistent directory `/tmp/kkit/`.
To change the directory use the environment variable `KKIT_CACHE_DIR=/path/to/dir`.

To disable the cache define the `KKIT_CACHE_DISABLE` environment variable.

To wipe the cache use the env var `KKIT_CACHE_CLEAR` or call `karpatkit.cache.clear()`.

## Running the test

```
pip install -r requirements-dev.txt

pytest -vs tests/

```

## Building and distributing

```
pip install --upgrade build

python -m build

```

## Contributing

PRs are welcome!

Found a Bug ? Create an Issue.

## Development

If it's possible, just use `make` witch gives you many shortcuts.  Otherwise try to mimic what it does by seeing
[Makefile](Makefile).

The easy way is to call `make`, which by default build the docker image with all the dependencies and install the git
pre-commit hook to encorage you to commit well formatted code.

### Build the docker image

Run `make build` to build the docker image used to following development workflows.

The default image name used is `defyes`. You could override it specify `make build image=...`.

The default `CONFIG_FILE` is defined as `config.json` from the current working directory.

Every time a command is executed using this image, all the repository directory is mounted at `/repo` which is also
the default current working directory inside the container. `.tmp` is mounted to `/tmp`, where the Internet
requests/responces cache lives by default. `/repo/.home` is defined as the `HOME` to keep bash and ipython
configurations and history across shell sessions.

### Install the pre-commit hook

To install or re-install the pre-commit hook, call `make install-pre-commit`. It'll copy the [.pre-commit](.pre-commit)
file into the `.git` directory, to check for [linting](#linting) every time you try to create a new git
commit.

### Linting

Linting is the process to check code format accomplishment.

From now on you shoud ensure linting before commiting code. To help you doing that, `make lint` just check the code
formatting.  It includes executing `black`, `isort` and `flake8` just in checking mode. If you want to apply `black`
and `isort` changes automatically to your working-copy code, use `make pretty`.

`flake8` just check the code, but it doesn't apply chages automatically. You shoud decide how to make the changes
yourself base on the `flake8` output. Anyway, many code formatting rules `flake8` checks are sanitized automatically
by `black`.

> To change the default path where the linter checks, you could specify `make lint path=...`.

### Testing

The short way is just calling `make test`, which runs all tests.

if you want to specify a non default `config.json`, just overrider the env like
```shell
make test CONFIG_FILE=...
```

To have more control see [below](#a-shell-inside-the-container).

### IPython

Run `make ipython` straight away or `make shell` and then `ipython`.

### A shell inside the container

If you prefer to have more control over linting, testing, or debugging, execute `make shell`.

`make shell` starts `bash` inside the container. Then, you can directly run `pytest`, `black`, `ipython`, etc.

`bash`, as well as `ipython`, keeps its command history ;)

`make shell` executes `bash` with the same user you called it (your own user). As the repo volume is mounted in
read-write mode inside the container, it's useful to have new files or modified files with the same user you own
instead of the root user. But, if you prefer to run it with the root user, call `make rootshell` instead.

> `make` is always intended to be used outside the container.

Some examples of calling `pytest` inside the container could be:

```shell
pytest -v
```
```shell
pytest -vvs tests/test_azuro.py
```
```shell
pytest tests/test_azuro.py --pdb -k deposit
```

> The `.home` directory is git ignored to prevent committing command history among other things, except for the
> `.home/.bashrc` which you can edit and commit changes on it to define new bash command aliases or change the prompt.


# DeepLink.Framework

Deeplink.framework 是 DeepLink 推出的介于 AI 训练框架和硬件语言之间的训练系统，同时支持 Eager 模式和 Graph 模式两种链路。Framework 仓库主要由 DIPU (Device Independent Process Unit) 和 DICP (Device Independent Compile Protocol) 两部分组成，framework 中对 Eager 模式的支持，主要由 DIPU 完成，framework 中对 Graph 模式的支持主要由 DICP 完成。

## 仓库结构

### DIPU

DIPU (Device Independent Process Unit) 是由一组抽象设备 runtime 接口，一组框架能力相关的运行时基类/接口，一个针对 DIOPI 标准算子的适配层共同组成的拓展包。用来在训练框架 PyTorch 上接入 DIOPI 算子库，实现 Eager 模式的推理和训练。其能够在编译时，决定抽象设备被影射的方式；并使用统一的运行时，减少在多硬件上适配训练框架的成本。DIPU 即可以基于统一的设备运行时来屏蔽厂商的实际设备；也可以基于统一的框架相关的运行时基类，由厂商自行实现特有的运行时逻辑。

### DICP

标准编译协议 (Device-Independent Compile Protocol, DICP) 定义了统一的计算描述（中间表示），通过计算图获取深度学习模型中的计算任务表达为上述中间表示，然后通过计算图优化技术自动生成人工智能芯片设备代码，从而提高研发效率和计算的执行性能。中间表示是介于源语言和目标语言之间的程序表示，能够极大程度地提高编译流程的可拓展性，同时也能降低优化流程对前端和后端的破坏。多层次中间表示包含从应用到芯片端的多种表示层次，不同层次旨在解决不同尺度的问题。

## 相关索引

* [DIPU 介绍](./dipu/README.md)
* [DIPU 文档](https://deeplink.readthedocs.io/zh-cn/latest/doc/DIPU/Introduction.html)
* [DICP 介绍](./dicp/readme.md)
* [DICP 文档](https://deeplink.readthedocs.io/zh-cn/latest/doc/DICP/introduction.html)


# Optax

![CI status](https://github.com/google-deepmind/optax/actions/workflows/tests.yml/badge.svg?branch=main)
[![Documentation Status](https://readthedocs.org/projects/optax/badge/?version=latest)](http://optax.readthedocs.io)
![pypi](https://img.shields.io/pypi/v/optax)

## Introduction

Optax is a gradient processing and optimization library for JAX.

Optax is designed to facilitate research by providing building blocks
that can be easily recombined in custom ways.

Our goals are to

*   Provide simple, well-tested, efficient implementations of core components.
*   Improve research productivity by enabling to easily combine low-level
    ingredients into custom optimisers (or other gradient processing components).
*   Accelerate adoption of new ideas by making it easy for anyone to contribute.

We favour focusing on small composable building blocks that can be effectively
combined into custom solutions. Others may build upon these basic components
in more complicated abstractions. Whenever reasonable, implementations prioritise
readability and structuring code to match standard equations, over code reuse.

An initial prototype of this library was made available in JAX's experimental
folder as `jax.experimental.optix`. Given the wide adoption across DeepMind
of `optix`, and after a few iterations on the API, `optix` was eventually moved
out of `experimental` as a standalone open-source library, and renamed `optax`.

Documentation on Optax can be found at [optax.readthedocs.io](https://optax.readthedocs.io/).

## Installation

You can install the latest released version of Optax from PyPI via:

```sh
pip install optax
```

or you can install the latest development version from GitHub:

```sh
pip install git+https://github.com/google-deepmind/optax.git
```

## Quickstart

Optax contains implementations of [many popular optimizers](https://optax.readthedocs.io/en/latest/api/optimizers.html) and
[loss functions](https://optax.readthedocs.io/en/latest/api/losses.html).
For example, the following code snippet uses the Adam optimizer from `optax.adam`
and the mean squared error from `optax.l2_loss`. We initialize the optimizer
state using the `init` function and `params` of the model.

```python
optimizer = optax.adam(learning_rate)
# Obtain the `opt_state` that contains statistics for the optimizer.
params = {'w': jnp.ones((num_weights,))}
opt_state = optimizer.init(params)
```

To write the update loop we need a loss function that can be differentiated by
Jax (with `jax.grad` in this
example) to obtain the gradients.

```python
compute_loss = lambda params, x, y: optax.l2_loss(params['w'].dot(x), y)
grads = jax.grad(compute_loss)(params, xs, ys)
```

The gradients are then converted via `optimizer.update` to obtain the updates
that should be applied to the current parameters to obtain the new ones.
`optax.apply_updates` is a convenience utility to do this.

```python
updates, opt_state = optimizer.update(grads, opt_state)
params = optax.apply_updates(params, updates)
```

You can continue the quick start in [the Optax 🚀 Getting started notebook.](https://github.com/google-deepmind/optax/blob/main/docs/getting_started.ipynb)

## Development

We welcome new contributors.

### Source code

You can check the latest sources with the following command.

```sh
git clone https://github.com/google-deepmind/optax.git
```
### Testing

To run the tests, please execute the following script.

```sh
sh ./test.sh
```

### Documentation

To build the documentation, first ensure that all the dependencies are installed.
```sh
pip install -e ".[docs]"
```
Then, execute the following.
```sh
cd docs/
make html
```

## Benchmarks
If you feel lost in the crowd of available optimizers for deep learning, there
exist some extensive benchmarks:

[Benchmarking Neural Network Training Algorithms, Dahl G. et al, 2023](https://arxiv.org/pdf/2306.07179),

[Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers, Schmidt R. et al, 2021](https://proceedings.mlr.press/v139/schmidt21a).

If you are interested in developing your own benchmark for some tasks,
consider the following framework

[Benchopt: Reproducible, efficient and collaborative optimization benchmarks, Moreau T. et al, 2022](https://arxiv.org/abs/2206.13424).

Finally, if you are searching for some recommendations on tuning optimizers,
consider taking a look at

[Deep Learning Tuning Playbook, Godbole V. et al, 2023](https://github.com/google-research/tuning_playbook).

## Citing Optax

This repository is part of the DeepMind JAX Ecosystem, to cite Optax
please use the citation:

```bibtex
@software{deepmind2020jax,
  title = {The {D}eep{M}ind {JAX} {E}cosystem},
  author = {DeepMind and Babuschkin, Igor and Baumli, Kate and Bell, Alison and Bhupatiraju, Surya and Bruce, Jake and Buchlovsky, Peter and Budden, David and Cai, Trevor and Clark, Aidan and Danihelka, Ivo and Dedieu, Antoine and Fantacci, Claudio and Godwin, Jonathan and Jones, Chris and Hemsley, Ross and Hennigan, Tom and Hessel, Matteo and Hou, Shaobo and Kapturowski, Steven and Keck, Thomas and Kemaev, Iurii and King, Michael and Kunesch, Markus and Martens, Lena and Merzic, Hamza and Mikulik, Vladimir and Norman, Tamara and Papamakarios, George and Quan, John and Ring, Roman and Ruiz, Francisco and Sanchez, Alvaro and Sartran, Laurent and Schneider, Rosalia and Sezener, Eren and Spencer, Stephen and Srinivasan, Srivatsan and Stanojevi\'{c}, Milo\v{s} and Stokowiec, Wojciech and Wang, Luyu and Zhou, Guangyao and Viola, Fabio},
  url = {http://github.com/google-deepmind},
  year = {2020},
}
```


  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://github.com/reagento/adaptix/blob/v3.0.0b6/docs/logo/adaptix-with-title-dark.png?raw=true">
    <source media="(prefers-color-scheme: light)" srcset="https://github.com/reagento/adaptix/blob/v3.0.0b6/docs/logo/adaptix-with-title-light.png?raw=true">

  </picture>

  [![PyPI version](https://img.shields.io/pypi/v/adaptix.svg?color=blue)](https://pypi.org/project/adaptix/)
  [![downloads](https://img.shields.io/pypi/dm/adaptix.svg)](https://pypistats.org/packages/adaptix)
  [![versions](https://img.shields.io/pypi/pyversions/adaptix.svg)](https://github.com/reagento/adaptix)
  [![license](https://img.shields.io/github/license/reagento/dataclass_factory.svg)](https://github.com/reagento/adaptix/blob/master/LICENSE)

An extremely flexible and configurable data model conversion library.

> [!IMPORTANT]
> Adaptix is ready for production!
> The beta version only means there may be some backward incompatible changes, so you need to pin a specific version.

📚 [Documentation](https://adaptix.readthedocs.io/)

## TL;DR

Install
```bash
pip install adaptix==3.0.0b7
```

Use for model loading and dumping.

```python
from dataclasses import dataclass

from adaptix import Retort

@dataclass
class Book:
    title: str
    price: int

data = {
    "title": "Fahrenheit 451",
    "price": 100,
}

# Retort is meant to be global constant or just one-time created
retort = Retort()

book = retort.load(data, Book)
assert book == Book(title="Fahrenheit 451", price=100)
assert retort.dump(book) == data
```

Use for converting one model to another.

```python
from dataclasses import dataclass

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

from adaptix.conversion import get_converter

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = 'books'

    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str]
    price: Mapped[int]

@dataclass
class BookDTO:
    id: int
    title: str
    price: int

convert_book_to_dto = get_converter(Book, BookDTO)

assert (
    convert_book_to_dto(Book(id=183, title="Fahrenheit 451", price=100))
    ==
    BookDTO(id=183, title="Fahrenheit 451", price=100)
)
```

## Use cases

* Validation and transformation of received data for your API.
* Conversion between data models and DTOs.
* Config loading/dumping via codec that produces/takes dict.
* Storing JSON in a database and representing it as a model inside the application code.
* Creating API clients that convert a model to JSON sending to the server.
* Persisting entities at cache storage.
* Implementing fast and primitive ORM.

## Advantages

* Sane defaults for JSON processing, no configuration is needed for simple cases.
* Separated model definition and rules of conversion
  that allow preserving [SRP](https://blog.cleancoder.com/uncle-bob/2014/05/08/SingleReponsibilityPrinciple.html)
  and have different representations for one model.
* Speed. It is one of the fastest data parsing and serialization libraries.
* There is no forced model representation, adaptix can adjust to your needs.
* Support [dozens](https://adaptix.readthedocs.io/en/latest/loading-and-dumping/specific-types-behavior.html) of types,
  including different model kinds:
  ``@dataclass``, ``TypedDict``, ``NamedTuple``,
  [``attrs``](https://www.attrs.org/en/stable/), [``sqlalchemy``](https://docs.sqlalchemy.org/en/20/) and [``pydantic``](https://docs.pydantic.dev/latest/).
* Working with self-referenced data types (such as linked lists or trees).
* Saving [path](https://adaptix.readthedocs.io/en/latest/loading-and-dumping/tutorial.html#error-handling)
  where an exception is raised (including unexpected errors).
* Machine-readable [errors](https://adaptix.readthedocs.io/en/latest/loading-and-dumping/tutorial.html#error-handling)
  that could be dumped.
* Support for user-defined generic models.
* Automatic name style conversion (e.g. `snake_case` to `camelCase`).
* [Predicate system](https://adaptix.readthedocs.io/en/latest/loading-and-dumping/tutorial.html#predicate-system)
  that allows to concisely and precisely override some behavior.
* Disabling additional checks to speed up data loading from trusted sources.
* No auto casting by default. The loader does not try to guess value from plenty of input formats.


# NetPyNE (Python package)
## Description
NetPyNE is a Python package to facilitate the development, simulation, parallelization, analysis, and optimization of biological neuronal networks using the NEURON simulator.

For more details, installation instructions, documentation, tutorials, forums, videos and more, please visit: www.netpyne.org

This package is developed and maintained by Dura-Bernal lab (http://dura-bernal.org) and the Neurosim lab (www.neurosimlab.org).

[![Build](https://github.com/Neurosim-lab/netpyne/actions/workflows/tests.yml/badge.svg)](https://github.com/Neurosim-lab/netpyne/actions/workflows/tests.yml)

## Acknowledgements
This work was funded by grants from the NIH, NSF, NYS SCIRB, UK Welcome Trust and Australian Research Council. We are thankful to all the contributors that have collaborated in the development of this open source tool via GitHub.

This project was supported by:
- HBP Brain Simulation Platform funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 785907 (Human Brain Project SGA2).
- EBRAINS research infrastructure, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 945539 (Human Brain Project SGA3).

This project has received a Voucher "Integration of NetPyNE into EBRAINS Platform (EBRnetpyne)" from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement:
- No. 785907 (Human Brain Project SGA2).
- No. 945539 (Human Brain Project SGA3).


# llm-random
We are LLM-Random, a research group at [IDEAS NCBR](https://ideas-ncbr.pl/en/) (Warsaw, Poland). We develop this repo and use it to conduct research. To learn more about us and our research, check out our blog, [llm-random.github.io](https://llm-random.github.io/).

## Publications, preprints and blogposts
- Scaling Laws for Fine-Grained Mixture of Experts ([arxiv](https://arxiv.org/abs/2402.07871))
- MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts ([arxiv](https://arxiv.org/abs/2401.04081), [blogpost](https://llm-random.github.io/posts/moe_mamba/))
- Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation ([arxiv](https://arxiv.org/abs/2310.15961), [blogpost](https://llm-random.github.io/posts/mixture_of_tokens/))

## Development (WIP)
### Getting started
In the root directory run `./start-dev.sh`. This will create a virtual environment, install requirements and set up git hooks.

## Running Experiments (WIP)

### Experiments config
Use the baseline configuration as a template, which is in `configs/test/test_baseline.yaml`. Based on this template, create a new experiment config and put it in `lizrd/scripts/run_configs`.

### Running Locally
`python -m lizrd.grid path/to/config`

### Running Remotely
`bash scripts/run_exp_remotely.sh <remote_cluster_name> scripts/run_configs/<your_config>`

# License

This project is licensed under the terms of the Apache License, Version 2.0.

    Copyright 2023 LLM-Random Authors

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.