# nCoV Ingestion Pipeline

Internal tooling for the Nextstrain team to ingest and curate SARS-CoV-2 genome sequences. The pipeline is open source, but we are not intending to support it for use by outside groups.
Relies on data from <https://simplemaps.com/data/us-cities>.

Outputs documented here are part of `ncov-ingest`'s public API: https://docs.nextstrain.org/projects/ncov/en/latest/reference/remote_inputs.html

## Running locally

> NOTE: The full set of sequences from GISAID/GenBank will most likely require more compute resources than what is available on your local computer.

To debug all rules on a subset of the data, you can use the `config/debug_sample_genbank.yaml` and `config/debug_sample_gisaid.yaml` config files.
These will download raw data from AWS s3, randomly keeping only a subset of lines of the input files (configurable in the config file).
This way, the pipeline completes in a matter of minutes and acceptable storage requirements for local compute.
However, the output data should not be trusted, as biosample and cog-uk input lines are randomly selected independently of the main ndjson.

To get started, you can run the following:

```sh
snakemake -j all --configfile config/debug_sample_genbank.yaml  -pF --ri --nt
```

> **Warning**
> If you are running the pipeline without a Nextclade cache, it will do a full Nextclade run that aligns _all_ sequences,
> which will take significant time and resources!

Follow these instructions to run the ncov-ingest pipeline _without_ all the bells and whistles used by internal Nextstrain runs that involve AWS S3, Slack notifications, and GitHub Action triggers:

### GISAID

To pull sequences directly from GISAID, you are required to set two environment variables:

- `GISAID_API_ENDPOINT`
- `GISAID_USERNAME_AND_PASSWORD`

Then run the ncov-ingest pipeline with the nextstrain CLI:

```sh
nextstrain build \
  --image nextstrain/ncov-ingest \
  --env GISAID_API_ENDPOINT \
  --env GISAID_USERNAME_AND_PASSWORD \
  . \
    --configfile config/local_gisaid.yaml
```

### GenBank

Sequences can be pulled from GenBank without any environment variables.
Run the ncov-ingest pipeline with the nextstrain CLI:

```sh
nextstrain build \
  --image nextstrain/ncov-ingest \
  . \
  --configfile config/local_genbank.yaml \
```

## Running automatically

The ingest pipelines are triggered from the GitHub workflows `.github/workflows/ingest-master-*.yml` and `…/ingest-branch-*.yml` but run on AWS Batch via the `nextstrain build --aws-batch` infrastructure.
They're run on pushes to `master` that modify `source-data/*-annotations.tsv` and on pushes to other branches.
Pushes to branches other than `master` upload files to branch-specific paths in the S3 bucket, don't send notifications, and don't trigger Nextstrain rebuilds, so that they don't interfere with the production data.

AWS credentials are stored in this repository's secrets and are associated with the `nextstrain-ncov-ingest-uploader` IAM user in the Bedford Lab AWS account, which is locked down to reading and publishing only the `gisaid.ndjson`, `metadata.tsv`, and `sequences.fasta` files and their zipped equivalents in the `nextstrain-ncov-private` S3 bucket.

## Manually triggering the automation

A full run is now done in 3 steps via manual triggers:

1. Fetch new sequences and ingest them by running `./vendored/trigger nextstrain/ncov-ingest gisaid/fetch-and-ingest --user <your-github-username>`.
2. Add manual annotations, update location hierarchy as needed, and run ingest without fetching new sequences.
    - Pushes of `source-data/*-annotations.tsv` to the master branch will automatically trigger a run of ingest.
    - You can also run ingest manually by running `./vendored/trigger nextstrain/ncov-ingest gisaid/ingest --user <your-github-username>`.
3. Once all manual fixes are complete, trigger a rebuild of [nextstrain/ncov](https://github.com/nextstrain/ncov) by running `./vendored/trigger ncov gisaid/rebuild --user <your-github-username>`.

See the output of `./vendored/trigger nextstrain/ncov-ingest gisaid/fetch-and-ingest --user <your-github-username>`, `./vendored/trigger nextstrain/ncov-ingest gisaid/ingest` or `./vendored/trigger nextstrain/ncov-ingest rebuild` for more information about authentication with GitHub.

Note: running `./vendored/trigger nextstrain/ncov-ingest` posts a GitHub `repository_dispatch`.
Regardless of which branch you are on, it will trigger the specified action on the master branch.

Valid dispatch types for `./vendored/trigger nextstrain/ncov-ingest` are:

- `ingest` (both GISAID and GenBank)
- `gisaid/ingest`
- `genbank/ingest`
- `fetch-and-ingest` (both GISAID and GenBank)
- `gisaid/fetch-and-ingest`
- `genbank/fetch-and-ingest`

## Updating manual annotations

Manual annotations should be added to `source-data/gisaid_annotations.tsv`.
A common pattern is expected to be:

 1. Run <https://github.com/nextstrain/ncov>.
 2. Discover metadata that needs fixing.
 3. Update `source-data/gisaid_annotations.tsv`.
 4. Push changes to `master` and re-download `gisaid/metadata.tsv`.

## Configuring alerts for new GISAID data from specific location hierarchy areas

Some Nextstrain team members may be interested in receiving alerts when new GISAID strains are added from specific locations, e.g. Portugal or Louisiana.
To add a custom alert configuration, create a new entry in `new-sequence-alerts-config.json`.
Each resolution (region, division, country, location) accepts a list of strings of areas of interest.
Note that these strings must match the area name exactly.

To set up custom alerts, you'll need to retrieve your Slack member ID.
Note that the `user` field in each alert configuration is for human use only -- it need not match your Slack display name or username.
To view your Slack member ID, open up the Slack menu by clicking your name at the top, and then click on 'View profile'.
Then, click on 'More'.
You can then copy your Slack member ID from the menu that appears.
Enter this into the `slack_member_id` field of your alert configuration.

## Rerunning Nextclade ignoring cache after Nextclade dataset is updated

Clade assignments and other QC metadata output by Nextclade are currently cached in `nextclade.tsv` in the S3 bucket and only incremental additions for the new sequences are performed during the daily ingests.
Whenever the underlying nextclade dataset (reference tree, QC rules) and/or nextclade software are updated, it is necessary to perform a full update of `nextclade.tsv`, rerunning for all of the GISAID and GenBank sequences all over again, to account for changes in the data and Nextclade algorithms.

In order to tell ingest to not use the cached `nextclade.tsv`/`aligned.fasta` and instead perform a full rerun, you need to add an (empty) touchfile to the s3 bucket (available as `./scripts/developer_scripts/rerun-nextclade.sh`):

```bash
aws s3 cp - s3://nextstrain-ncov-private/nextclade.tsv.zst.renew < /dev/null
aws s3 cp - s3://nextstrain-data/files/ncov/open/nextclade.tsv.zst.renew < /dev/null
```

Ingest will automatically remove the touchfiles after it has completed the rerun.

To rerun Nextclade using the `sars-cov-2-21L` dataset - which is only necessary when the calculation of `immune_escape` and `ace2_binding` changes - you need to add an (empty) touchfile to the s3 bucket (available as `./scripts/developer_scripts/rerun-nextclade-21L.sh`:

```bash
aws s3 cp - s3://nextstrain-ncov-private/nextclade_21L.tsv.zst.renew < /dev/null
aws s3 cp - s3://nextstrain-data/files/ncov/open/nextclade_21L.tsv.zst.renew < /dev/null
```

## Required environment variables

- `GISAID_API_ENDPOINT`
- `GISAID_USERNAME_AND_PASSWORD`
- `AWS_DEFAULT_REGION`
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `SLACK_TOKEN`
- `SLACK_CHANNELS`

## `vendored`

This repository uses [`git subrepo`](https://github.com/ingydotnet/git-subrepo) to manage copies of ingest scripts in [`vendored`](./vendored), from [nextstrain/ingest](https://github.com/nextstrain/ingest). To pull new changes from the central ingest repository, first install `git subrepo`, then run:

See [vendored/README.md](vendored/README.md#vendoring) for instructions on how to update
the vendored scripts. Note that this repo is a special case and does not put vendored
scripts in an `ingest/` directory. Modify commands accordingly.


![SQLGlot logo](sqlglot.png)

SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It can be used to format SQL or translate between [21 different dialects](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/__init__.py) like [DuckDB](https://duckdb.org/), [Presto](https://prestodb.io/) / [Trino](https://trino.io/), [Spark](https://spark.apache.org/) / [Databricks](https://www.databricks.com/), [Snowflake](https://www.snowflake.com/en/), and [BigQuery](https://cloud.google.com/bigquery/). It aims to read a wide variety of SQL inputs and output syntactically and semantically correct SQL in the targeted dialects.

It is a very comprehensive generic SQL parser with a robust [test suite](https://github.com/tobymao/sqlglot/blob/main/tests/). It is also quite [performant](#benchmarks), while being written purely in Python.

You can easily [customize](#custom-dialects) the parser, [analyze](#metadata) queries, traverse expression trees, and programmatically [build](#build-and-modify-sql) SQL.

Syntax [errors](#parser-errors) are highlighted and dialect incompatibilities can warn or raise depending on configurations. However, SQLGlot does not aim to be a SQL validator, so it may fail to detect certain syntax errors.

Learn more about SQLGlot in the API [documentation](https://sqlglot.com/) and the expression tree [primer](https://github.com/tobymao/sqlglot/blob/main/posts/ast_primer.md).

Contributions are very welcome in SQLGlot; read the [contribution guide](https://github.com/tobymao/sqlglot/blob/main/CONTRIBUTING.md) to get started!

## Table of Contents

* [Install](#install)
* [Versioning](#versioning)
* [Get in Touch](#get-in-touch)
* [FAQ](#faq)
* [Examples](#examples)
   * [Formatting and Transpiling](#formatting-and-transpiling)
   * [Metadata](#metadata)
   * [Parser Errors](#parser-errors)
   * [Unsupported Errors](#unsupported-errors)
   * [Build and Modify SQL](#build-and-modify-sql)
   * [SQL Optimizer](#sql-optimizer)
   * [AST Introspection](#ast-introspection)
   * [AST Diff](#ast-diff)
   * [Custom Dialects](#custom-dialects)
   * [SQL Execution](#sql-execution)
* [Used By](#used-by)
* [Documentation](#documentation)
* [Run Tests and Lint](#run-tests-and-lint)
* [Benchmarks](#benchmarks)
* [Optional Dependencies](#optional-dependencies)

## Install

From PyPI:

```bash
pip3 install "sqlglot[rs]"

# Without Rust tokenizer (slower):
# pip3 install sqlglot
```

Or with a local checkout:

```
make install
```

Requirements for development (optional):

```
make install-dev
```

## Versioning

Given a version number `MAJOR`.`MINOR`.`PATCH`, SQLGlot uses the following versioning strategy:

- The `PATCH` version is incremented when there are backwards-compatible fixes or feature additions.
- The `MINOR` version is incremented when there are backwards-incompatible fixes or feature additions.
- The `MAJOR` version is incremented when there are significant backwards-incompatible fixes or feature additions.

## Get in Touch

We'd love to hear from you. Join our community [Slack channel](https://tobikodata.com/slack)!

## FAQ

I tried to parse SQL that should be valid but it failed, why did that happen?

* Most of the time, issues like this occur because the "source" dialect is omitted during parsing. For example, this is how to correctly parse a SQL query written in Spark SQL: `parse_one(sql, dialect="spark")` (alternatively: `read="spark"`). If no dialect is specified, `parse_one` will attempt to parse the query according to the "SQLGlot dialect", which is designed to be a superset of all supported dialects. If you tried specifying the dialect and it still doesn't work, please file an issue.

I tried to output SQL but it's not in the correct dialect!

* Like parsing, generating SQL also requires the target dialect to be specified, otherwise the SQLGlot dialect will be used by default. For example, to transpile a query from Spark SQL to DuckDB, do `parse_one(sql, dialect="spark").sql(dialect="duckdb")` (alternatively: `transpile(sql, read="spark", write="duckdb")`).

I tried to parse invalid SQL and it worked, even though it should raise an error! Why didn't it validate my SQL?

* SQLGlot does not aim to be a SQL validator - it is designed to be very forgiving. This makes the codebase more comprehensive and also gives more flexibility to its users, e.g. by allowing them to include trailing commas in their projection lists.

What happened to sqlglot.dataframe?

* The PySpark dataframe api was moved to a standalone library called [SQLFrame](https://github.com/eakmanrq/sqlframe) in v24. It now allows you to run queries as opposed to just generate SQL.

## Examples

### Formatting and Transpiling

Easily translate from one dialect to another. For example, date/time functions vary between dialects and can be hard to deal with:

```python
import sqlglot
sqlglot.transpile("SELECT EPOCH_MS(1618088028295)", read="duckdb", write="hive")[0]
```

```sql
'SELECT FROM_UNIXTIME(1618088028295 / POW(10, 3))'
```

SQLGlot can even translate custom time formats:

```python
import sqlglot
sqlglot.transpile("SELECT STRFTIME(x, '%y-%-m-%S')", read="duckdb", write="hive")[0]
```

```sql
"SELECT DATE_FORMAT(x, 'yy-M-ss')"
```

Identifier delimiters and data types can be translated as well:

```python
import sqlglot

# Spark SQL requires backticks (`) for delimited identifiers and uses `FLOAT` over `REAL`
sql = """WITH baz AS (SELECT a, c FROM foo WHERE a = 1) SELECT f.a, b.b, baz.c, CAST("b"."a" AS REAL) d FROM foo f JOIN bar b ON f.a = b.a LEFT JOIN baz ON f.a = baz.a"""

# Translates the query into Spark SQL, formats it, and delimits all of its identifiers
print(sqlglot.transpile(sql, write="spark", identify=True, pretty=True)[0])
```

```sql
WITH `baz` AS (
  SELECT
    `a`,
    `c`
  FROM `foo`
  WHERE
    `a` = 1
)
SELECT
  `f`.`a`,
  `b`.`b`,
  `baz`.`c`,
  CAST(`b`.`a` AS FLOAT) AS `d`
FROM `foo` AS `f`
JOIN `bar` AS `b`
  ON `f`.`a` = `b`.`a`
LEFT JOIN `baz`
  ON `f`.`a` = `baz`.`a`
```

Comments are also preserved on a best-effort basis:

```python
sql = """
/* multi
   line
   comment
*/
SELECT
  tbl.cola /* comment 1 */ + tbl.colb /* comment 2 */,
  CAST(x AS SIGNED), # comment 3
  y               -- comment 4
FROM
  bar /* comment 5 */,
  tbl #          comment 6
"""

# Note: MySQL-specific comments (`#`) are converted into standard syntax
print(sqlglot.transpile(sql, read='mysql', pretty=True)[0])
```

```sql
/* multi
   line
   comment
*/
SELECT
  tbl.cola /* comment 1 */ + tbl.colb /* comment 2 */,
  CAST(x AS INT), /* comment 3 */
  y /* comment 4 */
FROM bar /* comment 5 */, tbl /*          comment 6 */
```

### Metadata

You can explore SQL with expression helpers to do things like find columns and tables in a query:

```python
from sqlglot import parse_one, exp

# print all column references (a and b)
for column in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Column):
    print(column.alias_or_name)

# find all projections in select statements (a and c)
for select in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Select):
    for projection in select.expressions:
        print(projection.alias_or_name)

# find all tables (x, y, z)
for table in parse_one("SELECT * FROM x JOIN y JOIN z").find_all(exp.Table):
    print(table.name)
```

Read the [ast primer](https://github.com/tobymao/sqlglot/blob/main/posts/ast_primer.md) to learn more about SQLGlot's internals.

### Parser Errors

When the parser detects an error in the syntax, it raises a `ParseError`:

```python
import sqlglot
sqlglot.transpile("SELECT foo FROM (SELECT baz FROM t")
```

```
sqlglot.errors.ParseError: Expecting ). Line 1, Col: 34.
  SELECT foo FROM (SELECT baz FROM t
                                   ~
```

Structured syntax errors are accessible for programmatic use:

```python
import sqlglot
try:
    sqlglot.transpile("SELECT foo FROM (SELECT baz FROM t")
except sqlglot.errors.ParseError as e:
    print(e.errors)
```

```python
[{
  'description': 'Expecting )',
  'line': 1,
  'col': 34,
  'start_context': 'SELECT foo FROM (SELECT baz FROM ',
  'highlight': 't',
  'end_context': '',
  'into_expression': None
}]
```

### Unsupported Errors

It may not be possible to translate some queries between certain dialects. For these cases, SQLGlot may emit a warning and will proceed to do a best-effort translation by default:

```python
import sqlglot
sqlglot.transpile("SELECT APPROX_DISTINCT(a, 0.1) FROM foo", read="presto", write="hive")
```

```sql
APPROX_COUNT_DISTINCT does not support accuracy
'SELECT APPROX_COUNT_DISTINCT(a) FROM foo'
```

This behavior can be changed by setting the [`unsupported_level`](https://github.com/tobymao/sqlglot/blob/b0e8dc96ba179edb1776647b5bde4e704238b44d/sqlglot/errors.py#L9) attribute. For example, we can set it to either `RAISE` or `IMMEDIATE` to ensure an exception is raised instead:

```python
import sqlglot
sqlglot.transpile("SELECT APPROX_DISTINCT(a, 0.1) FROM foo", read="presto", write="hive", unsupported_level=sqlglot.ErrorLevel.RAISE)
```

```
sqlglot.errors.UnsupportedError: APPROX_COUNT_DISTINCT does not support accuracy
```

There are queries that require additional information to be accurately transpiled, such as the schemas of the tables referenced in them. This is because certain transformations are type-sensitive, meaning that type inference is needed in order to understand their semantics. Even though the `qualify` and `annotate_types` optimizer [rules](https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimizer) can help with this, they are not used by default because they add significant overhead and complexity.

Transpilation is generally a hard problem, so SQLGlot employs an "incremental" approach to solving it. This means that there may be dialect pairs that currently lack support for some inputs, but this is expected to improve over time. We highly appreciate well-documented and tested issues or PRs, so feel free to [reach out](#get-in-touch) if you need guidance!

### Build and Modify SQL

SQLGlot supports incrementally building SQL expressions:

```python
from sqlglot import select, condition

where = condition("x=1").and_("y=1")
select("*").from_("y").where(where).sql()
```

```sql
'SELECT * FROM y WHERE x = 1 AND y = 1'
```

It's possible to modify a parsed tree:

```python
from sqlglot import parse_one
parse_one("SELECT x FROM y").from_("z").sql()
```

```sql
'SELECT x FROM z'
```

Parsed expressions can also be transformed recursively by applying a mapping function to each node in the tree:

```python
from sqlglot import exp, parse_one

expression_tree = parse_one("SELECT a FROM x")

def transformer(node):
    if isinstance(node, exp.Column) and node.name == "a":
        return parse_one("FUN(a)")
    return node

transformed_tree = expression_tree.transform(transformer)
transformed_tree.sql()
```

```sql
'SELECT FUN(a) FROM x'
```

### SQL Optimizer

SQLGlot can rewrite queries into an "optimized" form. It performs a variety of [techniques](https://github.com/tobymao/sqlglot/blob/main/sqlglot/optimizer/optimizer.py) to create a new canonical AST. This AST can be used to standardize queries or provide the foundations for implementing an actual engine. For example:

```python
import sqlglot
from sqlglot.optimizer import optimize

print(
    optimize(
        sqlglot.parse_one("""
            SELECT A OR (B OR (C AND D))
            FROM x
            WHERE Z = date '2021-01-01' + INTERVAL '1' month OR 1 = 0
        """),
        schema={"x": {"A": "INT", "B": "INT", "C": "INT", "D": "INT", "Z": "STRING"}}
    ).sql(pretty=True)
)
```

```sql
SELECT
  (
    "x"."a" <> 0 OR "x"."b" <> 0 OR "x"."c" <> 0
  )
  AND (
    "x"."a" <> 0 OR "x"."b" <> 0 OR "x"."d" <> 0
  ) AS "_col_0"
FROM "x" AS "x"
WHERE
  CAST("x"."z" AS DATE) = CAST('2021-02-01' AS DATE)
```

### AST Introspection

You can see the AST version of the parsed SQL by calling `repr`:

```python
from sqlglot import parse_one
print(repr(parse_one("SELECT a + 1 AS z")))
```

```python
Select(
  expressions=[
    Alias(
      this=Add(
        this=Column(
          this=Identifier(this=a, quoted=False)),
        expression=Literal(this=1, is_string=False)),
      alias=Identifier(this=z, quoted=False))])
```

### AST Diff

SQLGlot can calculate the semantic difference between two expressions and output changes in a form of a sequence of actions needed to transform a source expression into a target one:

```python
from sqlglot import diff, parse_one
diff(parse_one("SELECT a + b, c, d"), parse_one("SELECT c, a - b, d"))
```

```python
[
  Remove(expression=Add(
    this=Column(
      this=Identifier(this=a, quoted=False)),
    expression=Column(
      this=Identifier(this=b, quoted=False)))),
  Insert(expression=Sub(
    this=Column(
      this=Identifier(this=a, quoted=False)),
    expression=Column(
      this=Identifier(this=b, quoted=False)))),
  Keep(
    source=Column(this=Identifier(this=a, quoted=False)),
    target=Column(this=Identifier(this=a, quoted=False))),
  ...
]
```

See also: [Semantic Diff for SQL](https://github.com/tobymao/sqlglot/blob/main/posts/sql_diff.md).

### Custom Dialects

[Dialects](https://github.com/tobymao/sqlglot/tree/main/sqlglot/dialects) can be added by subclassing `Dialect`:

```python
from sqlglot import exp
from sqlglot.dialects.dialect import Dialect
from sqlglot.generator import Generator
from sqlglot.tokens import Tokenizer, TokenType

class Custom(Dialect):
    class Tokenizer(Tokenizer):
        QUOTES = ["'", '"']
        IDENTIFIERS = ["`"]

        KEYWORDS = {
            **Tokenizer.KEYWORDS,
            "INT64": TokenType.BIGINT,
            "FLOAT64": TokenType.DOUBLE,
        }

    class Generator(Generator):
        TRANSFORMS = {exp.Array: lambda self, e: f"[{self.expressions(e)}]"}

        TYPE_MAPPING = {
            exp.DataType.Type.TINYINT: "INT64",
            exp.DataType.Type.SMALLINT: "INT64",
            exp.DataType.Type.INT: "INT64",
            exp.DataType.Type.BIGINT: "INT64",
            exp.DataType.Type.DECIMAL: "NUMERIC",
            exp.DataType.Type.FLOAT: "FLOAT64",
            exp.DataType.Type.DOUBLE: "FLOAT64",
            exp.DataType.Type.BOOLEAN: "BOOL",
            exp.DataType.Type.TEXT: "STRING",
        }

print(Dialect["custom"])
```

```
<class '__main__.Custom'>
```

### SQL Execution

SQLGlot is able to interpret SQL queries, where the tables are represented as Python dictionaries. The engine is not supposed to be fast, but it can be useful for unit testing and running SQL natively across Python objects. Additionally, the foundation can be easily integrated with fast compute kernels, such as [Arrow](https://arrow.apache.org/docs/index.html) and [Pandas](https://pandas.pydata.org/).

The example below showcases the execution of a query that involves aggregations and joins:

```python
from sqlglot.executor import execute

tables = {
    "sushi": [
        {"id": 1, "price": 1.0},
        {"id": 2, "price": 2.0},
        {"id": 3, "price": 3.0},
    ],
    "order_items": [
        {"sushi_id": 1, "order_id": 1},
        {"sushi_id": 1, "order_id": 1},
        {"sushi_id": 2, "order_id": 1},
        {"sushi_id": 3, "order_id": 2},
    ],
    "orders": [
        {"id": 1, "user_id": 1},
        {"id": 2, "user_id": 2},
    ],
}

execute(
    """
    SELECT
      o.user_id,
      SUM(s.price) AS price
    FROM orders o
    JOIN order_items i
      ON o.id = i.order_id
    JOIN sushi s
      ON i.sushi_id = s.id
    GROUP BY o.user_id
    """,
    tables=tables
)
```

```python
user_id price
      1   4.0
      2   3.0
```

See also: [Writing a Python SQL engine from scratch](https://github.com/tobymao/sqlglot/blob/main/posts/python_sql_engine.md).

## Used By

* [SQLMesh](https://github.com/TobikoData/sqlmesh)
* [Apache Superset](https://github.com/apache/superset)
* [Dagster](https://github.com/dagster-io/dagster)
* [Fugue](https://github.com/fugue-project/fugue)
* [ibis](https://github.com/ibis-project/ibis)
* [mysql-mimic](https://github.com/kelsin/mysql-mimic)
* [Querybook](https://github.com/pinterest/querybook)
* [Quokka](https://github.com/marsupialtail/quokka)
* [Splink](https://github.com/moj-analytical-services/splink)
* [SQLFrame](https://github.com/eakmanrq/sqlframe)

## Documentation

SQLGlot uses [pdoc](https://pdoc.dev/) to serve its API documentation.

A hosted version is on the [SQLGlot website](https://sqlglot.com/), or you can build locally with:

```
make docs-serve
```

## Run Tests and Lint

```
make style  # Only linter checks
make unit   # Only unit tests (or unit-rs, to use the Rust tokenizer)
make test   # Unit and integration tests (or test-rs, to use the Rust tokenizer)
make check  # Full test suite & linter checks
```

## Benchmarks

[Benchmarks](https://github.com/tobymao/sqlglot/blob/main/benchmarks/bench.py) run on Python 3.10.12 in seconds.

|           Query |         sqlglot |       sqlglotrs |        sqlfluff |         sqltree |        sqlparse |  moz_sql_parser |        sqloxide |
| --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
|            tpch |   0.00944 (1.0) | 0.00590 (0.625) | 0.32116 (33.98) | 0.00693 (0.734) | 0.02858 (3.025) | 0.03337 (3.532) | 0.00073 (0.077) |
|           short |   0.00065 (1.0) | 0.00044 (0.687) | 0.03511 (53.82) | 0.00049 (0.759) | 0.00163 (2.506) | 0.00234 (3.601) | 0.00005 (0.073) |
|            long |   0.00889 (1.0) | 0.00572 (0.643) | 0.36982 (41.56) | 0.00614 (0.690) | 0.02530 (2.844) | 0.02931 (3.294) | 0.00059 (0.066) |
|           crazy |   0.02918 (1.0) | 0.01991 (0.682) | 1.88695 (64.66) | 0.02003 (0.686) | 7.46894 (255.9) | 0.64994 (22.27) | 0.00327 (0.112) |

## Optional Dependencies

SQLGlot uses [dateutil](https://github.com/dateutil/dateutil) to simplify literal timedelta expressions. The optimizer will not simplify expressions like the following if the module cannot be found:

```sql
x + interval '1' month
```


[![Build Status](https://github.com/MatterMiners/tardis/actions/workflows/unittests.yml/badge.svg)](https://github.com/MatterMiners/tardis/actions/workflows/unittests.yml)
[![Verification](https://github.com/MatterMiners/tardis/actions/workflows/verification.yml/badge.svg)](https://github.com/MatterMiners/tardis/actions/workflows/verification.yml)
[![Deployment Tests](https://github.com/MatterMiners/tardis/actions/workflows/deployment-tests.yml/badge.svg)](https://github.com/MatterMiners/tardis/actions/workflows/deployment-tests.yml)
[![codecov](https://codecov.io/gh/MatterMiners/tardis/branch/master/graph/badge.svg)](https://codecov.io/gh/MatterMiners/tardis)
[![Documentation Status](https://readthedocs.org/projects/cobald-tardis/badge/?version=latest)](https://cobald-tardis.readthedocs.io/en/latest/?badge=latest)
[![Development and Help Chat](https://badges.gitter.im/MatterMiners.png)](https://gitter.im/MatterMiners/community)
[![PyPI version](https://badge.fury.io/py/cobald-tardis.svg)](https://badge.fury.io/py/cobald-tardis)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/cobald-tardis.svg?style=flat-square)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/MatterMiners/tardis/blob/master/LICENSE.txt)
[![DOI](https://zenodo.org/badge/132791417.svg)](https://zenodo.org/badge/latestdoi/132791417)
[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# TARDIS Resourcemanager

![](docs/pics/TARDIS_logo.svg)

**T**ransparent **A**daptive **R**esource **D**ynamic **I**ntegration **S**ystem
enables the dynamic integration of resources provided by different resource
providers into one overlay batch system.


# PDM

A modern Python package and dependency manager supporting the latest PEP standards.
[中文版本说明](README_zh.md)

![PDM logo](https://raw.githubusercontent.com/pdm-project/pdm/main/docs/assets/logo_big.png)

[![Docs](https://img.shields.io/badge/Docs-mkdocs-blue?style=for-the-badge)](https://pdm-project.org)
[![Twitter Follow](https://img.shields.io/twitter/follow/pdm_project?label=get%20updates&logo=twitter&style=for-the-badge)](https://twitter.com/pdm_project)
[![Discord](https://img.shields.io/discord/824472774965329931?label=discord&logo=discord&style=for-the-badge)](https://discord.gg/Phn8smztpv)

![Github Actions](https://github.com/pdm-project/pdm/workflows/Tests/badge.svg)
[![PyPI](https://img.shields.io/pypi/v/pdm?logo=python&logoColor=%23cccccc)](https://pypi.org/project/pdm)
[![codecov](https://codecov.io/gh/pdm-project/pdm/branch/main/graph/badge.svg?token=erZTquL5n0)](https://codecov.io/gh/pdm-project/pdm)
[![Packaging status](https://repology.org/badge/tiny-repos/pdm.svg)](https://repology.org/project/pdm/versions)
[![Downloads](https://pepy.tech/badge/pdm/week)](https://pepy.tech/project/pdm)
[![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org)

[![asciicast](https://asciinema.org/a/jnifN30pjfXbO9We2KqOdXEhB.svg)](https://asciinema.org/a/jnifN30pjfXbO9We2KqOdXEhB)

## What is PDM?

PDM is meant to be a next generation Python package management tool.
It was originally built for personal use. If you feel you are going well
with `Pipenv` or `Poetry` and don't want to introduce another package manager,
just stick to it. But if you are missing something that is not present in those tools,
you can probably find some goodness in `pdm`.

## Highlights of features

- Simple and fast dependency resolver, mainly for large binary distributions.
- A [PEP 517] build backend.
- [PEP 621] project metadata.
- Flexible and powerful plug-in system.
- Versatile user scripts.
- Install Pythons using [indygreg's python-build-standalone](https://github.com/indygreg/python-build-standalone).
- Opt-in centralized installation cache like [pnpm](https://pnpm.io/motivation#saving-disk-space-and-boosting-installation-speed).

[pep 517]: https://www.python.org/dev/peps/pep-0517
[pep 621]: https://www.python.org/dev/peps/pep-0621
[pnpm]: https://pnpm.io/motivation#saving-disk-space-and-boosting-installation-speed

## Comparisons to other alternatives

### [Pipenv](https://pipenv.pypa.io)

Pipenv is a dependency manager that combines `pip` and `venv`, as the name implies.
It can install packages from a non-standard `Pipfile.lock` or `Pipfile`.
However, Pipenv does not handle any packages related to packaging your code,
so it’s useful only for developing non-installable applications (Django sites, for example).
If you’re a library developer, you need `setuptools` anyway.

### [Poetry](https://python-poetry.org)

Poetry manages environments and dependencies in a similar way to Pipenv,
but it can also build .whl files with your code, and it can upload wheels and source distributions to PyPI.
It has a pretty user interface and users can customize it via a plugin. Poetry uses the `pyproject.toml` standard,
but it does not follow the standard specifying how metadata should be represented in a pyproject.toml file ([PEP 621]),
instead using a custom `[tool.poetry]` table. This is partly because Poetry came out before PEP 621.

### [Hatch](https://hatch.pypa.io)

Hatch can also manage environments, allowing multiple environments per project. By default it has a central location for all environments but it can be configured to put a project's environment(s) in the project root directory. It can manage packages but without lockfile support. It can also be used to package a project (with PEP 621 compliant pyproject.toml files) and upload it to PyPI.

### This project

PDM can manage virtual environments (venvs) in both project and centralized locations, similar to Pipenv. It reads project metadata from a standardized `pyproject.toml` file and supports lockfiles. Users can add additional functionality through plugins, which can be shared by uploading them as distributions.

Unlike Poetry and Hatch, PDM is not limited to a specific build backend; users have the freedom to choose any build backend they prefer.

## Installation

PDM requires python version 3.8 or higher.

### Via Install Script

Like Pip, PDM provides an installation script that will install PDM into an isolated environment.

**For Linux/Mac**

```bash
curl -sSL https://pdm-project.org/install-pdm.py | python3 -
```

**For Windows**

```powershell
(Invoke-WebRequest -Uri https://pdm-project.org/install-pdm.py -UseBasicParsing).Content | py -
```

For security reasons, you should verify the checksum of `install-pdm.py`.
It can be downloaded from [install-pdm.py.sha256](https://pdm-project.org/install-pdm.py.sha256).

The installer will install PDM into the user site and the location depends on the system:

- `$HOME/.local/bin` for Linux
- `$HOME/Library/Python/<version>/bin` for MacOS
- `%APPDATA%\Python\Scripts` on Windows

You can pass additional options to the script to control how PDM is installed:

```
usage: install-pdm.py [-h] [-v VERSION] [--prerelease] [--remove] [-p PATH] [-d DEP]

optional arguments:
  -h, --help            show this help message and exit
  -v VERSION, --version VERSION | envvar: PDM_VERSION
                        Specify the version to be installed, or HEAD to install from the main branch
  --prerelease | envvar: PDM_PRERELEASE    Allow prereleases to be installed
  --remove | envvar: PDM_REMOVE            Remove the PDM installation
  -p PATH, --path PATH | envvar: PDM_HOME  Specify the location to install PDM
  -d DEP, --dep DEP | envvar: PDM_DEPS     Specify additional dependencies, can be given multiple times
```

You can either pass the options after the script or set the env var value.

### Alternative Installation Methods

If you are on macOS and using `homebrew`, install it by:

```bash
brew install pdm
```

If you are on Windows and using [Scoop](https://scoop.sh/), install it by:

```
scoop bucket add frostming https://github.com/frostming/scoop-frostming.git
scoop install pdm
```

Otherwise, it is recommended to install `pdm` in an isolated environment with `pipx`:

```bash
pipx install pdm
```

Or you can install it under a user site:

```bash
pip install --user pdm
```

With [asdf-vm](https://asdf-vm.com/)

```bash
asdf plugin add pdm
asdf install pdm latest
```

## Quickstart

**Initialize a new PDM project**

```bash
pdm init
```

Answer the questions following the guide, and a PDM project with a `pyproject.toml` file will be ready to use.

**Install dependencies**

```bash
pdm add requests flask
```

You can add multiple dependencies in the same command. After a while, check the `pdm.lock` file to see what is locked for each package.

## Badges

Tell people you are using PDM in your project by including the markdown code in README.md:

```markdown
[![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org)
```

[![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org)

## Packaging Status

[![Packaging status](https://repology.org/badge/vertical-allrepos/pdm.svg)](https://repology.org/project/pdm/versions)

## PDM Eco-system

[Awesome PDM](https://github.com/pdm-project/awesome-pdm) is a curated list of awesome PDM plugins and resources.

## Sponsors

## Credits

This project is strongly inspired by [pyflow] and [poetry].

[pyflow]: https://github.com/David-OConnor/pyflow
[poetry]: https://github.com/python-poetry/poetry

## License

This project is open sourced under MIT license, see the [LICENSE](LICENSE) file for more details.


# pyjanitor

[![](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/ericmjl/pyjanitor/dev)

<!-- pypi-doc -->
`pyjanitor` is a Python implementation of the R package [`janitor`][janitor], and
provides a clean API for cleaning data.

[janitor]: https://github.com/sfirke/janitor

## Quick start

- Installation: `conda install -c conda-forge pyjanitor`. Read more installation instructions [here](https://pyjanitor-devs.github.io/pyjanitor/#installation).
- Check out the collection of [general functions](https://pyjanitor-devs.github.io/pyjanitor/api/functions/).

## Why janitor?

Originally a port of the R package,
`pyjanitor` has evolved from a set of convenient data cleaning routines
into an experiment with the [`method chaining`][mc] paradigm.

[mc]: https://towardsdatascience.com/the-unreasonable-effectiveness-of-method-chaining-in-pandas-15c2109e3c69

Data preprocessing usually consists of a series of steps
that involve transforming raw data into an understandable/usable format.
These series of steps need to be run in a certain sequence to achieve success.
We take a base data file as the starting point,
and perform actions on it,
such as removing null/empty rows,
replacing them with other values,
adding/renaming/removing columns of data,
filtering rows and others.
More formally, these steps along with their relationships
and dependencies are commonly referred to as a Directed Acyclic Graph (DAG).

The `pandas` API has been invaluable for the Python data science ecosystem,
and implements method chaining of a subset of methods as part of the API.
For example, resetting indexes (`.reset_index()`),
dropping null values (`.dropna()`), and more,
are accomplished via the appropriate `pd.DataFrame` method calls.

Inspired by the ease-of-use
and expressiveness of the `dplyr` package
of the R statistical language ecosystem,
we have evolved `pyjanitor` into a language
for expressing the data processing DAG for `pandas` users.
<!-- pypi-doc -->

To accomplish this,
actions for which we would need to invoke imperative-style statements,
can be replaced with method chains
that allow one to read off the logical order of actions taken.
Let us see the annotated example below.
First off, here is the textual description of a data cleaning pathway:

1. Create a `DataFrame`.
2. Delete one column.
3. Drop rows with empty values in two particular columns.
4. Rename another two columns.
5. Add a new column.

Let's import some libraries
and begin with some sample data for this example:

```python
# Libraries
import numpy as np
import pandas as pd
import janitor

# Sample Data curated for this example
company_sales = {
    'SalesMonth': ['Jan', 'Feb', 'Mar', 'April'],
    'Company1': [150.0, 200.0, 300.0, 400.0],
    'Company2': [180.0, 250.0, np.nan, 500.0],
    'Company3': [400.0, 500.0, 600.0, 675.0]
}
```

In `pandas` code, most users might type something like this:

```python
# The Pandas Way

# 1. Create a pandas DataFrame from the company_sales dictionary
df = pd.DataFrame.from_dict(company_sales)

# 2. Delete a column from the DataFrame. Say 'Company1'
del df['Company1']

# 3. Drop rows that have empty values in columns 'Company2' and 'Company3'
df = df.dropna(subset=['Company2', 'Company3'])

# 4. Rename 'Company2' to 'Amazon' and 'Company3' to 'Facebook'
df = df.rename(
    {
        'Company2': 'Amazon',
        'Company3': 'Facebook',
    },
    axis=1,
)

# 5. Let's add some data for another company. Say 'Google'
df['Google'] = [450.0, 550.0, 800.0]

# Output looks like this:
# Out[15]:
#   SalesMonth  Amazon  Facebook  Google
# 0        Jan   180.0     400.0   450.0
# 1        Feb   250.0     500.0   550.0
# 3      April   500.0     675.0   800.0
```

Slightly more advanced users might take advantage of the functional API:

```python
df = (
    pd.DataFrame(company_sales)
    .drop(columns="Company1")
    .dropna(subset=["Company2", "Company3"])
    .rename(columns={"Company2": "Amazon", "Company3": "Facebook"})
    .assign(Google=[450.0, 550.0, 800.0])
)

# The output is the same as before, and looks like this:
# Out[15]:
#   SalesMonth  Amazon  Facebook  Google
# 0        Jan   180.0     400.0   450.0
# 1        Feb   250.0     500.0   550.0
# 3      April   500.0     675.0   800.0
```

With `pyjanitor`, we enable method chaining with method names
that are *explicitly named verbs*, which describe the action taken.

```python
df = (
    pd.DataFrame.from_dict(company_sales)
    .remove_columns(["Company1"])
    .dropna(subset=["Company2", "Company3"])
    .rename_column("Company2", "Amazon")
    .rename_column("Company3", "Facebook")
    .add_column("Google", [450.0, 550.0, 800.0])
)

# Output looks like this:
# Out[15]:
#   SalesMonth  Amazon  Facebook  Google
# 0        Jan   180.0     400.0   450.0
# 1        Feb   250.0     500.0   550.0
# 3      April   500.0     675.0   800.0
```

As such,
`pyjanitor`'s etymology has a two-fold relationship to "cleanliness".
Firstly, it's about extending Pandas with convenient data cleaning routines.
Secondly, it's about providing a cleaner, method-chaining, verb-based API
for common pandas routines.

<!-- pypi-doc -->
## Installation

`pyjanitor` is currently installable from PyPI:

```bash
pip install pyjanitor
```

`pyjanitor` also can be installed by the conda package manager:

```bash
conda install pyjanitor -c conda-forge
```

`pyjanitor` can be installed by the pipenv environment manager too. This requires enabling prerelease dependencies:

```bash
pipenv install --pre pyjanitor
```

`pyjanitor` requires Python 3.6+.

## Functionality

Current functionality includes:

- Cleaning columns name (multi-indexes are possible!)
- Removing empty rows and columns
- Identifying duplicate entries
- Encoding columns as categorical
- Splitting your data into features and targets (for machine learning)
- Adding, removing, and renaming columns
- Coalesce multiple columns into a single column
- Date conversions (from matlab, excel, unix) to Python datetime format
- Expand a single column that has delimited, categorical values
  into dummy-encoded variables
- Concatenating and deconcatenating columns, based on a delimiter
- Syntactic sugar for filtering the dataframe based on queries on a column
- Experimental submodules for finance, biology, chemistry, engineering, and pyspark
<!-- pypi-doc -->

## API

The idea behind the API is two-fold:

- Copy the R package function names,
  but enable Pythonic use with method chaining or `pandas` piping.
- Add other utility functions
  that make it easy to do data cleaning/preprocessing in `pandas`.

Continuing with the company_sales dataframe previously used:

```python
import pandas as pd
import numpy as np
company_sales = {
    'SalesMonth': ['Jan', 'Feb', 'Mar', 'April'],
    'Company1': [150.0, 200.0, 300.0, 400.0],
    'Company2': [180.0, 250.0, np.nan, 500.0],
    'Company3': [400.0, 500.0, 600.0, 675.0]
}
```

As such, there are three ways to use the API.
The first, and most strongly recommended one, is to use `pyjanitor`'s functions
as if they were native to pandas.

```python
import janitor  # upon import, functions are registered as part of pandas.

# This cleans the column names as well as removes any duplicate rows
df = pd.DataFrame.from_dict(company_sales).clean_names().remove_empty()
```

The second is the functional API.

```python
from janitor import clean_names, remove_empty

df = pd.DataFrame.from_dict(company_sales)
df = clean_names(df)
df = remove_empty(df)
```

The final way is to use the [`pipe()`][pipe] method:

[pipe]: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pipe.html

```python
from janitor import clean_names, remove_empty
df = (
    pd.DataFrame.from_dict(company_sales)
    .pipe(clean_names)
    .pipe(remove_empty)
)
```

## Contributing

Follow the [development guide](https://pyjanitor-devs.github.io/pyjanitor/devguide/) for a full description of the process of contributing to `pyjanitor`.

## Adding new functionality

Keeping in mind the etymology of pyjanitor,
contributing a new function to pyjanitor
is a task that is not difficult at all.

### Define a function

First off, you will need to define the function
that expresses the data processing/cleaning routine,
such that it accepts a dataframe as the first argument,
and returns a modified dataframe:

```python
import pandas_flavor as pf

@pf.register_dataframe_method
def my_data_cleaning_function(df, arg1, arg2, ...):
    # Put data processing function here.
    return df
```

We use [`pandas_flavor`](https://github.com/Zsailer/pandas_flavor) to register the function natively on a `pandas.DataFrame`.

### Add a test case

Secondly, we ask that you contribute a test case,
to ensure that the function works as intended.
Follow the [contribution] docs for further details.

[contribution]: https://pyjanitor-devs.github.io/pyjanitor/contributing.html#unit-test-guidelines

### Feature requests

If you have a feature request,
please post it as an issue on the GitHub repository issue tracker.
Even better, put in a PR for it!
We are more than happy to guide you through the codebase
so that you can put in a contribution to the codebase.

Because `pyjanitor` is currently maintained by volunteers
and has no fiscal support,
any feature requests will be prioritized according to
what maintainers encounter as a need in our day-to-day jobs.
Please temper expectations accordingly.

## API Policy

`pyjanitor` only extends or aliases the `pandas` API
(and other dataframe APIs),
but will never fix or replace them.

Undesirable `pandas` behaviour should be reported upstream
in the `pandas` [issue tracker](https://github.com/pandas-dev/pandas/issues).
We explicitly do not fix the `pandas` API.
If at some point the `pandas` devs
decide to take something from `pyjanitor`
and internalize it as part of the official `pandas` API,
then we will deprecate it from `pyjanitor`,
while acknowledging the original contributors' contribution
as part of the official deprecation record.

## Credits

Test data for chemistry submodule can be found at [Predictive Toxicology][predtox].

[predtox]: https://www.predictive-toxicology.org/data/ntp/corrected_smiles.txt


# Intel(R) Extension for Scikit-learn*

 Speed up your scikit-learn applications for Intel(R) CPUs and GPUs across single- and multi-node configurations

[Releases](https://github.com/intel/scikit-learn-intelex/releases)   |   [Documentation](https://intel.github.io/scikit-learn-intelex/)   |   [Examples](https://github.com/intel/scikit-learn-intelex/tree/master/examples/notebooks)   |   [Support]()   |  [License](https://github.com/intel/scikit-learn-intelex/blob/master/LICENSE)

[![Build Status](https://dev.azure.com/daal/daal4py/_apis/build/status/CI?branchName=main)](https://dev.azure.com/daal/daal4py/_build/latest?definitionId=9&branchName=main)
[![Coverity Scan Build Status](https://scan.coverity.com/projects/21716/badge.svg)](https://scan.coverity.com/projects/daal4py)
[![Join the community on GitHub Discussions](https://badgen.net/badge/join%20the%20discussion/on%20github/black?icon=github)](https://github.com/intel/scikit-learn-intelex/discussions)
[![PyPI Version](https://img.shields.io/pypi/v/scikit-learn-intelex)](https://pypi.org/project/scikit-learn-intelex/)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/scikit-learn-intelex)](https://anaconda.org/conda-forge/scikit-learn-intelex)
[![python version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
[![scikit-learn supported versions](https://img.shields.io/badge/sklearn-1.0%20%7C%201.2%20%7C%201.3%20%7C%201.4-blue)](https://img.shields.io/badge/sklearn-01.0%20%7C%201.2%20%7C%201.3%20%7C%201.4-blue)

---

## Overview

Intel(R) Extension for Scikit-learn is a **free software AI accelerator** designed to deliver over **10-100X** acceleration to your existing scikit-learn code.
The software acceleration is achieved with vector instructions, AI hardware-specific memory optimizations, threading, and optimizations for all upcoming Intel(R) platforms at launch time.

With Intel(R) Extension for Scikit-learn, you can:

* Speed up training and inference by up to 100x with the equivalent mathematical accuracy
* Benefit from performance improvements across different Intel(R) hardware configurations
* Integrate the extension into your existing Scikit-learn applications without code modifications
* Continue to use the open-source scikit-learn API
* Enable and disable the extension with a couple of lines of code or at the command line

Intel(R) Extension for Scikit-learn is also a part of [Intel(R) AI Tools](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html).

## Acceleration

![](https://raw.githubusercontent.com/intel/scikit-learn-intelex/master/doc/sources/_static/scikit-learn-acceleration.PNG)

[Benchmarks code](https://github.com/IntelPython/scikit-learn_bench)

## Intel(R) Optimizations

- **Enable Intel(R) CPU optimizations**

    ```py
    import numpy as np
    from sklearnex import patch_sklearn
    patch_sklearn()

    from sklearn.cluster import DBSCAN

    X = np.array([[1., 2.], [2., 2.], [2., 3.],
                [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
    clustering = DBSCAN(eps=3, min_samples=2).fit(X)
    ```

- **Enable Intel(R) GPU optimizations**

    ```py
    import numpy as np
    import dpctl
    from sklearnex import patch_sklearn, config_context
    patch_sklearn()

    from sklearn.cluster import DBSCAN

    X = np.array([[1., 2.], [2., 2.], [2., 3.],
                [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
    with config_context(target_offload="gpu:0"):
        clustering = DBSCAN(eps=3, min_samples=2).fit(X)
    ```
:eyes: Check out available [notebooks](https://github.com/intel/scikit-learn-intelex/tree/master/examples/notebooks) for more examples.

## Installation

To install Intel(R) Extension for Scikit-learn, run:

```
pip install scikit-learn-intelex
```

See all installation instructions in the [Installation Guide](https://intel.github.io/scikit-learn-intelex/latest/installation.html).

## Integration

The software acceleration is achieved through patching. It means, replacing the stock scikit-learn algorithms with their optimized versions provided by the extension.

The patching only affects [supported algorithms and their parameters](https://intel.github.io/scikit-learn-intelex/latest/algorithms.html).
You can still use not supported ones in your code, the package simply fallbacks into the stock version of scikit-learn.

> **_TIP:_** Enable [verbose mode](https://intel.github.io/scikit-learn-intelex/latest/verbose.html) to see which implementation of the algorithm is currently used.

To patch scikit-learn, you can:
* Use the following command-line flag:
  ```
  python -m sklearnex my_application.py
  ```
* Add the following lines to the script:
  ```
  from sklearnex import patch_sklearn
  patch_sklearn()
  ```

:eyes: Read about [other ways to patch scikit-learn](https://intel.github.io/scikit-learn-intelex/index.html#usage).

## Documentation

* [Quick Start](https://intel.github.io/scikit-learn-intelex/latest/quick-start.html)
* [Documentation and Tutorials](https://intel.github.io/scikit-learn-intelex/latest/index.html)
* [Release Notes](https://github.com/intel/scikit-learn-intelex/releases)
* [Medium Blogs](https://intel.github.io/scikit-learn-intelex/latest/blogs.html)
* [Code of Conduct](https://github.com/intel/scikit-learn-intelex/blob/master/CODE_OF_CONDUCT.md)

### daal4py and oneDAL

The acceleration is achieved through the use of the Intel(R) oneAPI Data Analytics Library (oneDAL). Learn more:
- [About Intel(R) oneAPI Data Analytics Library](https://github.com/oneapi-src/oneDAL)
- [About daal4py](https://github.com/intel/scikit-learn-intelex/tree/main/daal4py)

## Samples & Examples

* [Examples](https://github.com/intel/scikit-learn-intelex/tree/master/examples/notebooks)
* [Samples](https://intel.github.io/scikit-learn-intelex/latest/samples.html)
* [Kaggle Kernels](https://intel.github.io/scikit-learn-intelex/latest/kaggle.html)

## How to Contribute

We welcome community contributions, check our [Contributing Guidelines](https://github.com/intel/scikit-learn-intelex/blob/master/CONTRIBUTING.md) to learn more.

------------------------------------------------------------------------
\* The Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.


# OpenSIEM Logstash Parsing
Logstash Parsing Configurations for Elastisearch SIEM and OpenDistro for Elasticsearch SIEM

## Why this project exists
The overhead of implementing Logstash parsing and applying Elastic Common Schema (ECS) across audit, security, and
system logs can be a large drawback when using Elasticsearch as a SIEM (Security Incident and Event Management). The
Cargill SIEM team has spent significant time on developing quality Logstash parsing processors for many well-known log
vendors and wants to share this work with the community. In addition to Logstash processors, we have also included log
collection programs for API-based log collection, as well as the setup scripts used to generate our [pipeline-to-pipeline
architecture](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html).

## Quick start Instructions
Follow [GETTING_STARTED.md](./GETTING_STARTED.md) to get started. For detailed info on the architecture and working see README in [build_scripts](./build_scripts)

## Contributions
We welcome and encourage individual contributions to this repo. Please see the Contribution.md guide in the root of the repo.
Please note that we reserve the right to close pull requests or issues that appear to be out of scope for our project, or
for other reasons not specified.

## Questions, Comments & Expected Level of Attention
Please create an issue and someone will **try** to respond to your issue within 5 business days. However, it should be
noted that while we will try revisit the repository semi-regularly, we are not held beholden to this response time (life happens).
We welcome other individuals' answers and input as well.

## Licensing
Apache-2.0


# ansible-navigator

[//]: # (DO-NOT-REMOVE-docs-intro-START)

A text-based user interface (TUI) for Ansible.

A demo of the interface can be found [on YouTube][YT demo].

[YT demo]: https://www.youtube.com/watch?v=J9PBKi8ydi4

[//]: # (DO-NOT-REMOVE-docs-intro-END)

## Contributing

Any kind of contribution to this project is very welcome and appreciated,
whether it is a documentation improvement, [bug report][issue],
[pull request][pull request] review, or a patch.

See the [Contributing guidelines][contributing guidelines] for details.

[issue]:https://github.com/ansible/ansible-navigator/issues
[pull request]:https://github.com/ansible/ansible-navigator/pulls
[contributing guidelines]:
https://ansible.readthedocs.io/projects/navigator/contributing/guidelines/

## Quick start

### Installing

Getting started with ansible-navigator is as simple as:

```
pip3 install 'ansible-navigator[ansible-core]'
ansible-navigator --help
```

(Users wishing to install within a virtual environment might find the relevant
[Python documentation][Python venv doc] useful.)

By default, ansible-navigator uses a container runtime (`podman` or `docker`,
whichever it finds first) and runs Ansible within an execution environment
(a pre-built container image which includes [ansible-core] along with a set
of Ansible collections.)

This default behavior can be disabled by starting ansible-navigator with
`--execution-environment false`. In this case, Ansible and any collections
needed must be installed manually on the system.

[ansible-core]: https://docs.ansible.com/ansible-core/devel
[Python venv doc]: https://docs.python.org/3/library/venv.html

Additional `Linux`, `macOS` and `Windows with WSL2` installation
instructions are available in the [Installation guide].

[Installation guide]:
https://ansible-navigator.readthedocs.io/installation/

## Welcome

When running `ansible-navigator` with no arguments, you will be presented with
the *welcome page*. From this page, you can run playbooks, browse collections,
explore inventories, read Ansible documentation, and more.

A full list of key bindings can be viewed by typing `:help`.

## Output modes

There are two modes in which ansible-navigator can be run:

* The **interactive** mode, which provides a curses-based user interface and
  allows you to "zoom in" on data in real time, filter it, and navigate between
  various Ansible components; and
* The **stdout** mode, which does *not* use curses, and simply returns the
  output to the terminal's standard output stream, as Ansible's commands
  would.

The **interactive** mode is the default and this default can be overwritten by
passing `--mode stdout` (`-m stdout`) or setting `mode` in
[configuration][settings documentation].

[settings documentation]: https://ansible-navigator.readthedocs.io/settings/

## Example commands

All of ansible-navigator's features can be accessed from the *welcome page*
described above, but as a shortcut, commands can also be provided directly as
command-line arguments.

Some examples:

* Review and explore available collections: `ansible-navigator collections`
* Review and explore current Ansible configuration: `ansible-navigator config`
* Review and explore Ansible documentation:
  `ansible-navigator doc ansible.netcommon.cli_command`
* Review execution environment images available locally:
  `ansible-navigator images`
* Review and explore an inventory:
  `ansible-navigator inventory -i inventory.yaml`
* Run and explore a playbook:
  `ansible-navigator run site.yaml -i inventory.yaml`

Or using the **stdout** mode described above:

* Show the current Ansible configuration:
  `ansible-navigator config dump -m stdout`
* Show documentation: `ansible-navigator doc sudo -t become  -m stdout`

... and so on. A full list of subcommands and their relation to Ansible
commands can be found in the [subcommand documentation].

[subcommand documentation]:
https://ansible-navigator.readthedocs.io/subcommands/

## Configuring ansible-navigator

There are several ways to configure ansible-navigator and users and projects
are free to choose the most convenient method for them. The full hierarchy of
how various configuration sources are applied can be found in the FAQ mentioned
below.

Of note, projects making use of ansible-navigator can include a project-wide
configuration file with the project. If one is not found, ansible-navigator
will look for a user-specific configuration file in the user's home directory.
Details about this can be found in the [settings documentation].

## Frequently Asked Questions (FAQ)

We maintain a [list of common questions][FAQ] which provides a good
resource to check if something is tripping you up. We also encourage additions
to this document for the greater community!

[FAQ]: https://ansible-navigator.readthedocs.io/faq/

## License

ansible-navigator is released under the Apache License version 2. See the
[LICENSE] file for more details.

[LICENSE]: https://github.com/ansible/ansible-navigator/blob/main/LICENSE


# Lake: An Agile Framework for Designing and Automatically Configuring Physical Unified Buffers

Lake is a framework for generating synthesizable memory modules from a high-level behavioral specification and widely-available memory macros. Lake also comprises a library of generalized hardware modules aimed at memory controller designs.

## Install
`git clone github.com/StanfordAHA/lake`

`cd lake && pip install -e .`

## Run a test
To run a test, you can simply generate the verilog and push through your favorite verilog simulator. Alternatively, Lake uses the pytest framework for unit tests of constituent modules. These tests leverage [fault](https://github.com/leonardt/fault) and [verilator](https://www.veripool.org/wiki/verilator) for open source simulation. Tests should run and pass on Linux and MacOS.

## Documentation
Check out the [wiki](https://github.com/StanfordAHA/lake/wiki) of this github repo.


# Hacktoberfest Swag List 2023

![Hacktoberfest logo](docs/img/hf10_horz_fcd_rgb.png)

Hello all you beautiful nerds and welcome to another year of [Hacktoberfest](https://hacktoberfest.com/)! at the [Hacktoberfest Swag List](https://hacktoberfestswaglist.com)

## [Click here to be taken to the 2023 Hacktoberfest Swag List!](https://hacktoberfestswaglist.com/list)

[As the official website states](https://hacktoberfest.com/#prepare-to-hack), Hacktoberfest is a time for everyone to celebrate open-source and come together to make meaningful contributions towards software and organizations you love. Whether this is your first Hacktoberfest or your 9th, everyone can contribute to making open-source software even better!

[Low-code or no-code contributions are encouraged](https://hacktoberfest.com/about/#low-or-non-code), so everyone has a chance, regardless of skill level, to contribute and make open-source projects even better!

Please take a look at the new [Contributing.md](docs/contributing.md) file as the way to add organizations to the list has changed.

---

## Sponsored By

<a href="<https://apwr.dev/3RHDflS">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/img/appwrite-logotype-light.svg">
  <source media="(prefers-color-scheme: light)" srcset="docs/img/appwrite-logotype-dark.svg">

</picture>

[Appwrite](https://apwr.dev/3RHDflS) is a proud sponsor of *all of Hacktoberfest*, as well as the Hacktoberfest Swag List! They understand the importance of connecting developers to open-source projects that could use some help.
Appwrite is a self-hosted backend-as-a-service platform that provides developers with all the core APIs required to build any application.

---

## What is the Hacktoberfest Swag List?

[The Hacktoberfest Swag List site](https://hacktoberfestswaglist.com) and [GitHub repo](https://github.com/crweiner/hacktoberfest-swag-list) are here to showcase all of the amazing companies that go above and beyond by offering their own swag for the holiday. **The goal of this project is to connect contributors to open-source projects that they might not have heard about before and drive traffic towards organizations that celebrate Hacktoberfest and can use a little more help.**

### This website/repo is not about collecting all of the swag you can. It is about the open-source community and highlighting organizations that purposely want to drive traffic to their projects during Hacktoberfest by offering unique swag.

### Swag is an added benefit of helping out these awesome organizations, not the reason for you to contribute in the first place!

Participating in Hacktoberfest requires following the [official values](https://hacktoberfest.com/participation/#values), including [**not spamming** the participating companies](https://hacktoberfest.com/participation/#spam).

---

## [Click here to be taken to the 2023 Hacktoberfest Swag List!](https://hacktoberfestswaglist.com/list)

---

*Disclaimer*: This website is a fan and community made creation. It is not affiliated with [Hacktoberfest](https://hacktoberfest.com/) or any company offering swag.

![Presented by DigitalOcean](docs/img/hf10_icon_fcd_rgb.png)

---

If you're looking for the Swag List from 2018 through 2022 [click here](https://github.com/crweiner/hacktoberfest-swag-list/releases) for the GitHub releases, [click here](https://github.com/crweiner/hacktoberfest-swag-list/tags) for the tags, and see the [2018](https://github.com/crweiner/hacktoberfest-swag-list/tree/2018), [2019](https://github.com/crweiner/hacktoberfest-swag-list/tree/2019), [2020](https://github.com/crweiner/hacktoberfest-swag-list/tree/2020), [2021](https://github.com/crweiner/hacktoberfest-swag-list/tree/2021), and [2022](https://github.com/crweiner/hacktoberfest-swag-list/tree/2022) branches.


# Docker images for Schema Registry

This repo provides build files for [schema-registry](https://www.confluent.io/confluent-schema-registry) Docker images.

## Properties

Properties are inherited from a top-level POM. Properties may be overridden on the command line (`-Ddocker.registry=testing.example.com:8080/`), or in a subproject's POM.

- *docker.skip-build*: (Optional) Set to `false` to include Docker images as part of build. Default is 'false'.
- *docker.skip-test*: (Optional) Set to `false` to include Docker image integration tests as part of the build. Requires Python 2.7, `tox`. Default is 'true'.
- *docker.registry*: (Optional) Specify a registry other than `placeholder/`. Used as `DOCKER_REGISTRY` during `docker build` and testing. Trailing `/` is required. Defaults to `placeholder/`.
- *docker.tag*: (Optional) Tag for built images. Used as `DOCKER_TAG` during `docker build` and testing. Defaults to the value of `project.version`.
- *docker.upstream-registry*: (Optional) Registry to pull base images from. Trailing `/` is required. Used as `DOCKER_UPSTREAM_REGISTRY` during `docker build`. Defaults to the value of `docker.registry`.
- *docker.upstream-tag*: (Optional) Use the given tag when pulling base images. Used as `DOCKER_UPSTREAM_TAG` during `docker build`. Defaults to the value of `docker.tag`.
- *docker.test-registry*: (Optional) Registry to pull test dependency images from. Trailing `/` is required. Used as `DOCKER_TEST_REGISTRY` during testing. Defaults to the value of `docker.upstream-registry`.
- *docker.test-tag*: (Optional) Use the given tag when pulling test dependency images. Used as `DOCKER_TEST_TAG` during testing. Defaults to the value of `docker.upstream-tag`.
- *docker.os_type*: (Optional) Specify which operating system to use as the base image by using the Dockerfile with this extension. Valid values are `ubi8`. Default value is `ubi8`.
- *CONFLUENT_PACKAGES_REPO*: (Required) Specify the location of the Confluent Platform packages repository. Depending on the type of OS for the image you are building you will need to either provide a Debian or RPM repository. For example this is the repository for the 5.4.0 release of the Debian packages: `https://s3-us-west-2.amazonaws.com/confluent-packages-5.4.0/deb/5.4` This is the repository for the 5.4.0 release of the RPM's: `https://s3-us-west-2.amazonaws.com/confluent-packages-5.4.0/rpm/5.4`
- *CONFLUENT_VERSION*: (Required) Specify the full Confluent Platform release version. Example: 5.4.0

## Building

This project uses `maven-assembly-plugin` and `dockerfile-maven-plugin` to build Docker images via Maven.

To build SNAPSHOT images, configure `.m2/settings.xml` for SNAPSHOT dependencies. These must be available at build time.

```
mvn clean package -Pdocker -DskipTests # Build local images
```

## License

Usage of this image is subject to the license terms of the software contained within. Please refer to Confluent's Docker images documentation [reference](https://docs.confluent.io/platform/current/installation/docker/image-reference.html) for further information. The software to extend and build the custom Docker images is available under the Apache 2.0 License.


[//]: # (Setup Github Badges)
[![GitHub](https://img.shields.io/github/license/mgear-dev/mgear4?style=flat-square)](https://github.com/mgear-dev/mgear4/blob/master/LICENSE)

[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/mgear-dev/mgear4/Code%20Tests%20(Python%202)?label=Py2%20Tests&style=flat-square)](https://github.com/mgear-dev/mgear4/actions?query=workflow%3A%22Code+Tests+%28Python+2%29%22)
[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/mgear-dev/mgear4/Code%20Tests%20(Python%203)?label=Py3%20Tests&style=flat-square)](https://github.com/mgear-dev/mgear4/actions?query=workflow%3A%22Code+Tests+%28Python+3%29%22)

![GitHub repo size](https://img.shields.io/github/repo-size/mgear-dev/mgear4?style=flat-square)
[![GitHub All Releases](https://img.shields.io/github/downloads/mgear-dev/mgear4/total?style=flat-square)](https://github.com/mgear-dev/mgear4/tags)
[![GitHub last commit](https://img.shields.io/github/last-commit/mgear-dev/mgear4?style=flat-square)](https://github.com/mgear-dev/mgear4/commits/)
[![GitHub issues](https://img.shields.io/github/issues/mgear-dev/mgear4?style=flat-square)](https://github.com/mgear-dev/mgear4/issues)

# mgear 4
mGear v.4.x.x (python 3 ready)

mGear is a rigging framework for Autodesk Maya. mGear provides a set of convenient modules, tools and c++ solvers to streamline the development of rigging and animation tools.

Originally mGear was design and develope by Jeremie Passerin and Miquel Campos. Currently  mGear is maintained by the mgear-dev organization team.

**MGEAR is under the terms of the MIT License**

### Useful links:

- [mGear Web](http://www.mgear-framework.com/)
- [mGear Forum](http://forum.mgear-framework.com/)
- [mGear Youtube channel](https://www.youtube.com/c/mgearriggingframework)
- [Release Log](https://mgear4.readthedocs.io/en/latest/releaseLog.html)

### [Latest release](https://github.com/mgear-dev/mgear4/releases)

## Documentation
The documentation if auto generated and can be found in here:
https://mgear4.readthedocs.io


# neurodiffeq

[![Downloads](https://static.pepy.tech/badge/neurodiffeq)](https://pepy.tech/project/neurodiffeq)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/eada52ca726e4919923e213b81ee6420)](https://app.codacy.com/gh/odegym/neurodiffeq?utm_source=github.com&utm_medium=referral&utm_content=odegym/neurodiffeq&utm_campaign=Badge_Grade_Settings)
![PyPI](https://img.shields.io/pypi/v/neurodiffeq?color=blueviolet&label=PyPI&logoColor=blueviolet) ![GitHub issues](https://img.shields.io/github/issues/NeuroDiffGym/neurodiffeq?color=green) [![Build Status](https://app.travis-ci.com/NeuroDiffGym/neurodiffeq.svg?branch=master)](https://app.travis-ci.com/NeuroDiffGym/neurodiffeq) [![codecov](https://codecov.io/gh/NeuroDiffGym/neurodiffeq/branch/master/graph/badge.svg)](https://codecov.io/gh/NeuroDiffGym/neurodiffeq) [![Documentation Status](https://readthedocs.org/projects/neurodiffeq/badge/?version=latest)](https://neurodiffeq.readthedocs.io/en/latest/?badge=latest) [![DOI](https://joss.theoj.org/papers/10.21105/joss.01931/status.svg)](https://doi.org/10.21105/joss.01931)

# Citation

```
@article{chen2020neurodiffeq,
  title={NeuroDiffEq: A Python package for solving differential equations with neural networks},
  author={Chen, Feiyu and Sondak, David and Protopapas, Pavlos and Mattheakis, Marios and Liu, Shuheng and Agarwal, Devansh and Di Giovanni, Marco},
  journal={Journal of Open Source Software},
  volume={5},
  number={46},
  pages={1931},
  year={2020}
}
```

------

**🔥🔥🔥Did you know that neurodiffeq supports solution bundles and can be used to solve reverse problems? [See here](#solution-bundle-and-reverse-problems)!**

:mortar_board: **Already familiar with neurodiffeq?** :point_down: **[Jump to FAQs](#faq).**

------

# Introduction

`neurodiffeq` is a package for solving differential equations with neural networks. Differential equations are equations that relate some function with its derivatives. They emerge in various scientific and engineering domains. Traditionally these problems can be solved by numerical methods (e.g. finite difference, finite element). While these methods are effective and adequate, their expressibility is limited by their function representation. It would be interesting if we can compute solutions for differential equations that are continuous and differentiable.

As universal function approximators, artificial neural networks have been shown to have the potential to solve ordinary differential equations (ODEs) and partial differential equations (PDEs) with certain initial/boundary conditions. The aim of `neurodiffeq` is to implement these existing techniques of using ANN to solve differential equations in a way that allow the software to be flexible enough to work on a wide range of user-defined problems.

# Installation

## Using pip

Like most standard libraries, `neurodiffeq` is hosted on [PyPI](https://pypi.org/project/neurodiffeq/). To install the latest stable relesase,

```bash
pip install -U neurodiffeq  # '-U' means update to latest version
```

## Manually

Alternatively, you can install the library manually to get early access to our new features. This is the recommended way for developers who want to contribute to the library.

```bash
git clone https://github.com/NeuroDiffGym/neurodiffeq.git
cd neurodiffeq && pip install -r requirements
pip install .  # To make changes to the library, use `pip install -e .`
pytest tests/  # Run tests. Optional.
```

# Getting Started

We are happy to help you with any questions. In the meantime, you can checkout the [FAQs](#faq).

To view complete tutorials and documentation of `neurodiffeq`, please check [Official Documentation](https://neurodiffeq.readthedocs.io/en/latest/).

In addition to the documentations, we have recently made a quick walkthrough [Demo Video](https://youtu.be/VDLwyFD-sXQ) with [slides](https://drive.google.com/file/d/1XTbwkZ0g7ufzD7lvMB-Cl8s5nh6jKgHk/view?usp=sharing).

## Example Usages

### Imports

```python
from neurodiffeq import diff
from neurodiffeq.solvers import Solver1D, Solver2D
from neurodiffeq.conditions import IVP, DirichletBVP2D
from neurodiffeq.networks import FCNN, SinActv
```

### ODE System Example

Here we solve a non-linear system of two ODEs, known as the [Lotka–Volterra](https://en.wikipedia.org/wiki/Lotka–Volterra_equations) equations. There are two unknown functions (`u` and `v`) and a single independent variable (`t`).

```python
def ode_system(u, v, t):
    return [diff(u,t)-(u-u*v), diff(v,t)-(u*v-v)]

conditions = [IVP(t_0=0.0, u_0=1.5), IVP(t_0=0.0, u_0=1.0)]
nets = [FCNN(actv=SinActv), FCNN(actv=SinActv)]

solver = Solver1D(ode_system, conditions, t_min=0.1, t_max=12.0, nets=nets)
solver.fit(max_epochs=3000)
solution = solver.get_solution()
```

`solution` is a callable object, you can pass in numpy arrays or torch tensors to it like

```python
u, v = solution(t, to_numpy=True)  # t can be np.ndarray or torch.Tensor
```

Plotting `u` and `v` against their analytical solutions yields something like:

![lotka–volterra-solution](resources/lotka–volterra-solution.png)

### PDE System Example

Here we solve a Laplace Equation with Dirichlet boundary conditions on a rectangle. Note that we choose Laplace equation for its simplicity of computing analytical solution. **In practice, you can attempt any nonlinear, chaotic PDEs**, provided you tune the solver well enough.

Solving a 2-D PDE system is quite similar to solving ODEs, except there are *two* variables `x` and `y` for boundary value problems or `x` and `t` for initial boundary value problems, both of which are supported.

```python
def pde_system(u, x, y):
    return [diff(u, x, order=2) + diff(u, y, order=2)]

conditions = [
    DirichletBVP2D(
        x_min=0, x_min_val=lambda y: torch.sin(np.pi*y),
        x_max=1, x_max_val=lambda y: 0,
        y_min=0, y_min_val=lambda x: 0,
        y_max=1, y_max_val=lambda x: 0,
    )
]
nets = [FCNN(n_input_units=2, n_output_units=1, hidden_units=(512,))]

solver = Solver2D(pde_system, conditions, xy_min=(0, 0), xy_max=(1, 1), nets=nets)
solver.fit(max_epochs=2000)
solution = solver.get_solution()
```

The signature of `solution` for a 2D PDE is slightly different from that of an ODE. Again, it takes in either numpy arrays or torch tensors.

```python
u = solution(x, y, to_numpy=True)
```
Evaluating u on `[0,1] × [0,1]` yields the following plots

|                 ANN-Based Solution                  |                    Residual of PDE                           |
| :-------------------------------------------------: | :----------------------------------------------------------: |
| ![laplace-solution](resources/laplace-solution.png) | ![laplace-error](resources/laplace-error.png)                |

### Using a Monitor

A monitor is a tool for visualizing PDE/ODE solutions as well as history of loss and custom metrics during training. Jupyter Notebooks users need to run the `%matplotlib notebook` magic. For Jupyter Lab users, try `%matplotlib widget`.

```python
from neurodiffeq.monitors import Monitor1D
...
monitor = Monitor1D(t_min=0.0, t_max=12.0, check_every=100)
solver.fit(..., callbacks=[monitor.to_callback()])
```

You should see the plots update *every 100 epoch* as well as *on the last epoch*, showing two plots — one for solution visualization on the interval `[0,12]` and the other for loss history (training and validation).

![monitor](resources/monitor.gif)

### Custom Networks

For convenience, we have implemented an `FCNN` – fully-connected neural network, whose hidden units and activation functions can be customized.

```python
from neurodiffeq.networks import FCNN
# Default: n_input_units=1, n_output_units=1, hidden_units=[32, 32], activation=torch.nn.Tanh
net1 = FCNN(n_input_units=..., n_output_units=..., hidden_units=[..., ..., ...], activation=...)
...
nets = [net1, net2, ...]
```

`FCNN` is usually a good starting point. For advanced users, solvers are compatible with any custom `torch.nn.Module`. The only constraints are:

1. The modules takes in a tensor of shape `(None, n_coords)` and the outputs a tensor of shape `(None, 1)`.

2. There must be a total of `n_funcs` modules in `nets` to be passed to `solver = Solver(..., nets=nets)`.

![monitor](resources/nets.png)

*Acutally, `neurodiffeq` has a **single_net** feature that doesn't obey the above rules, which won't be covered here.*

Read the PyTorch [tutorial](https://pytorch.org/docs/stable/notes/modules.html) on building your own network (a.k.a module) architecture.

### Transfer Learning

Transfer learning is easily done by serializing `old_solver.nets` (a list of torch modules) to disk and then loading them and passing to a new solver:

```python
old_solver.fit(max_epochs=...)
# ... dump `old_solver.nets` to disk

# ... load the networks from disk, store them in some `loaded_nets` variable
new_solver = Solver(..., nets=loaded_nets)
new_solver.fit(max_epochs=...)
```

We currently working on wrapper functions to save/load networks and other internal variables of Solvers. In the meantime, you can read the PyTorch [tutorial](https://pytorch.org/tutorials/beginner/saving_loading_models.html) on saving and loading your networks.

### Sampling Strategies

In neurodiffeq, the networks are trained by minimizing loss (ODE/PDE residuals) evaluated on a set of points in the domain. The points are randonly resampled every time. To control the number, distribution, and bounding domain of sampled points, you can specify your own training/valiadation `generator`s.

```python
from neurodiffeq.generators import Generator1D

# Default t_min=0.0, t_max=1.0, method='uniform', noise_std=None
g1 = Generator1D(size=..., t_min=..., t_max=..., method=..., noise_std=...)
g2 = Generator1D(size=..., t_min=..., t_max=..., method=..., noise_std=...)

solver = Solver1D(..., train_generator=g1, valid_generator=g2)
```

Here are  some sample distributions of a `Generator1D`.

|      `Generator1D(8192, 0.0, 1.0, method='uniform')`      | `Generator1D(8192, -1.0, 0.0, method='log-spaced-noisy', noise_std=1e-3)` |
| :-------------------------------------------------------: | :----------------------------------------------------------: |
| ![generator1d-uniform](resources/generator1d-uniform.jpg) | ![generator1d-log-spaced-noisy](resources/generator1d-log-spaced-noisy.jpg) |

Note that when both `train_generator` and `valid_generator` are specified, `t_min` and `t_max` can be omitted in `Solver1D(...)`. In fact, even if you pass `t_min`, `t_max`, `train_generator`, `valid_generator` together, the `t_min` and `t_max` will still be ignored.

#### Combining Generators

Another nice feature of the generators is that you can concatenate them, for example

```python
g1 = Generator2D((16, 16), xy_min=(0, 0), xy_max=(1, 1))
g2 = Generator2D((16, 16), xy_min=(1, 1), xy_max=(2, 2))
g = g1 + g2
```

Here, `g` will be a generator that outputs the combined samples of `g1` and `g2`

|                     `g1`                      |                     `g2`                      |                        `g1 + g2`                        |
| :-------------------------------------------: | :-------------------------------------------: | :-----------------------------------------------------: |
| ![generator2d-1](resources/generator2d-1.jpg) | ![generator2d-2](resources/generator2d-2.jpg) | ![generator2d-concat](resources/generator2d-concat.jpg) |

#### Sampling Higher Dimensions

You can use `Generator2D`, `Generator3D`, etc. for sampling points in higher dimensions. But there's also another way

```python
g1 = Generator1D(1024, 2.0, 3.0, method='uniform')
g2 = Generator1D(1024, 0.1, 1.0, method='log-spaced-noisy', noise_std=0.001)
g = g1 * g2
```

Here, `g` will be a generator which yields 1024 points in a 2-D rectangle `(2,3) × (0.1,1)` every time. The x-coordinates of them are drawn from `(2,3)` using strategy `uniform` and the y-coordinate drawn from `(0.1,1)` using strategy `log-spaced-noisy`.

|                      `g1`                       |                      `g2`                       |                          `g1 * g2`                           |
| :---------------------------------------------: | :---------------------------------------------: | :----------------------------------------------------------: |
| ![generator2d-1](resources/generator-ens-1.jpg) | ![generator2d-2](resources/generator-ens-2.jpg) | ![generator2d-concat](resources/generator-ens-ensembled.jpg) |

# Solution Bundle and Reverse Problems

Sometimes, it is interesting to solve a ***bundle*** of equations at once. For example, you may want to solve differential equations of the form `du/dt + λu = 0` under the initial condition `u(0) = U0`. You may want to solve this for all `λ` and `U0` at once, by treating them as inputs to the neural networks.

One such application is for chemical reactions, where the reaction rate is unknown. Different reaction rates correspond to different solutions, and only one solution matches observed data points. You maybe interested in first solving for a bundle of solutions, and then determining the best reaction rates (aka equation parameters). The second step is known as the ***inverse problem***.

Here's an example of how to do this using `neurodiffeq`:

1. Let's say we have an equation `du/dt + λu = 0` and initial condition `u(0) = U0` where `λ` and `U0` are unknown constants. We also have a set of observations `t_obs` and `u_obs`. We first import `BundleSolver` and `BundleIVP` which is necessary to obtaining a solution bundle:

   ```python
   from neurodiffeq.conditions import BundleIVP
   from neurodiffeq.solvers import BundleSolver1D

   import matplotlib.pyplot as plt
   import numpy as np
   import torch
   from neurodiffeq import diff
   ```

2. We determine the domain of input `t`, as well as the domain of parameters  `λ` and `U0`. We also need to make a decision of the order of the parameters. Namely, which should be the first parameter, and which should be the second. **For the purpose of this demo, we choose `λ` to be the first parameter (index 0), and `U0` to be the second (index 1). It is very important to keep track of the indices of the parameters.**

   ```python
   T_MIN, T_MAX = 0, 1
   LAMBDA_MIN, LAMBDA_MAX = 3, 5  # first parameter,  index = 0
   U0_MIN, U0_MAX = 0.2, 0.6       # second parameter, index = 1
   ```

3. We then define the `conditions` and `solver` as usual, except that we use `BundleIVP` and `BundleSolver1D` instead of `IVP` and `Solver1D`. The interface of these two is very similar to `IVP` and `Solver1D`. You can find out more in the [API reference](https://neurodiffeq.readthedocs.io/en/latest/api.html).

   ```python
   # equation parameters comes after inputs (usually temporal and spatial coordinates)
   diff_eq = lambda u, t, lmd: [diff(u, t) + lmd * u]

   # The keyword argument must be named "u_0" in BundleIVP. If you use anything else, e.g. `y0`, `u0`, etc., it won't work.
   conditions = [
       BundleIVP(t_0=0, u_0=None, bundle_param_lookup={'u_0': 1})  # u_0 has index 1
   ]

   solver = BundleSolver1D(
       ode_system=diff_eq,
       conditions=conditions,
       t_min=T_MIN, t_max=T_MAX,
       theta_min=[LAMBDA_MIN, U0_MIN],  # λ has index 0; u_0 has index 1
       theta_max=[LAMBDA_MAX, U0_MAX],  # λ has index 0; u_0 has index 1
       eq_param_index=(0,),             # λ is the only equation parameter, which has index 0
       n_batches_valid=1,
   )
   ```

   Since **`λ` is a parameter in the equation** and **`U0` is a parameter in the initial condition**, we must include `λ` in the `diff_eq` and `U0` in the condition. If a parameter is present in both the equation and the condition, it must be included in both places. **All elements of `conditions` passed to `BundleSovler1D` must be `Bundle*` conditions, even if they don't have parameters.**

4. Now, we can train it and obtain the solution as we normally would.

   ```python
   solver.fit(max_epochs=1000)
   solution = solver.get_solution(best=True)
   ```

   The solution expects three inputs - `t`, `λ` and `U0`. All inputs must have the same shape. For example, if you are interested in fixing `λ=4` and `U0=0.4` and plotting the solution `u` against `t ∈ [0,1]` , you can do the following

   ```python
   t = np.linspace(0, 1)
   lmd = 4 * np.ones_like(t)
   u0 = 0.4 * np.ones_like(t)

   u = solution(t, lmd, u0, to_numpy=True)

   import matplotlib.pyplot as plt
   plt.plot(t, u)
   ```

5. Once you have a bundled `solution`, you can find a set of parameters `(λ, U0)` that matches observed data points `(t_i, u_i)` most closely. This is achieved using simple gradient descent. In the following toy example, we assume there are only three data points `u(0.2) = 0.273`, `u(0.5)=0.129`, and `u(0.8) = 0.0609`. The following is classical PyTorch workflow.

   ```python
   # observed data points
   t_obs = torch.tensor([0.2, 0.5, 0.8]).reshape(-1, 1)
   u_obs = torch.tensor([0.273, 0.129, 0.0609]).reshape(-1, 1)

   # random intialization of λ and U0; keep track of their gradient
   lmd_tensor = torch.rand(1) * (LAMBDA_MAX - LAMBDA_MIN) + LAMBDA_MIN
   u0_tensor = torch.rand(1) * (U0_MAX - U0_MIN) + U0_MIN
   adam = torch.optim.Adam([lmd_tensor.requires_grad_(True), u0_tensor.requires_grad_(True)], lr=1e-2)

   # run gradient descent for 10000 epochs
   for _ in range(10000):
       output = solution(t_obs, lmd_tensor * torch.ones_like(t_obs), u0_tensor * torch.ones_like(t_obs))
       loss = ((output - u_obs) ** 2).mean()
       loss.backward()
       adam.step()
       adam.zero_grad()

   print(f"λ = {lmd_tensor.item()}, U0={u0_tensor.item()}, loss = {loss.item()}")
   ```

# FAQ

#### Q: How to use GPU for training?

Simple. When importing neurodiffeq, the library automatically detects if CUDA is available on your machine. Since the library is based on PyTorch, it will set default tensor type to `torch.cuda.DoubleTensor` for if a compatible GPU device is found.

#### Q: How to use pretrained nets?

Refer to Sections [Custom Networks](#custom-networks) and [Transfer Learning](#transfer-learning).

#### Q: How to change the learning rate?

The standard PyTorch way.

1. Build your networks as explained in [Custom Networks](#custom-networks): `nets = [FCNN(), FCN(), ...]`

2. Instantiate a custom optimizer and pass all parameters of these networks to it

   ```python
   parameters = [p for net in nets for p in net.parameters()]  # list of paramters of all networks
   MY_LEARNING_RATE = 5e-3
   optimizer = torch.optim.Adam(parameters, lr=MY_LEARNING_RATE, ...)
   ```

3. Pass BOTH your `nets ` and your `optimizer` to the solver: `solver = Solver1D(..., nets=nets, optimizer=optimizer)`

#### Q: I got a bad solution.

Unlike traditional numerial methods (FEM, FVM, etc.), the NN-based solution requires some hypertuning. The library offers the utmost flexibility to try any combination of hyperparameters.

- To use a different network architecture, you can pass in your custom `torch.nn.Module`s.
- To use a different optimizer, you can pass in your own optimizer to `solver = Solver(..., optimizer=my_optim)`.
- To use a different sampling distribution, you can use [built-in generators](https://neurodiffeq.readthedocs.io/en/latest/api.html#module-neurodiffeq.generators) or write your own generators from scratch.
- To use a different sampling size, you can tweak the generators or change `solver = Solver(..., n_batches_train)`.
- To dynamically change hyperparameters during training, checkout our [callbacks](https://neurodiffeq.readthedocs.io/en/latest/api.html#module-neurodiffeq.callbacks) feature.

#### Q: Any rules of thumbs?

- Don't use `ReLU` for activation, because its second-order derivative is identically 0.
- Re-scale your PDE/ODE in dimensionless form, preferably make everything range in `[0,1]`. Working with a domain like `[0,1000000]` is prone to failure because **a)** PyTorch initializes the modules weights to be relatively small and **b)** most activation functions (like Sigmoid, Tanh, Swish) are most nonlinear near 0.
- If your PDE/ODE is too complicated, consider trying curriculum learning. Start training your networks on a smaller domain, and then gradually expand until the whole domain is covered.

# Contributing

Everyone is welcome to contribute to this project.

When contributing to this repository, we consider the following process:

1. Open an issue to discuss the change you are planning to make.
2. Go through [Contribution Guidelines](CONTRIBUTING.md).
3. Make the change on a forked repository and update the README.md if changes are made to the interface.
4. Open a pull request.


[![](https://img.shields.io/badge/Community%20Extension-An%20open%20source%20community%20maintained%20project-FF4700)](https://github.com/camunda-community-hub/community)
[![](https://img.shields.io/badge/Lifecycle-Stable-brightgreen)](https://github.com/Camunda-Community-Hub/community/blob/main/extension-lifecycle.md#stable-)
![Compatible with: Camunda Platform 8](https://img.shields.io/badge/Compatible%20with-Camunda%20Platform%208-0072Ce)

[![Coverage Status](https://coveralls.io/repos/github/JonatanMartens/pyzeebe/badge.svg?branch=master)](https://coveralls.io/github/JonatanMartens/pyzeebe?branch=master)
![Test pyzeebe](https://github.com/camunda-community-hub/pyzeebe/workflows/Test%20pyzeebe/badge.svg)
![Integration test pyzeebe](https://github.com/camunda-community-hub/pyzeebe/workflows/Integration%20test%20pyzeebe/badge.svg)

![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/camunda-community-hub/pyzeebe)
![PyPI](https://img.shields.io/pypi/v/pyzeebe)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyzeebe)

[![Documentation Status](https://readthedocs.org/projects/pyzeebe/badge/?version=latest)](https://pyzeebe.readthedocs.io/en/latest/?badge=stable)

# Pyzeebe

pyzeebe is a python grpc client for Zeebe.

Zeebe version support:

| Pyzeebe version | Tested Zeebe versions  |
| :-------------: | ---------------------- |
|      4.x.x      | 1.3, 8.1, 8.2, 8.3, 8.4 |
|      3.x.x      | 1.0.0                  |
|      2.x.x      | 0.23, 0.24, 0.25, 0.26 |
|      1.x.x      | 0.23, 0.24             |

## Getting Started

To install:

`pip install pyzeebe`

For full documentation please visit: https://pyzeebe.readthedocs.io/en/stable/

## Usage

### Worker

The `ZeebeWorker` class gets jobs from the gateway and runs them.

```python
import asyncio

from pyzeebe import ZeebeWorker, Job, create_insecure_channel

channel = create_insecure_channel(hostname="localhost", port=26500) # Create grpc channel
worker = ZeebeWorker(channel) # Create a zeebe worker

async def on_error(exception: Exception, job: Job):
    """
    on_error will be called when the task fails
    """
    print(exception)
    await job.set_error_status(f"Failed to handle job {job}. Error: {str(exception)}")

@worker.task(task_type="example", exception_handler=on_error)
def example_task(input: str) -> dict:
    return {"output": f"Hello world, {input}!"}

@worker.task(task_type="example2", exception_handler=on_error)
async def another_example_task(name: str) -> dict: # Tasks can also be async
    return {"output": f"Hello world, {name} from async task!"}

loop = asyncio.get_running_loop()
loop.run_until_complete(worker.work()) # Now every time that a task with type `example` or `example2` is called, the corresponding function will be called
```

Stop a worker:

```python
await zeebe_worker.stop() # Stops worker after all running jobs have been completed
```

### Client

```python
from pyzeebe import ZeebeClient, create_insecure_channel

# Create a zeebe client
channel = create_insecure_channel(hostname="localhost", port=26500)
zeebe_client = ZeebeClient(channel)

# Run a Zeebe process instance
process_instance_key = await zeebe_client.run_process(bpmn_process_id="My zeebe process", variables={})

# Run a process and receive the result
process_instance_key, process_result = await zeebe_client.run_process_with_result(
    bpmn_process_id="My zeebe process",
    timeout=10000
)

# Deploy a BPMN process definition
await zeebe_client.deploy_process("process.bpmn")

# Cancel a running process
await zeebe_client.cancel_process_instance(process_instance_key=12345)

# Publish message
await zeebe_client.publish_message(name="message_name", correlation_key="some_id")

```

## Tests

Use the package manager [pip](https://pip.pypa.io/en/stable/) to install pyzeebe

`pytest tests/unit`

## Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

## Versioning

We use [SemVer](semver.org) for versioning. For the versions available, see the tags on this repository.

In order to bump the current version run:

```shell
$ bump2version <part>
```

where part is the part that will be bumped (major/minor/patch/rc).

This will bump the version in all relevant files as well as create a git commit.

## License

We use the MIT license, see [LICENSE.md](LICENSE.md) for details