[](https://img.shields.io/github/languages/top/vEnhance/otis-web)
[](https://github.com/vEnhance/otis-web/blob/main/LICENSE.txt)
[](https://img.shields.io/github/last-commit/vEnhance/otis-web)

[](https://github.com/vEnhance/otis-web/actions)
[](https://github.com/vEnhance/otis-web/actions)

[](https://www.djlint.com)
[](https://github.com/psf/black)
[](https://github.com/Microsoft/pyright)
[](https://github.com/astral-sh/ruff)
[](https://coveralls.io/github/vEnhance/otis-web?branch=main)

# OTIS-WEB

These are the source files for the internal website
that I use to teach my course [OTIS](https://web.evanchen.cc/otis.html).
It is pretty standard Django, so you should be able to spin
up a local server on a standard Unix environment.

## Installation instructions

Video tutorial: https://youtu.be/W27XcqeXp20

### Standard Linux environment

If you're just trying to get a local copy of the OTIS-WEB code
and don't plan to submit any code back, then skip steps 0 and 4,
and replace `YOUR_USERNAME` in step 2 with `vEnhance`.

0. Create an account on GitHub if you haven't already, and
   [fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo).
1. Install the following standard tools: `python` (version 3.10), `pip`, `git`.
   (For each tool, search "[name of thing to install] [your OS]" on the web.)
2. Clone this repository using the command
   `git clone https://github.com/YOUR_USERNAME/otis-web`.
3. Run `git submodule update --init --recursive`
   in order to pull the `evans_django_tools` submodule.
4. Type `git checkout -b feature` to checkout a new branch named `feature`;
   this will make your life a bit easier than working on `main`.
   If you already know what feature you're trying to implement,
   you can use that, e.g. `git checkout -b add-bunny-pictures`.
5. [Install Poetry](https://python-poetry.org/docs/).
   This enables you to use the `poetry` command;
   verify this by checking `poetry --help` produces a help message.
6. Once you have `poetry`, hit `poetry install`.
   This will automatically create a
   Python virtual environment and install stuff inside it.
   (If you are an expert familiar with Python virtual environments
   and want to use your own rather than Poetry's auto-created one,
   then activate said environment before running `poetry install`.)

   On some systems, `poetry install` could fail when trying to install
   `mysqlclient`. You will have to separately install `pkg-config`
   [mysql](https://github.com/PyMySQL/mysqlclient#install) in that case;
   see that link for instructions, under the "Install" section.

7. If it isn't already activated,
   [activate the Python virtual environment][activate]
   that was created in the previous step.
   The easiest way to do so is to run `poetry shell`.

   (_Note_: you have to do this step every time you start working on the
   project. That is, always run `poetry shell` before doing any work, or
   for experts, activate the virtual environment using your preferred method.)

8. If everything is working, `python manage.py check` should
   run with no errors.
9. Run `python manage.py migrate` to create the local database.
10. Run `python manage.py createsuperuser` to create an admin user.
11. Run `python manage.py runserver`.
    The command will spin up a continuously running server.
12. Point your browser to `http://127.0.0.1:8000`.
    You should be able to log in with the user you defined in step 9.
13. The website is functional now, but it is a bit bare-bones.
    To populate it with some test data, use `http://127.0.0.1:8000/admin`
    or run `./fixtures/load-all.sh`.

[activate]: https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment

Optional steps:

- If you need to set up environment variables,
  copy `env` to `.env` and uncomment the relevant lines.
- If you want to test the Stripe stuff, a few more steps are needed.
  Briefly: install the Stripe CLI.
  Create some API keys and `stripe login`.
  Add these API keys to `.env` (the three `STRIPE_*` variables).
  Then run `stripe listen --forward-to localhost:8000/payments/webhook/`.

[venv]: https://djangocentral.com/how-to-a-create-virtual-environment-for-python/

### Using Docker

0. Follow steps 0 - 3 from the above tutorial.
1. Make sure to install [Docker](https://www.docker.com/) and make sure you
   download a version compatible with your computer.
2. Set up the 3 environment variables at the bottom of `env` by copying them to
   `.env` and uncommenting them. Note that you should not have any spaces (it is
   fine to leave the values as it is, but if you want to change anything, just
   make sure there is no whitespace surrounding the `=`).
3. Run `docker compose build`. Wait for it to finish.
4. To start the server, run `docker compose up -d`. To execute a command inside
   the container, run `docker exec -it otis-web /bin/bash`. To stop the server,
   run `docker compose down`.
5. Follow steps 11 - 12 from above.

Note: You may need to delete db.sqlite3 if you're not getting desired results,
as it serves as a cache. At this point, spinning up 2 separate containers with
separate data stores is not supported.

## Feature requests or bug reports

Submit an [issue on GitHub](https://github.com/vEnhance/otis-web/issues).

## Pull requests

For OTIS students: pull requests welcome!
If you think the website can be improved in some way, feel free to implement it.
See [contributing guidelines](CONTRIBUTING.md).
It's OK if you don't have much code experience; I'm willing to guide you along.

`526561645265616452656164`


Xopt
====

| **`Documentation`**                                                                                |
|----------------------------------------------------------------------------------------------------------------------------------|
| [![Documentation](https://img.shields.io/badge/Xopt-documentation-blue.svg)](https://ChristopherMayes.github.io/Xopt/index.html) |

| Name | Downloads | Version | Platforms |
| --- | --- | --- | --- |
| [![Conda Recipe](https://img.shields.io/badge/recipe-xopt-green.svg)](https://anaconda.org/conda-forge/xopt) | [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/xopt.svg)](https://anaconda.org/conda-forge/xopt) | [![Conda Version](https://img.shields.io/conda/vn/conda-forge/xopt.svg)](https://anaconda.org/conda-forge/xopt) | [![Conda Platforms](https://img.shields.io/conda/pn/conda-forge/xopt.svg)](https://anaconda.org/conda-forge/xopt) |

Flexible optimization of arbitrary problems in Python.

The goal of this package is to provide advanced algorithmic support for arbitrary
optimization problems (simulations/control systems) with minimal required coding. Users
can easily connect
arbitrary evaluation functions to advanced algorithms with minimal coding with
support for multi-threaded or MPI-enabled execution.

Currenty **Xopt** provides:

- Optimization algorithms:
  - Genetic algorithms
    - `cnsga` Continuous NSGA-II with constraints
  - Bayesian optimization (BO) algorithms:
    - `upper_confidence_bound` BO using Upper Confidence Bound acquisition function
      (w/ or w/o constraints, serial or parallel)
    - `expected_improvement` BO using Expected Improvement acquisition function
      (w/ or w/o constraints, serial or parallel)
    - `mobo` Multi-objective BO (w/ or w/o constraints, serial or parallel)
    - `bayesian_exploration` Autonomous function characterization using Bayesian
      Exploration
    - `mggpo` Parallelized hybrid Multi-Generation Multi-Objective Bayesian
      optimization
    - `multi_fidelity` Multi-fidelity single or multi objective optimization
    - `BAX` Bayesian algorithm execution using virtual measurements
    - BO customization:
      - Trust region BO
      - Heteroskedastic noise specification
      - Multiple acquisition function optimization stratigies
  - `extremum_seeking` Extremum seeking time-dependent optimization
  - `rcds` Robust Conjugate Direction Search (RCDS)
  - `neldermead` Nelder-Mead Simplex
- Sampling algorithms:
  - `random` Uniform random sampling
- Convenient YAML/JSON based input format
- Driver programs:
  - `xopt.mpi.run` Parallel MPI execution using this input format

Xopt does **not** provide:
- your custom simulation or experimental measurement via an `evaluate` function.

Installing Xopt
===============

Installing `xopt` from the `conda-forge` channel can be achieved by adding `conda-forge` to your channels with:

```shell
conda config --add channels conda-forge
```

Once the `conda-forge` channel has been enabled, `xopt` can be installed with:

```shell
conda install xopt
```

It is possible to list all of the versions of `xopt` available on your platform with:

```shell
conda search xopt --channel conda-forge
```

Configuring an Xopt run
===============
Xopt runs can be specified via a YAML file or dictonary input. This requires `generator`, `evaluator`, and `vocs` to be specified, along with optional general options such as `max_evaluations`. An example to run a multi-objective optimiation of a user-defined function `my_function` is:
```yaml
generator:
    name: cnsga
    population_size: 64
    population_file: test.csv
    output_path: .

evaluator:
    function: my_function
    function_kwargs:
      my_arguments: 42

vocs:
    variables:
        x1: [0, 3.14159]
        x2: [0, 3.14159]
    objectives:
        y1: MINIMIZE
        y2: MINIMIZE
    constraints:
        c1: [GREATER_THAN, 0]
        c2: [LESS_THAN, 0.5]
    constants: {a: dummy_constant}

max_evaluations: 6400
```

Xopt can also be used through a simple Python interface.
```python
import math

from xopt.vocs import VOCS
from xopt.evaluator import Evaluator
from xopt.generators.bayesian import UpperConfidenceBoundGenerator
from xopt import Xopt

# define variables and function objectives
vocs = VOCS(
    variables={"x": [0, 2 * math.pi]},
    objectives={"f": "MINIMIZE"},
)

# define the function to optimize
def sin_function(input_dict):
    return {"f": math.sin(input_dict["x"])}

# create Xopt evaluator, generator, and Xopt objects
evaluator = Evaluator(function=sin_function)
generator = UpperConfidenceBoundGenerator(vocs=vocs)
X = Xopt(evaluator=evaluator, generator=generator, vocs=vocs)

# call X.random_evaluate() to generate + evaluate 3 initial points
X.random_evaluate(3)

# run optimization for 10 steps
for i in range(10):
    X.step()

# view collected data
print(X.data)
```

Defining an evaluation function
===============
Xopt can interface with arbitrary evaluate functions (defined in Python) with the
following form:
```python
def evaluate(inputs: dict) -> dict:
    """ your code here """
```
Evaluate functions must accept a dictionary object that **at least** has the keys
specified in `variables, constants` and returns a dictionary
containing **at least** the
keys contained in `objectives, constraints`. Extra dictionary keys are tracked and
used in the evaluate function but are not modified by xopt.

Using MPI
===============
Example MPI run, with `xopt.yaml` as the only user-defined file:
```b
mpirun -n 64 python -m mpi4py.futures -m xopt.mpi.run xopt.yaml
```

Citing Xopt
==========
If you use ```Xopt``` for your research, please consider adding the following
citation to your publications.
```
R. Roussel., et al., "Xopt: A simplified framework for optimization of accelerator problems using advanced algorithms",
in Proc. IPAC'23, Venezia.doi:https://doi.org/10.18429/JACoW-14th International Particle Accelerator Conference-THPL164
```

BibTex entry:
```bibtex
@inproceedings{Xopt,
	title        = {Xopt: A simplified framework for optimization of accelerator problems using advanced algorithms},
	author       = {R. Roussel and A. Edelen and A. Bartnik and C. Mayes},
	year         = 2023,
	month        = {05},
	booktitle    = {Proc. IPAC'23},
	publisher    = {JACoW Publishing, Geneva, Switzerland},
	series       = {IPAC'23 - 14th International Particle Accelerator Conference},
	number       = 14,
	pages        = {4796--4799},
	doi          = {doi:10.18429/jacow-ipac2023-thpl164},
	isbn         = {978-3-95450-231-8},
	issn         = {2673-5490},
	url          = {https://indico.jacow.org/event/41/contributions/2556},
	paper        = {THPL164},
	venue        = {Venezia},
	language     = {english}
}
```

Particular versions of Xopt can be cited from [Zenodo](https://zenodo.org/record/7983779)

Developers
==========

Clone this repository with a truncated git history (recommended):
```shell
git clone --depth=1 https://github.com/ChristopherMayes/Xopt.git
```

Or, clone this repository with the full git history (> 970 MB):
```shell
git clone https://github.com/ChristopherMayes/Xopt.git
```

Create an environment `xopt-dev` with all the dependencies:
```shell
conda env create -f environment.yml
```

Install as editable:
```shell
conda activate xopt-dev
pip install --no-dependencies -e .
```

Install pre-commit hooks:
```
pre-commit install
```

The pre-commit hooks perform autoformatting and report style-compliance errors.
* [ufmt](https://pypi.org/project/ufmt/) formats files w.r.t. [black](https://github.com/psf/black) a strict style enforcer, and [μsort](https://usort.readthedocs.io/en/stable/), which sorts imports in Python modules.
* [flake8](https://flake8.pycqa.org/en/latest/) confirms compliance. Occasionally black misses long-line comments/docstrings and they require manual format.

Pre-commit runs the hooks against your files. If the commit fails, correct the reported errors and then re-add the file with `git add my_file.py`.

### VSCode
The source control integration packaged with VSCode requires additional configuration. Git commands are run in the integrated terminal, which does not inherit the Python interpreter configured with the VSCode project thus breaking the pre-commit hooks. The integration terminal can be configured to use the conda Python environment by including a `.env` file in your project repository:

```
#!/usr/bin/bash
source /path/to/xopt-dev/bin/activate
```


# topological_navigation

A topological navigation planning framework


# Green Web Foundation Admin Portal

Welcome to the administration system for the Green Web Foundation.

This is the system visible at https://admin.thegreenwebfoundation.org.

It lets staff working for hosting companies update information about how they power their infrastructure and generates the open datasets we publish each month.

Just here for the datasets? See the [datasets](https://github.com/thegreenwebfoundation/greenwebfoundation-admin/blob/master/docs/working-with-greenweb-datasets.md)

## Documentation

For a comprehensive installation guide and further information about the configuration of this project, we would like to refer you to [our documentation platform](https://greenweb.readthedocs.io/).


[![Logo](https://img.shields.io/badge/OSMF-OpenFreeEnergy-%23002f4a)](https://openfree.energy/)
[![build](https://github.com/OpenFreeEnergy/openfe/actions/workflows/ci.yaml/badge.svg?event=schedule)](https://github.com/OpenFreeEnergy/openfe/actions/workflows/ci.yaml)
[![coverage](https://codecov.io/gh/OpenFreeEnergy/openfe/branch/main/graph/badge.svg)](https://codecov.io/gh/OpenFreeEnergy/openfe)
[![documentation](https://readthedocs.org/projects/openfe/badge/?version=stable)](https://docs.openfree.energy/en/stable/?badge=stable)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8344248.svg)](https://doi.org/10.5281/zenodo.8344248)

# `openfe` - A Python package for executing alchemical free energy calculations.

The `openfe` package is the flagship project of [Open Free Energy](https://openfree.energy),
a pre competitive consortium aiming to provide robust, permissively licensed open source tools for molecular simulation in the drug discovery field.

Using `openfe` you can easily plan and execute alchemical free energy calculations.

See our [website](https://openfree.energy/) for more information on the project,
[try for yourself](https://try.openfree.energy) from the comfort of your browser,
and we have [documentation on using the package](https://docs.openfree.energy/en/latest/index.html).

## License

This library is made available under the [MIT](https://opensource.org/licenses/MIT) open source license.

## Install

### Latest release

The latest release of `openfe` can be installed via `mamba`, `docker`, or a `single file installer`. See [our installation instructions](https://docs.openfree.energy/en/stable/installation.html) for more details.
Dependencies can be installed via conda through:

### Development version

The development version of `openfe` can be installed directly from the `main` branch of this repository.

First install the package dependencies using `mamba`:

```bash
mamba env create -f environment.yml
```

The openfe library can then be installed via:

```
python -m pip install --no-deps .
```

## Authors

The OpenFE development team.

## Acknowledgements

OpenFE is an [Open Molecular Software Foundation](https://omsf.io/) hosted project.


[![Actions Status](https://github.com/bytewax/bytewax/workflows/CI/badge.svg)](https://github.com/bytewax/bytewax/actions)
[![PyPI](https://img.shields.io/pypi/v/bytewax.svg?style=flat-square)](https://pypi.org/project/bytewax/)
[![Bytewax User Guide](https://img.shields.io/badge/user-guide-brightgreen?style=flat-square)](https://docs.bytewax.io/stable/guide/index.html)

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/bytewax/bytewax/assets/53014647/cd47293b-72c9-423c-b010-2c4990206c60" width="350">
  <source media="(prefers-color-scheme: light)" srcset="https://github.com/bytewax/bytewax/assets/53014647/f376c9e8-5bd4-4563-ba40-3df8761b13fc" width="350">

</picture>

## Python Stateful Stream Processing Framework

Bytewax is a Python framework that simplifies event and stream processing. Because Bytewax couples the stream and event processing capabilities of Flink, Spark, and Kafka Streams with the friendly and familiar interface of Python, you can re-use the Python libraries you already know and love. Connect data sources, run stateful transformations and write to various different downstream systems with built-in connectors or existing Python libraries.

### How it all works

Bytewax is a Python framework and Rust distributed processing engine that uses a dataflow computational model to provide parallelizable stream processing and event processing capabilities similar to Flink, Spark, and Kafka Streams. You can use Bytewax for a variety of workloads from moving data à la Kafka Connect style all the way to advanced online machine learning workloads. Bytewax is not limited to streaming applications but excels anywhere that data can be distributed at the input and output.

Bytewax has an accompanying command line interface, [waxctl](https://docs.bytewax.io/stable/guide/deployment/waxctl.html), which supports the deployment of dataflows on cloud servers or Kubernetes. You can download it [here](https://bytewax.io/waxctl).

---

### Getting Started with Bytewax

```sh
pip install bytewax
```

[_Install waxctl_](https://bytewax.io/waxctl)

#### Dataflow, Input and Operators

A Bytewax dataflow is Python code that will represent an input, a series of processing steps, and an output. The inputs could range from a Kafka stream to a WebSocket and the outputs could vary from a data lake to a key-value store.

```python
import json

from bytewax import operators as op
from bytewax.connectors.kafka import operators as kop
from bytewax.dataflow import Dataflow
```

Bytewax has input and output helpers for common input and output data sources but you can also create your own with the [Sink and Source API](https://docs.bytewax.io/stable/guide/advanced-concepts/custom-connectors.html).

At a high-level, the dataflow compute model is one in which a program execution is conceptualized as data flowing through a series of operator-based steps. Operators like `map` and `filter` are the processing primitives of Bytewax. Each of them gives you a “shape” of data transformation, and you give them regular Python functions to customize them to a specific task you need. See the documentation for a list of the [available operators](https://docs.bytewax.io/stable/api/bytewax/bytewax.operators.html)

```python
BROKERS = ["localhost:19092"]
IN_TOPICS = ["in_topic"]
OUT_TOPIC = "out_topic"
ERR_TOPIC = "errors"

def deserialize(kafka_message):
    return json.loads(kafka_message.value)

def anonymize_email(event_data):
    event_data["email"] = "@".join(["******", event_data["email"].split("@")[-1]])
    return event_data

def remove_bytewax(event_data):
    return "bytewax" not in event_data["email"]

flow = Dataflow("kafka_in_out")
stream = kop.input("inp", flow, brokers=BROKERS, topics=IN_TOPICS)
# we can inspect the stream coming from the kafka topic to view the items within on std out for debugging
op.inspect("inspect-oks", stream.oks)
# we can also inspect kafka errors as a separate stream and raise an exception when one is encountered
errs = op.inspect("errors", stream.errs).then(op.raises, "crash-on-err")
deser_msgs = op.map("deserialize", stream.oks, deserialize)
anon_msgs = op.map("anon", deser_msgs, anonymize_email)
filtered_msgs = op.filter("filter_employees", anon_msgs, remove_bytewax)
processed = op.map("map", anon_msgs, lambda m: KafkaSinkMessage(None, json.dumps(m)))
# and finally output the cleaned data to a new topic
kop.output("out1", processed, brokers=BROKERS, topic=OUT_TOPIC)
```

#### Windowing, Reducing and Aggregating

Bytewax is a stateful stream processing framework, which means that some operations remember information across multiple events. Windows and aggregations are also stateful, and can be reconstructed in the event of failure. Bytewax can be configured with different [state recovery mechanisms](https://docs.bytewax.io/stable/api/bytewax/bytewax.recovery.html) to durably persist state in order to recover from failure.

There are multiple stateful operators available like `reduce`, `stateful_map` and `fold_window`. The complete list can be found in the [API documentation for all operators](https://docs.bytewax.io/stable/api/bytewax/bytewax.operators.html). Below we use the `fold_window` operator with a tumbling window based on system time to gather events and calculate the number of times events have occurred on a per-user basis.

```python
from datetime import datetime, timedelta, timezone

from bytewax.dataflow import Dataflow
import bytewax.operators.windowing as win
from bytewax.operators.windowing import EventClock, TumblingWindower
from bytewax.testing import TestingSource

flow = Dataflow("window_eg")

src = [
    {"user_id": "123", "value": 5, "time": "2023-1-1T00:00:00Z"},
    {"user_id": "123", "value": 7, "time": "2023-1-1T00:00:01Z"},
    {"user_id": "123", "value": 2, "time": "2023-1-1T00:00:07Z"},
]
inp = op.input("inp", flow, TestingSource(src))
keyed_inp = op.key_on("keyed_inp", inp, lambda x: x["user_id"])

# This function instructs the event clock on how to retrieve the
# event's datetime from the input.
# Note that the datetime MUST be UTC. If the datetime is using a different
# representation, we would have to convert it here.
def get_event_time(event):
    return datetime.fromisoformat(event["time"])

# Configure the `fold_window` operator to use the event time.
clock = EventClock(get_event_time, wait_for_system_duration=timedelta(seconds=10))

# And a 5 seconds tumbling window
align_to = datetime(2023, 1, 1, tzinfo=timezone.utc)
windower = TumblingWindower(align_to=align_to, length=timedelta(seconds=5))

five_sec_buckets_win_out = win.collect_window(
    "five_sec_buckets", keyed_inp, clock, windower
)

def calc_avg(bucket):
    values = [event["value"] for event in bucket]
    if len(values) > 0:
        return sum(values) / len(values)
    else:
        return None

five_sec_avgs = op.map_value("avg_in_bucket", five_sec_buckets_win_out.down, calc_avg)
```

#### Merges and Joins

Merging or Joining multiple input streams is a common task for stream processing, Bytewax enables different types of joins to facilitate different patters.

##### Merging Streams

Merging streams is like concatenating, there is no logic and the resulting stream will potentially include heterogeneous records.

```python
from bytewax import operators as op

from bytewax.connectors.stdio import StdOutSink
from bytewax.dataflow import Dataflow
from bytewax.testing import TestingSource

flow = Dataflow("merge")

src_1 = [
    {"user_id": "123", "name": "Bumble"},
]
inp1 = op.input("inp1", flow, TestingSource(src_1))

src_2 = [
    {"user_id": "123", "email": "bee@bytewax.com"},
    {"user_id": "456", "email": "hive@bytewax.com"},
]
inp2 = op.input("inp2", flow, TestingSource(src_2))
merged_stream = op.merge("merge", inp1, inp2)
op.inspect("debug", merged_stream)
```

##### Joining Streams

Joining streams is different than merging because it uses logic to join the records in the streams together. The joins in Bytewax can be running or not. A regular join in streaming is more closely related to an inner join in SQL in that the dataflow will emit data downstream from a join when all of the sides of the join have matched on the key.

```python
from bytewax import operators as op

from bytewax.connectors.stdio import StdOutSink
from bytewax.dataflow import Dataflow
from bytewax.testing import TestingSource

flow = Dataflow("join")

src_1 = [
    {"user_id": "123", "name": "Bumble"},
]
inp1 = op.input("inp1", flow, TestingSource(src_1))
keyed_inp_1 = op.key_on("key_stream_1", inp1, lambda x: x["user_id"])
src_2 = [
    {"user_id": "123", "email": "bee@bytewax.com"},
    {"user_id": "456", "email": "hive@bytewax.com"},
]
inp2 = op.input("inp2", flow, TestingSource(src_2))
keyed_inp_2 = op.key_on("key_stream_2", inp2, lambda x: x["user_id"])

merged_stream = op.join("join", keyed_inp_1, keyed_inp_2)
op.inspect("debug", merged_stream)
```

#### Output

Output in Bytewax is described as a sink and these are grouped into [connectors](https://docs.bytewax.io/stable/api/bytewax/bytewax.connectors.html). There are a number of basic connectors in the Bytewax repo to help you during development. In addition to the built-in connectors, it is possible to use the input and output API to build a custom sink and source. There is also a hub for connectors built by the community, partners and Bytewax. Below is an example of a custom connector for Postgres using the psycopg2 library.

% skip: next

```python
import psycopg2

from bytewax import operators as op
from bytewax.outputs import FixedPartitionedSink, StatefulSinkPartition

class PsqlSink(StatefulSinkPartition):
    def __init__(self):
        self.conn = psycopg2.connect("dbname=website user=bytewax")
        self.conn.set_session(autocommit=True)
        self.cur = self.conn.cursor()

    def write_batch(self, values):
        query_string = """
            INSERT INTO events (user_id, data)
            VALUES (%s, %s)
            ON CONFLICT (user_id)
            DO UPDATE SET data = %s;
        """
        self.cur.execute_values(query_string, values)

    def snapshot(self):
        pass

    def close(self):
        self.conn.close()

class PsqlOutput(FixedPartitionedSink):
    def list_parts(self):
        return ["single"]

    def build_part(step_id, for_part, resume_state):
        return PsqlSink()
```

#### Execution

Bytewax dataflows can be executed in a single Python process, or on multiple processes on multiple hosts with multiple worker threads. When processing data in a distributed fashion, Bytewax uses routing keys to ensure your state is updated in a correct way automatically.

```sh
# a single worker locally
python -m bytewax.run my_dataflow:flow

# Start two worker threads in a single process.
python -m bytewax.run my_dataflow -w 2

# Start a process on two separate machines to form a Bytewax cluster.
# Start the first process with two worker threads on `machine_one`.
machine_one$ python -m bytewax.run my_dataflow -w 2 -i0 -a "machine_one:2101;machine_two:2101"

# Then start the second process with three worker threads on `machine_two`.
machine_two$ python -m bytewax.run my_dataflow -w 3 -i1 -a "machine_one:2101;machine_two:2101"
```

It can also be run in a Docker container as described further in the [documentation](https://docs.bytewax.io/stable/guide/deployment/container.html).

#### Kubernetes

The recommended way to run dataflows at scale is to leverage the [kubernetes ecosystem](https://docs.bytewax.io/stable/guide/deployment/waxctl.html). To help manage deployment, we built [waxctl](https://docs.bytewax.io/stable/guide/deployment/waxctl.html), which allows you to easily deploy dataflows that will run at huge scale across multiple compute nodes.

```sh
waxctl df deploy my_dataflow.py --name my-dataflow
```

## Why Bytewax?

At a high level, Bytewax provides a few major benefits:

- The operators in Bytewax are largely “data-parallel”, meaning they can operate on independent parts of the data concurrently.
- Bytewax offers the ability to express higher-level control constructs, like iteration.
- Bytewax allows you to develop and run your code locally, and then easily scale that code to multiple workers or processes without changes.
- Bytewax can be used in both a streaming and batch context
- Ability to leverage the Python ecosystem directly

## Community

[Slack](https://join.slack.com/t/bytewaxcommunity/shared_invite/zt-vkos2f6r-_SeT9pF2~n9ArOaeI3ND2w) is the main forum for communication and discussion.

[GitHub Issues](https://github.com/bytewax/bytewax/issues) is reserved only for actual issues. Please use the community Slack for discussions.

[Code of Conduct](https://github.com/bytewax/bytewax/blob/main/CODE_OF_CONDUCT.md)

## More Examples

For a more complete example, and documentation on the available operators, check out the [User Guide](https://docs.bytewax.io/stable/guide/index.html) and the [/examples](examples) folder.

## License

Bytewax is licensed under the [Apache-2.0](https://opensource.org/licenses/APACHE-2.0) license.

## Contributing

Contributions are welcome! This community and project would not be what it is without the [contributors](https://github.com/bytewax/bytewax/graphs/contributors). All contributions, from bug reports to new features, are welcome and encouraged.

Please view the [Contribution Guide](https://docs.bytewax.io/stable/guide/contributing/contributing.html) for how to get started.

 With ❤ Bytewax


The Evaluation & Testing framework for LLMs & ML models

Control risks of performance, bias and security issues in AI models

  [![GitHub release](https://img.shields.io/github/v/release/Giskard-AI/giskard)](https://github.com/Giskard-AI/giskard/releases)
  [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/Giskard-AI/giskard/blob/main/LICENSE)
  [![CI](https://github.com/Giskard-AI/giskard/actions/workflows/build-python.yml/badge.svg?branch=main)](https://github.com/Giskard-AI/giskard/actions/workflows/build-python.yml?query=branch%3Amain)
  [![Sonar](https://sonarcloud.io/api/project_badges/measure?project=giskard&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=giskard)
  [![Giskard on Discord](https://img.shields.io/discord/939190303397666868?label=Discord)](https://gisk.ar/discord)

   Docs •
   Blog •
  Website •
  Discord

## Install Giskard 🐢
Install the latest version of Giskard from PyPi using pip:
```sh
pip install "giskard[llm]" -U
```
We officially support Python 3.9, 3.10 and 3.11.
## Try in Colab 📙
[Open Colab notebook](https://colab.research.google.com/github/giskard-ai/giskard/blob/main/docs/getting_started/quickstart/quickstart_llm.ipynb)

______________________________________________________________________

Giskard is an open-source Python library that **automatically detects performance, bias & security issues in AI applications**. The library covers LLM-based applications such as RAG agents, all the way to traditional ML models for tabular data.

## Scan: Automatically assess your LLM-based agents for performance, bias & security issues ⤵

Issues detected include:
- Hallucinations
- Harmful content generation
- Prompt injection
- Robustness issues
- Sensitive information disclosure
- Stereotypes & discrimination
- many more...

## RAG Evaluation Toolkit (RAGET): Automatically generate evaluation datasets & evaluate RAG application answers ⤵

If you're testing a RAG application, you can get an even more in-depth assessment using **RAGET**, Giskard's RAG Evaluation Toolkit.

- **RAGET** can generate automatically a list of `question`, `reference_answer` and `reference_context` from the knowledge base of the RAG. You can then use this generated test set to evaluate your RAG agent.
- **RAGET** computes scores *for each component of the RAG agent*. The scores are computed by aggregating the correctness of the agent’s answers on different question types.

  - Here is the list of components evaluated with **RAGET**:
    - `Generator`: the LLM used inside the RAG to generate the answers
    - `Retriever`: fetch relevant documents from the knowledge base according to a user query
    - `Rewriter`: rewrite the user query to make it more relevant to the knowledge base or to account for chat history
    - `Router`: filter the query of the user based on his intentions
    - `Knowledge Base`: the set of documents given to the RAG to generate the answers

Giskard works with any model, in any environment and integrates seamlessly with your favorite tools ⤵

# Contents

- 🤸♀ **[Quickstart](#quickstart)**
    - **1**. 🏗 [Build a LLM agent](#build-a-llm-agent)
    - **2**. 🔎 [Scan your model for issues](#scan-your-model-for-issues)
    - **3**. 🪄 [Automatically generate an evaluation dataset for your RAG applications](#automatically-generate-an-evaluation-dataset-for-your-rag-applications)
- 👋 **[Community](#community)**

🤸♀ Quickstart

1. 🏗 Build a LLM agent

Let's build an agent that answers questions about climate change, based on the 2023 Climate Change Synthesis Report by the IPCC.

Before starting let's install the required libraries:
```sh
pip install langchain tiktoken "pypdf<=3.17.0"
```

```python
from langchain import OpenAI, FAISS, PromptTemplate
from langchain.embeddings import OpenAIEmbeddings
from langchain.document_loaders import PyPDFLoader
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Prepare vector store (FAISS) with IPPC report
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100, add_start_index=True)
loader = PyPDFLoader("https://www.ipcc.ch/report/ar6/syr/downloads/report/IPCC_AR6_SYR_LongerReport.pdf")
db = FAISS.from_documents(loader.load_and_split(text_splitter), OpenAIEmbeddings())

# Prepare QA chain
PROMPT_TEMPLATE = """You are the Climate Assistant, a helpful AI assistant made by Giskard.
Your task is to answer common questions on climate change.
You will be given a question and relevant excerpts from the IPCC Climate Change Synthesis Report (2023).
Please provide short and clear answers based on the provided context. Be polite and helpful.

Context:
{context}

Question:
{question}

Your answer:
"""

llm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0)
prompt = PromptTemplate(template=PROMPT_TEMPLATE, input_variables=["question", "context"])
climate_qa_chain = RetrievalQA.from_llm(llm=llm, retriever=db.as_retriever(), prompt=prompt)
```

2. 🔎 Scan your model for issues

Next, wrap your agent to prepare it for Giskard's scan:

```python
import giskard
import pandas as pd

def model_predict(df: pd.DataFrame):
    """Wraps the LLM call in a simple Python function.

    The function takes a pandas.DataFrame containing the input variables needed
    by your model, and must return a list of the outputs (one for each row).
    """
    return [climate_qa_chain.run({"query": question}) for question in df["question"]]

# Don’t forget to fill the `name` and `description`: they are used by Giskard
# to generate domain-specific tests.
giskard_model = giskard.Model(
    model=model_predict,
    model_type="text_generation",
    name="Climate Change Question Answering",
    description="This model answers any question about climate change based on IPCC reports",
    feature_names=["question"],
)
```

✨✨✨Then run Giskard's magical scan✨✨✨
```python
scan_results = giskard.scan(giskard_model)
```
Once the scan completes, you can display the results directly in your notebook:

```python
display(scan_results)

# Or save it to a file
scan_results.to_html("scan_results.html")
```

*If you're facing issues, check out our [docs](https://docs.giskard.ai/en/stable/open_source/scan/scan_llm/index.html) for more information.*

3. 🪄 Automatically generate an evaluation dataset for your RAG applications

If the scan found issues in your model, you can automatically extract an evaluation dataset based on the issues found:

```python
test_suite = scan_results.generate_test_suite("My first test suite")
```

By default, RAGET automatically generates 6 different question types (these can be selected if needed, see advanced question generation). The total number of questions is divided equally between each question type. To make the question generation more relevant and accurate, you can also provide a description of your agent.

```python

from giskard.rag import generate_testset, KnowledgeBase

# Load your data and initialize the KnowledgeBase
df = pd.read_csv("path/to/your/knowledge_base.csv")

knowledge_base = KnowledgeBase.from_pandas(df, columns=["column_1", "column_2"])

# Generate a testset with 10 questions & answers for each question types (this will take a while)
testset = generate_testset(
    knowledge_base,
    num_questions=60,
    language='en',  # optional, we'll auto detect if not provided
    agent_description="A customer support chatbot for company X", # helps generating better questions
)
```

Depending on how many questions you generate, this can take a while. Once you’re done, you can save this generated test set for future use:

```python
# Save the generated testset
testset.save("my_testset.jsonl")
```
You can easily load it back

```python
from giskard.rag import QATestset

loaded_testset = QATestset.load("my_testset.jsonl")

# Convert it to a pandas dataframe
df = loaded_testset.to_pandas()
```

Here’s an example of a generated question:

| question                               | reference_context                                                                                | reference_answer                                             | metadata                                               |
|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|-------------------------------------------------------|
| For which countries can I track my shipping? | Document 1: We offer free shipping on all orders over $50. For orders below $50, we charge a flat rate of $5.99. We offer shipping services to customers residing in all 50 states of the US, in addition to providing delivery options to Canada and Mexico. Document 2: Once your purchase has been successfully confirmed and shipped, you will receive a confirmation email containing your tracking number. You can simply click on the link provided in the email or visit our website’s order tracking page. | We ship to all 50 states in the US, as well as to Canada and Mexico. We offer tracking for all our shippings. | `{"question_type": "simple", "seed_document_id": 1, "topic": "Shipping policy"}` |

Each row of the test set contains 5 columns:

- `question`: the generated question
- `reference_context`: the context that can be used to answer the question
- `reference_answer`: the answer to the question (generated with GPT-4)
- `conversation_history`: not shown in the table above, contain the history of the conversation with the agent as a list, only relevant for conversational question, otherwise it contains an empty list.
- `metadata`: a dictionary with various metadata about the question, this includes the question_type, seed_document_id the id of the document used to generate the question and the topic of the question

👋 Community

We welcome contributions from the AI community! Read this [guide](./CONTRIBUTING.md) to get started, and join our thriving community on [Discord](https://gisk.ar/discord).

🌟 [Leave us a star](https://github.com/Giskard-AI/giskard), it helps the project to get discovered by others and keeps us motivated to build awesome open-source tools! 🌟

❤ If you find our work useful, please consider [sponsoring us](https://github.com/sponsors/Giskard-AI) on GitHub. With a monthly sponsoring, you can get a sponsor badge, display your company in this readme, and get your bug reports prioritized. We also offer one-time sponsoring if you want us to get involved in a consulting project, run a workshop, or give a talk at your company.


====================================
Pegasus for Single Cell Analysis
====================================

|PyPI| |PyPIDownload| |Conda| |Python| |License| |Docs| |Build|

.. |PyPI| image:: https://img.shields.io/pypi/v/pegasuspy.svg
   :target: https://pypi.org/project/pegasuspy
.. |PyPIDownload| image:: https://pepy.tech/badge/pegasuspy
   :target: https://pepy.tech/project/pegasuspy
.. |Conda| image:: https://img.shields.io/conda/v/bioconda/pegasuspy
   :target: https://anaconda.org/bioconda/pegasuspy
.. |Python| image:: https://img.shields.io/pypi/pyversions/pegasuspy
   :target: https://pypi.org/project/pegasuspy
.. |License| image:: https://img.shields.io/github/license/lilab-bcb/pegasus
   :target: https://github.com/lilab-bcb/pegasus/blob/master/LICENSE
.. |Docs| image:: https://readthedocs.org/projects/pegasus/badge/?version=latest
   :target: https://pegasus.readthedocs.io/
.. |Build| image:: https://github.com/lilab-bcb/pegasus/workflows/Pegasus%20CI/badge.svg
   :target: https://github.com/lilab-bcb/pegasus/actions?query=workflow%3A%22Pegasus+CI%22

Pegasus is a tool for analyzing transcriptomes of millions of single cells. It is a command line tool, a python package and a base for Cloud-based analysis workflows.

`Read documentation <http://pegasus.readthedocs.io>`__


![CG Header](https://raw.githubusercontent.com/chaos-genius/.github/main/images/CG-banner.png)

    Website •
    Docs •
    Blog •
    LinkedIn •
    Community Slack

ML powered analytics engine for outlier detection and root cause analysis

## ✨ What is Chaos Genius?

Chaos Genius is an open source ML powered analytics engine for outlier detection and root cause analysis. Chaos Genius can be used to monitor and analyse high dimensionality business, data and system metrics at scale.

Using Chaos Genius, users can segment large datasets by key performance metrics (e.g. Daily Active Users, Cloud Costs, Failure Rates) and important dimensions (e.g., countryID, DeviceID, ProductID, DayofWeek) across which they want to monitor and analyse the key metrics.

### Use Chaos Genius if you want:
- Multidimensional Drill Downs & Insights
- Anomaly Detection
- Smart Alerting
- Seasonality Detection*
- Automated Root Cause Analysis*
- Forecasting*
- What-If Analysis*

**in Short and Medium-term Roadmap*

### Demo

[A small demo of Chaos Genius](https://chaosgenius-public.s3.amazonaws.com/test-cg-1-small.gif)

## ⚙ Quick Start

```
git clone https://github.com/chaos-genius/chaos_genius

cd chaos_genius

docker-compose up
```

Visit [http://localhost:8080](http://localhost:8080/)

Follow this [Quick Start guide](https://docs.chaosgenius.io/docs/Quick_Start/prereqs) or read our [Documentation](https://docs.chaosgenius.io/docs/introduction) for more details.

## :dizzy: Key Features

### 1. Automated DeepDrills

Generate multidimensional drilldowns to identify the key drivers of change in defined metrics (e.g. Sales) across a large number of high cardinality dimensions (e.g. CountryID, ProductID, BrandID, Device_type).
- Techniques: Statistical Filtering, A* like path based search to deal with combinatorial explosion

![DD](https://raw.githubusercontent.com/chaos-genius/.github/main/images/DeepDrills.png)

### 2. Anomaly Detection

Modular anomaly detection toolkit for monitoring high-dimensional time series with ability to select from different models. Tackle variations caused by seasonality, trends and holidays in the time series data.
- Models: Prophet, EWMA, EWSTD, Neural Prophet, Greykite

![Anomaly](https://raw.githubusercontent.com/chaos-genius/.github/main/images/AnomalyDrillDowns.png)

### 3. Smart Alerts

Actionable alerts with self-learning thresholds. Configurations to setup alert frequency & reporting to combat alert fatigue.
- Channels: Email, Slack

![Alerting](https://raw.githubusercontent.com/chaos-genius/.github/main/images/Alerting-Dark.png)

## :octocat: Community

For any help, discussions and suggestions feel free to reach out to the Chaos Genius team and the community here:

-   [GitHub](https://github.com/chaos-genius/.github) (report bugs, contribute, follow roadmap)

-   [Slack](https://join.slack.com/t/chaosgenius/shared_invite/zt-140042uac-rrm~xbx9o_aydi6PTmp_Mg) (discuss with the community and Chaos Genius team)

-   [Book Office Hours](https://calendly.com/chaosgenius/30min) (set up time with the Chaos Genius team for any questions or help with setup)

-   [Blog](https://chaosgenius.io/blog/) (follow us on latest trends on Data, Machine Learning, Open Source and more)

## 🚦 Roadmap

Our goal is to make Chaos Genius production ready for all organisations irrespective of their data infrasturcture, data sources and scale requirements. With that in mind we have created a [roadmap](https://docs.chaosgenius.io/docs/roadmap/) for Chaos Genius. If you see something missing or wish to make suggestions, please drop us a line on our [Community Slack](https://join.slack.com/t/chaosgenius/shared_invite/zt-140042uac-rrm~xbx9o_aydi6PTmp_Mg) or raise an issue.

## :seedling: Contributing

Want to contribute? Get started with:

-   Show us some love - Give us a :star2:!

-   Submit an issue.

-   Share a part of the documentation that you find difficult to follow.

-   [Translate our Readme](https://github.com/chaos-genius/chaos_genius/blob/main/README.md).

-   Create a pull request. Here's a [list of issues](https://github.com/chaos-genius/chaos_genius/issues) to start with. Please review our [contribution guidelines](https://github.com/chaos-genius/chaos_genius/blob/main/CONTRIBUTING.md) before opening a pull request. Thank you for contributing!

## :heart: Contributors

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

pshrimal21
📆 📖 🤔 🎨

Harshit Surana
💻 🔣 🔬 🐛

Manas Solanki
💻 👀 🔧 🐛

Kartikay Bagla
💻 🚧 🔬

Varun P
💻 🚧 🔬

Keshav Pradeep
💻 🔣 📖

Daj Katal
🔌 📖

Amatullah Sethjiwala
💻 🔣 ⚠

juzarbhori
💻 🎨

Amogh Dhar Diwan
💻 🔣 🐛

Samyak Sarnayak
💻 📦 🐛

Aayush Naik
💻 🐛 📦

Kshitij Agarwal
💻 🔧 🐛

Bhargav S. Kumar
💻 📦 🐛

moghankumar06
💻 🎨

Santhoshkumar1023
💻 🎨

Mansi-Chauhan27
🔌

davidhayter-karhoo
🐛

Marijn van Aerle
🐛

gxu-kangaroo
🐛

RamneekKaur983
💻

arvind-27
🔣

Josh Taylor
🐛

ChartistDev
💻 🎨 🐛 👀

Rajdeep Sharma
💻 👀

balakumar9493
💻 🎨

Ikko Ashimine
💻

rohit sohlot
💻

athul-osmo
🐛

Kumar Shivendu
🐛 🤔

Pratham Sharma
🐛

churchill1973
🤔

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

## 📜 License

Chaos Genius is licensed under the [MIT license](https://github.com/chaos-genius/chaos_genius/blob/main/LICENSE.md).


[![Stars][gh-stars-badge]][gh-stars-link]
[![PyPI][pypi-badge]][pypi-link]
[![PyPIDownloads][pepy-badge]][pepy-link]
[![CondaDownloads][conda-badge]][conda-link]
[![Docs][docs-badge]][docs-link]
[![Build][build-badge]][build-link]
[![Coverage][coverage-badge]][coverage-link]

[scvi-tools] (single-cell variational inference tools) is a package for probabilistic modeling and
analysis of single-cell omics data, built on top of [PyTorch] and [AnnData].

# Analysis of single-cell omics data

scvi-tools is composed of models that perform many analysis tasks across single-cell, multi, and
spatial omics data:

- Dimensionality reduction
- Data integration
- Automated annotation
- Factor analysis
- Doublet detection
- Spatial deconvolution
- and more!

In the [user guide], we provide an overview of each model. All model implementations have a
high-level API that interacts with [Scanpy] and includes standard save/load functions, GPU
acceleration, etc.

# Rapid development of novel probabilistic models

scvi-tools contains the building blocks to develop and deploy novel probablistic models. These
building blocks are powered by popular probabilistic and machine learning frameworks such as
[PyTorch Lightning] and [Pyro]. For an overview of how the scvi-tools package is structured, you
may refer to the [codebase overview] page.

We recommend checking out the [skeleton repository] as a starting point for developing and
deploying new models with scvi-tools.

# Basic installation

For conda,

```bash
conda install scvi-tools -c conda-forge
```

and for pip,

```bash
pip install scvi-tools
```

Please be sure to install a version of [PyTorch] that is compatible with your GPU (if applicable).

# Resources

- Tutorials, API reference, and installation guides are available in the [documentation].
- For discussion of usage, check out our [forum].
- Please use the [issues] to submit bug reports.
- If you'd like to contribute, check out our [contributing guide].
- If you find a model useful for your research, please consider citing the corresponding
    publication.

# Reference

If you use `scvi-tools` in your work, please cite

> **A Python library for probabilistic analysis of single-cell omics data**
>
> Adam Gayoso, Romain Lopez, Galen Xing, Pierre Boyeau, Valeh Valiollah Pour Amiri, Justin Hong,
> Katherine Wu, Michael Jayasuriya, Edouard Mehlman, Maxime Langevin, Yining Liu, Jules Samaran,
> Gabriel Misrachi, Achille Nazaret, Oscar Clivio, Chenling Xu, Tal Ashuach, Mariano Gabitto,
> Mohammad Lotfollahi, Valentine Svensson, Eduardo da Veiga Beltrame, Vitalii Kleshchevnikov,
> Carlos Talavera-López, Lior Pachter, Fabian J. Theis, Aaron Streets, Michael I. Jordan,
> Jeffrey Regier & Nir Yosef
>
> _Nature Biotechnology_ 2022 Feb 07. doi: [10.1038/s41587-021-01206-w](https://doi.org/10.1038/s41587-021-01206-w).

along with the publicaton describing the model used.

You can cite the scverse publication as follows:

> **The scverse project provides a computational ecosystem for single-cell omics data analysis**
>
> Isaac Virshup, Danila Bredikhin, Lukas Heumos, Giovanni Palla, Gregor Sturm, Adam Gayoso,
> Ilia Kats, Mikaela Koutrouli, Scverse Community, Bonnie Berger, Dana Pe’er, Aviv Regev,
> Sarah A. Teichmann, Francesca Finotello, F. Alexander Wolf, Nir Yosef, Oliver Stegle &
> Fabian J. Theis
>
> _Nature Biotechnology_ 2023 Apr 10. doi: [10.1038/s41587-023-01733-8](https://doi.org/10.1038/s41587-023-01733-8).

scvi-tools is part of the scverse project ([website](https://scverse.org),
[governance](https://scverse.org/about/roles)) and is fiscally sponsored by [NumFOCUS]. Please
consider making a tax-deductible [donation] to help the project pay for developer time,
professional services, travel, workshops, and a variety of other needs.

[anndata]: https://anndata.readthedocs.io/en/latest/
[build-badge]: https://github.com/scverse/scvi-tools/actions/workflows/build.yml/badge.svg
[build-link]: https://github.com/scverse/scvi-tools/actions/workflows/build.yml/
[codebase overview]: https://docs.scvi-tools.org/en/stable/user_guide/background/codebase_overview.html
[conda-badge]: https://img.shields.io/conda/dn/conda-forge/scvi-tools?logo=Anaconda
[conda-link]: https://anaconda.org/conda-forge/scvi-tools
[contributing guide]: https://docs.scvi-tools.org/en/stable/contributing/index.html
[coverage-badge]: https://codecov.io/gh/scverse/scvi-tools/branch/main/graph/badge.svg
[coverage-link]: https://codecov.io/gh/scverse/scvi-tools
[docs-badge]: https://readthedocs.org/projects/scvi/badge/?version=latest
[docs-link]: https://scvi.readthedocs.io/en/stable/?badge=stable
[documentation]: https://docs.scvi-tools.org/
[donation]: https://numfocus.org/donate-to-scverse
[forum]: https://discourse.scvi-tools.org
[gh-stars-badge]: https://img.shields.io/github/stars/scverse/scvi-tools?style=flat&logo=GitHub&color=blue
[gh-stars-link]: https://github.com/scverse/scvi-tools/stargazers
[issues]: https://github.com/scverse/scvi-tools/issues
[numfocus]: https://numfocus.org/
[pepy-badge]: https://static.pepy.tech/badge/scvi-tools
[pepy-link]: https://pepy.tech/project/scvi-tools
[pypi-badge]: https://img.shields.io/pypi/v/scvi-tools.svg
[pypi-link]: https://pypi.org/project/scvi-tools
[pyro]: https://pyro.ai/
[pytorch]: https://pytorch.org
[pytorch lightning]: https://lightning.ai/docs/pytorch/stable/
[scanpy]: http://scanpy.readthedocs.io/
[scvi-tools]: https://scvi-tools.org/
[skeleton repository]: https://github.com/scverse/simple-scvi
[user guide]: https://docs.scvi-tools.org/en/stable/user_guide/index.html


==================================================================
                    tmt - Test Management Tool
==================================================================

The ``tmt`` tool provides a user-friendly way to work with tests.
You can comfortably create new tests, safely and easily run tests
across different environments, review test results, debug test
code and enable tests in the CI using a consistent and concise
config.

The python module and command-line tool implement the Metadata
Specification which allows storing all needed test execution data
directly within a git repository. Together with possibility to
reference remote repositories it makes it easy to share test
coverage across projects and distros.

The Flexible Metadata Format ``fmf`` is used to store data in both
human and machine readable way close to the source code. Thanks to
inheritance and elasticity metadata are organized in the structure
efficiently, preventing unnecessary duplication.

See the https://tmt.readthedocs.io web for detailed documentation.


[![Documentation Status](https://readthedocs.org/projects/flask-fa/badge/?version=latest)](https://flask-fa.readthedocs.io)
[![Translation progress](https://jalkhov.github.io/docspro/badge/fa_progress.svg)](https://github.com/Jalkhov/docspro)

# فلسک

## راهنمای مشارکت

### نصب

* روی دکمه [Fork](https://github.com/flaskcwg/flask-docs-fa/fork) برای گرفتن یک فورک از این مخزن کلیک کنید
* فورک خودتان را کلون کنید. ( `{username}` را با نام کاربری خودتان جایگزین کنید )

```
$ git clone https://github.com/{username}/flask-docs-fa
$ cd flask-docs-fa
$ git remote add upstream https://github.com/flaskcwg/flask-docs-fa
```

* یک محیط مجازی ایجاد کنید و پیش نیاز‌ها را نصب کنید:

برای Linux/macOS:

```
$ python3 -m venv env
$ source env/bin/activate
$ python -m pip install --upgrade pip setuptools wheel
$ pip install -r requirements/dev.txt
$ pip install -e .
$ pre-commit install
```

برای Windows:

```
> python -m venv env
> env\Scripts\activate
> python -m pip install --upgrade pip setuptools
> pip install -r .\requirements\dev.txt
> pip install -e .
> pre-commit install
```

### ترجمه

لطفا قبل از شروع ترجمه یک [issue](https://git.io/JStRB) با نام ترجمه بخش درخواستی خودتان ایجاد کنید.

سپس یک برنچ با نام بخشی({name} را با نام برنچ خودتان جایگزین کنید) که می‌خواهید ترجمه کنید ایجاد کنید:

```
$ git fetch upstream
$ git checkout -b {name} upstream/main
```

و تغییرات خود را در اینجا انجام دهید.

#### نرم افزار‌های پیشنهادی برای ترجمه

* [POEdit](https://poedit.net/)
پیشنهاد اصلی ما POEdit است و خیلی از مترجمان از آن استفاده میکنند!

* [lokalize](https://apps.kde.org/lokalize/)
نرم افزاری دیگر از KDE که شاید بخواهید از آن استفاده کنید.

مطمئنا نرم افزار‌های دیگری نیز برای ویرایش فایل های `po` وجود دارد. برخی از آن برنامه ها را می‌توانید از [اینجا](https://alternativeto.net/software/better-po-editor/) بیابید

> __**نکته:**__ استفاده از نرم افزار برای ترجمه الزامی نیست و می‌توانید از یک ویرایشگر متن هم استفاده کنید

#### ابزار‌های پیشنهادی ترجمه

شما میتونید از ابزار‌های زیر برای ترجمه استفاده کنید:

* مترجم گوگل: [Google Translate](https://translate.google.com/)
* مترجم ترگمان: [Targoman](https://targoman.ir/)

#### شروع ترجمه

فایل‌های ترجمه در `docs/locales/fa/LC_MESSAGES` قرار دارد. فایل انتخابی خود با با نرم افزار مورد علاقه خودتان باز کنید و ترجمه را شروع کنید.

اگر از نرم افزار‌ها برای ترجمه نمی‌خواهید استفاده کنید، موارد زیر برای شماست:

* فایل .po را اینگونه ترجمه کنید:

```po
#: ../../index.rst:4
msgid "Welcome to Flask"
msgstr "<اینجا را با ترجمه پر کنید.>"
```

البته دقت کنید که بعضی از موارد نباید ترجمه شوند مثل ``` :doc:`installation` ``` و  در مثال زیر ``` :doc:`quickstart` ```:

```po
#: ../../index.rst:11
msgid ""
"Welcome to Flask's documentation. Get started with :doc:`installation` "
"and then get an overview with the :doc:`quickstart`."
msgstr ""
"اینجا را با ترجمه پر کنید. اینجا را با ترجمه پر کنید. :doc:`installation`"
"اینجا را با ترجمه پر کنید. اینجا را با ترجمه پر کنید. :doc:`quickstart`."
```

این راهنما شاید بتواند کمک کند: [راهنما](https://www.gnu.org/software/trans-coord/manual/web-trans/html_node/PO-Editors.html)

#### اتمام ترجمه

* اگر اولین مترجم هستید، `FIRST AUTHOR <EMAIL@ADDRESS>` را در بالای فایل‌های `.po` با نام و ایمیل خود جایگزین کنید.
* مقدار فیلد «Last-Translator» را در بالای فایل «.po» به‌روزرسانی کنید. (اگر از نرم افزاری مانند poedit استفاده میکنید به اینکار نیازی نیست)
* اگر ترجمه بخشی را تمام کردید در فایل `README.md`، بخش را به عنوان تکمیل شده علامت بزنید به عنوان مثال:

```
- [x] example [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
```

تغییرات را کامیت کنید:

```
$ git add docs/locales/fa/LC_MESSAGES/example.po README.md
$ git commit -m "Translated docs/example.po"
```

##### بررسی و تست

برای ساختن مستندات کار‌های زیر را انجام دهید:

* در Unix/Linux از دستورات زیر استفاده کنید:

```
$ cd docs
$ make dirhtml
$ python -m http.server --directory _build/dirhtml
```

در ویندوز:

```
> cd docs
> .\make.bat dirhtml
> python -m http.server --directory _build\dirhtml
```

سپس در مرورگر خود این آدرس را باز کنید تا تغییرات را مشاهده کنید:

* <http://127.0.0.1:8000>

اگر همه چیز خوب باشد تغییرات خودتان را پوش کنید({name} را با نام برنچ خودتان جایگزین کنید).

```
$ git push origin {name}
```

#### ایجاد یک PR

* به فورک خود بروید. یک اعلانی خواهید دید. روی `Compare & pull request` کلیک کنید تا یک PR ایجاد کنید.
* اطلاعات درخواست شده در PR را تکمیل کنید
* ممنونم! PR شما را بررسی خواهیم کرد.

## فهرست ترجمه

### docs/

* [x] advanced_foreword [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [ ] async-await
* [ ] blueprints
* [ ] changes
* [ ] cli
* [ ] config
* [X] contributing [@mdk84](https://github.com/mdk84) MDK
* [X] debugging [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [ ] design
* [X] errorhandling [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [X] extensiondev [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [x] extensions [@mdk84](https://github.com/mdk84) MDK
* [x] foreword [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [ ] htmlfaq
* [x] index [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [x] installation [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [X] logging [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [X] quickstart [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh
* [ ] reqcontext
* [ ] security
* [X] server [@Abol-010](https://github.com/Abol-010) Abol
* [x] shell [@ja74d](https://github.com/ja74d) Javad Pournosrat
* [ ] signals
* [X] templating [@mmdbalkhi](https://github.com/mmdbalkhi/) komeil parseh
* [X] testing [@mmdbalkhi](https://github.com/mmdbalkhi/) komeil parseh
* [ ] views

### docs/tutorial/ [@mmdbalkhi](https://github.com/mmdbalkhi/) Komeil Parseh

* [x] blog
* [x] database
* [x] deploy
* [x] factory
* [x] index
* [x] install
* [x] layout
* [x] next
* [x] static
* [x] templates
* [X] tests
* [x] views

### docs/deploying/

* [x] asgi [@mdk84](https://github.com/mdk84) MDK
* [ ] index
* [ ] mod_wsgi
* [ ] uwsgi
* [ ] wsgi-standalone

### docs/patterns/ [@mmdbalkhi](https://github.com/mmdbalkhi/)  Komeil Parseh

* [X] appdispatch [@mmdbalkhi](https://github.com/mmdbalkhi/)  Komeil Parseh)
* [X] appfactories
* [ ] caching
* [ ] celery
* [ ] deferredcallbacks
* [ ] distribute
* [ ] fabric
* [ ] favicon
* [ ] fileuploads
* [ ] flashing
* [X] index [@mmdbalkhi](https://github.com/mmdbalkhi/)  Komeil Parseh
* [ ] jquery
* [ ] lazyloading
* [ ] methodoverrides
* [ ] mongoengine
* [X] packages [@mmdbalkhi](https://github.com/mmdbalkhi/)  Komeil Parseh
* [ ] requestchecksum
* [ ] singlepageapplications
* [ ] sqlalchemy
* [ ] sqlite3
* [ ] streaming
* [ ] subclassing
* [ ] templateinheritance
* [ ] urlprocessors [@mmdbalkhi](https://github.com/mmdbalkhi/)  Komeil Parseh
* [ ] viewdecorators
* [ ] wtforms

## docs/api

* [ ] L0~L1000
* [ ] L1000~L1500
* [ ] L1500~L2000
* [ ] L2000~L2500
* [ ] L2500~L3000
* [ ] L3000~L3500
* [ ] L3500~L4000
* [ ] L4000~L4500
* [ ] L4500~L5000
* [ ] L5000~L5500
* [ ] L5500~L6000
* [ ] L6000~L6500


# [dbt native data observability](https://www.elementary-data.com/)

## What is Elementary?

This dbt package is part of Elementary, the dbt-native data observability solution for data and analytics engineers.
Set up in minutes, gain immediate visibility, detect data issues, send actionable alerts, and understand impact and root cause.
Available as self-hosted or Cloud service with premium features.

#### Table of Contents

- [Quick start - dbt package](#quick-start---dbt-package)
- [Get more out of Elementary](#get-more-out-of-elementary-dbt-package)
- [Run results and dbt artifacts](#run-results-and-dbt-artifacts)
- [Data anomaly detection as dbt tests](#data-anomaly-detection-as-dbt-tests)
- [How Elementary works?](#how-elementary-works)
- [Community & Support](#community--support)
- [Contribution](#contributions)

## Quick start - dbt package

1. Add to your `packages.yml`:

```yml packages.yml
packages:
  - package: elementary-data/elementary
    version: 0.15.2
    ## Docs: https://docs.elementary-data.com
```

2. Run `dbt deps`

3. Add to your `dbt_project.yml`:

```yml
models:
  ## elementary models will be created in the schema '<your_schema>_elementary'
  ## for details, see docs: https://docs.elementary-data.com/
  elementary:
    +schema: "elementary"
```

4. Run `dbt run --select elementary`

Check out the [full documentation](https://docs.elementary-data.com/).

## Get more out of Elementary dbt package

Elementary has 3 offerings: This dbt package, Elementary Community (OSS) and Elementary (cloud service).

- **dbt package**
  - For basic data monitoring and dbt artifacts collection, Elementary offers a dbt package. The package adds logging, artifacts uploading, and Elementary tests (anomaly detection and schema) to your project.
- **Elementary Community**
  - An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on [Slack](https://www.elementary-data.com/community) if needed.
- **Elementary Cloud**
  - Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more, [book a demo](https://cal.com/maayansa/elementary-intro-github-package) or [start a trial](https://www.elementary-data.com/signup).

## Run Results and dbt artifacts

The package automatically uploads dbt artifacts and run results to tables in your data warehouse:

Run results tables:

- dbt_run_results
- model_run_results
- snapshot_run_results
- dbt_invocations
- elementary_test_results (all dbt test results)

Metadata tables:

- dbt_models
- dbt_tests
- dbt_sources
- dbt_exposures
- dbt_metrics
- dbt_snapshots

Here you can find [additional details about the tables](https://docs.elementary-data.com/guides/modules-overview/dbt-package).

## Data anomaly detection as dbt tests

Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc.
Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.

**Elementary tests are configured and executed like native tests in your project!**

Example of Elementary test config in `properties.yml`:

```yml
models:
  - name: your_model_name
    config:
      elementary:
        timestamp_column: updated_at
    tests:
      - elementary.table_anomalies
      - elementary.all_columns_anomalies
```

Read about the available [Elementary tests and configuration](https://docs.elementary-data.com/data-tests/introduction).

## How Elementary works?

Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs.

The cloud service or the CLI tool read the data from these tables, send alerts and present the results in the UI.

## Community & Support

- [Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-uehfrq2f-zXeVTtXrjYRbdE_V6xq4Rg) (Talk to us, support, etc.)
- [GitHub issues](https://github.com/elementary-data/elementary/issues) (Bug reports, feature requests)

## Contributions

Thank you :orange_heart: Whether it’s a bug fix, new feature, or additional documentation - we greatly appreciate contributions!

Check out the [contributions guide](https://docs.elementary-data.com/general/contributions) and [open issues](https://github.com/elementary-data/elementary/issues) in the main repo.


# Karrio

-   Website: <https://karrio.io>
-   Documentation: [https://docs.karrio.io/](https://docs.karrio.io/)
-   Forums: [Github Discussions](https://github.com/orgs/karrioapi/discussions)
-   Discord: [Karrio Discord server](https://discord.gg/gS88uE7sEx)
-   Issues: [Issue Tracker](https://github.com/karrioapi/karrio/issues)
-   Blog: [Blog](https://karrio.io/blog)

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/karrioapi/karrio/main/apps/api/karrio/server/static/extra/branding/logo-inverted.svg" height="100px" />

  </picture>

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](./LICENSE)
[![puprlship-tests](https://github.com/karrioapi/karrio/actions/workflows/tests.yml/badge.svg)](https://github.com/karrioapi/karrio/actions/workflows/tests.yml)
[![karrio-build](https://github.com/karrioapi/karrio/actions/workflows/build.yml/badge.svg)](https://github.com/karrioapi/karrio/actions/workflows/build.yml)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/cc2ac4fcb6004bca84e42a90d8acfe41)](https://www.codacy.com/gh/karrioapi/karrio/dashboard?utm_source=github.com&utm_medium=referral&utm_content=karrioapi/karrio&utm_campaign=Badge_Grade)
[![Discord](https://img.shields.io/static/v1?label=Discord&message=chat&color=738adb&logo=discord)](https://discord.gg/gS88uE7sEx)
[![Docker pulls](https://img.shields.io/docker/pulls/karrio/server)](https://hub.docker.com/r/karrio/server)

Karrio is a developer-first shipping integration and logistics automation platform
that helps developers build shipping experiences into their platforms.

**Get up and running in 1 minute with:**

```sh
git clone --depth 1 https://github.com/karrioapi/karrio
cd karrio/docker
docker compose up
```

-   Karrio server accessible at <http://localhost:5002>
-   Karrio dashboard accessible at <http://localhost:3000>

Default Login: `admin@example.com` | `demo`

## Features

- **Headless Shipping**: Access a network of traditional and modern shipping carriers API-first.
- **Extensible**: Build anything with webhooks, APIs and metadata.
- **Multi-carrier SDK**: Use the karrio SDK to integrate with custom carrier APIs.
- **Fulfilment**: Connect carrier accounts, get live rates and purchase shipping labels.
- **Tracking**: Create package trackers, get real-time tracking status and deliver a great shopping experience.
- **Address Validation**: Validate shipping addresses using integrated 3rd party APIs.
- **Cloud**: Optimized for deployments using Docker.
- **Dashboard**: Use the karrio dashboard to orchestrate your logistics operations.

## Get started

### Quick start demo

[A short video (less than 4 minutes)](https://youtu.be/dFHZosJPaLI) that goes over creating your first label and tracking your first package.

### Open-source hobby deploy (Advanced)

You can deploy a hobby instance in one line on Linux with Docker (recommended 4GB memory):

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/karrioapi/karrio/HEAD/bin/deploy-hobby)"
```

## The problem: Shipping integration is still painful

-   Lack of documentation.
-   Multiple and inconsistent APIs.
-   Difficult access to account and API credentials for development.
-   Poor developer experience.

## Our philosophy

We help you integrate with your shipping carriers and improve your fulfilment processes without loosing control.

We believe that the logistics industry can greatly benefit from a unified, open and standardized shipping API that gives you direct access to a network of carriers, control over customers' personal data and **compliance**.

## What are the benefits?

Karrio is the only **platform-focused** open-source shipping platform with label generation, document generation and package tracking API that **you can host on your own infrastructure**.

We are an open-source alternative to expensive in-house carrier integration and multi-carrier Saas APIs. We're designed to be more **developer-friendly**, with the fullset of shipping functionalities without vendor-lockin.

### Shipping for platforms

With Karrio, you can extend your platform with native shipping capabilities. Improve merchants and customers experience on your Marketplace, eCommerce, ERP, WMS, OMS, 3PL and Logistics platform.
With Karrio, 3PLs and large volume shippers regain control over their shipping tech stack and processes.

### Shipping for enterprise

Karrio makes modern shipping accessible to brands, retailers as well as businesses in regulated industries.
Ideal for pharmacy shipping where compliance is a must.

## Support

If you have general questions about Karrio, want to say hello or just follow along, we'd like to invite you to join our [Discord Community](https://discord.gg/gS88uE7sEx).

If you run into any problems or issues, please create a Github issue and we'll try our best to help.

We strive to provide good support through our issue trackers on Github.

## Contributing

See our Docs for instructions on [development setup](https://docs.karrio.io/contributing/development).

We <3 contributions big or small, check out our [guide on how to get started](https://docs.karrio.io/contributing/guidlines).

Not sure where to start? [Send us an email](mailto:dev@karrio.com?subject=Pairing%20session&body=I'd%20like%20to%20do%20a%20pairing%20session!) to chat with a member of our team.

### Contributors

## License

This project uses the [Apache v2 license](LICENSE)

## Sponsors

We are building an open and global shipping platform that makes shipping integration and logistics automation fun and accessible using open source software.
Thanks for sponsoring and backing us.

### Past sponsors

### Become a sponsor


# Switcher Python Integration
[![pypi-version]][11] [![pypi-downloads]][11] [![license-badge]][4]

[![gh-build-status]][7] [![gh-pages-status]][8] [![codecov]][3]

PyPi module integrating with various [Switcher][12] devices.

Check out the [wiki pages][0] for a list of supported devices.

```shell
pip install aioswitcher
```

  Documentation
  Wiki
  Contributing

## Example Usage

  State Bridge

```python
async def print_devices(delay):
    def on_device_found_callback(device):
        # a switcher device will broadcast a state message approximately every 4 seconds
        print(asdict(device))

    async with SwitcherBridge(on_device_found_callback):
        await asyncio.sleep(delay)

# run the bridge for 60 seconds
asyncio.run(print_devices(60))
```

  Power Plug API

  ```python
  async def control_power_plug(device_ip, device_id, device_key) :
      # for connecting to a device we need its id, login key and ip address
      async with SwitcherType1Api(device_ip, device_id, device_key) as api:
          # get the device current state
          await api.get_state()
          # turn the device on
          await api.control_device(Command.ON)
          # turn the device off
          await api.control_device(Command.OFF)
          # set the device name to 'my new name'
          await api.set_device_name("my new name")

  asyncio.run(control_power_plug("111.222.11.22", "ab1c2d", "00"))
  ```

  Water Heater API

  ```python
  async def control_water_heater(device_ip, device_id, device_key) :
      # for connecting to a device we need its id, login key and ip address
      async with SwitcherType1Api(device_ip, device_id, device_key) as api:
          # get the device current state
          await api.get_state()
          # turn the device on for 15 minutes
          await api.control_device(Command.ON, 15)
          # turn the device off
          await api.control_device(Command.OFF)
          # set the device name to 'my new name'
          await api.set_device_name("my new name")
          # configure the device for 02:30 auto shutdown
          await api.set_auto_shutdown(timedelta(hours=2, minutes=30))
          # get the schedules from the device
          await api.get_schedules()
          # delete and existing schedule with id 1
          await api.delete_schedule("1")
          # create a new recurring schedule for 13:00-14:30
          # executing on sunday and friday
          await api.create_schedule("13:00", "14:30", {Days.SUNDAY, Days.FRIDAY})

  asyncio.run(control_water_heater("111.222.11.22", "ab1c2d", "00"))
  ```

  Runner API

  ```python
  async def control_runner(device_ip, device_id, device_key) :
      # for connecting to a device we need its id, login key and ip address
      async with SwitcherType2Api(device_ip, device_id, device_key) as api:
          # get the device current state
          await api.get_shutter_state()
          # open the shutter to 30%
          await api.set_position(30)
          # stop the shutter if currently rolling
          await api.stop()

  asyncio.run(control_runner("111.222.11.22", "ab1c2d", "00"))
  ```

  Breeze API

  ```python
  async def control_breeze(device_ip, device_id, device_key, remote_manager, remote_id) :
      # for connecting to a device we need its id, login key and ip address
      async with SwitcherType2Api(device_ip, device_id, device_key) as api:
          # get the device current state
          await api.get_breeze_state()
          # initialize the Breeze RemoteManager and get the remote
          remote = remote_manager.get_remote(remote_id)
          # prepare a control command that turns on the Breeze
          # set to 24 degree (Celsius) cooling with vertical swing
          # send command to the device
          await api.control_breeze_device(
              remote,
              DeviceState.ON,
              ThermostatMode.COOL,
              24,
              ThermostatFanLevel.MEDIUM,
              ThermostatSwing.ON,
          )

  # create the remote manager outside the context for re-using
  remote_manager = SwitcherBreezeRemoteManager()
  asyncio.run(control_breeze("111.222.11.22", "ab1c2d", "00", remote_manager, "DLK65863"))
  ```

## Command Line Helper Scripts

- [discover_devices.py](https://github.com/TomerFi/aioswitcher/blob/dev/scripts/discover_devices.py) can discover devices and their
  states (can be run by `poetry run discover_devices`).
- [control_device.py](https://github.com/TomerFi/aioswitcher/blob/dev/scripts/control_device.py) can control a device (can be run by `poetry run control_device`).

## Disclaimer

This is **NOT** an official module and it is **NOT** officially supported by the vendor.

That said, thanks are in order to all the people at [Switcher][12] for their cooperation and general support.

## Contributors [![all-contributors]][2]

Thanks goes to these wonderful people ([emoji key][1]):

Aviad Golan
🔣

Dolev Ben Aharon
📖

Fabian Affolter
💻

Itzik Ephraim
💻

Kesav890
📖

Liad Avraham
💻

Marc Mueller
💻

Or Bin
💻

Roeeeee
💻

Shai rod
🔣

Shay Levy
💻 🤔 🚧

YogevBokobza
💻 ⚠ 🚧

Yuval Moran
💻 ⚠

dmatik
📝 🤔 📓

jafar-atili
💻 📖

[0]: https://github.com/TomerFi/aioswitcher/wiki
[1]: https://allcontributors.org/docs/en/emoji-key
[2]: https://allcontributors.org
[3]: https://codecov.io/gh/TomerFi/aioswitcher
[4]: https://github.com/TomerFi/aioswitcher
[7]: https://github.com/TomerFi/aioswitcher/actions/workflows/stage.yml
[8]: https://aioswitcher.tomfi.info/
[11]: https://pypi.org/project/aioswitcher
[12]: https://www.switcher.co.il/

[all-contributors]: https://img.shields.io/github/all-contributors/tomerfi/aioswitcher?color=ee8449&style=flat-square
[codecov]: https://codecov.io/gh/TomerFi/aioswitcher/graph/badge.svg
[gh-build-status]: https://github.com/TomerFi/aioswitcher/actions/workflows/stage.yml/badge.svg
[gh-pages-status]: https://github.com/TomerFi/aioswitcher/actions/workflows/pages.yml/badge.svg
[license-badge]: https://img.shields.io/github/license/tomerfi/aioswitcher
[pypi-downloads]: https://img.shields.io/pypi/dm/aioswitcher.svg?logo=pypi&color=1082C2
[pypi-version]: https://img.shields.io/pypi/v/aioswitcher?logo=pypi


[![PyPI version](https://badge.fury.io/py/ostorlab.svg)](https://badge.fury.io/py/ostorlab)
[![Downloads](https://static.pepy.tech/badge/ostorlab/month)](https://pepy.tech/project/ostorlab)
[![Ostorlab blog](https://img.shields.io/badge/blog-ostorlab%20news-red)](https://blog.ostorlab.co/)
[![Twitter Follow](https://img.shields.io/twitter/follow/ostorlabsec.svg?style=social)](https://twitter.com/ostorlabsec)

# OXO Scan Orchestration Engine

OXO is a security scanning framework built for modularity, scalability and simplicity.

OXO Engine combines specialized tools to work cohesively to find vulnerabilities and perform actions like recon, enumeration, fingerprinting ...

![Scan Run](images/scan_run.gif)

# Requirements

Docker is required to run scans locally. To install docker, please follow these
[instructions](https://docs.docker.com/get-docker/).

# Installing

OXO ships as a Python package on pypi. To install it, simply run the following command if you have `pip` already
installed.

```shell
pip install -U ostorlab
```

# Getting Started

OXO ships with a store that boasts dozens of agents, from network scanning agents like nmap, openvas, nuclei or
tsunami,
web scanner like Zap, web fingerprinting tools like Whatweb and Wappalyzer, DNS brute forcing like Subfinder and Dnsx,
malware file scanning like Virustotal and much more.

To run any of these tools combined, simply run the following command:

> OXO CLI is accessible using the `oxo` command.

```shell
oxo scan run --install --agent agent/ostorlab/nmap --agent agent/ostorlab/openvas --agent agent/ostorlab/tsunami --agent agent/ostorlab/nuclei ip 8.8.8.8
```

or

```shell
oxo scan run --install --agent agent/ostorlab/nmap --agent agent/ostorlab/openvas --agent agent/ostorlab/tsunami --agent agent/ostorlab/nuclei ip 8.8.8.8
```

This command will download and install the following scanning agents:

* `agent/ostorlab/nmap`
* `agent/ostorlab/tsunami`
* `agent/ostorlab/nuclei`
* `agent/ostorlab/openvas`

And will scan the target IP address `8.8.8.8`.

Agents are shipped as standard docker images.

To check the scan status, run:

```shell
oxo scan list
```

Once the scan has completed, to access the scan results, run:

```shell
oxo vulnz list --scan-id <scan-id>
oxo vulnz describe --vuln-id <vuln-id>
```

# Examples

Agents accept argument to tweak their behavior, like setting the default ports for nmap or passing the API key to
Virustotal. Agents composition can also be saved to a YAML file for latter use. See below some examples

* Run a network scan:

```yaml
# file: agent_group.yaml
kind: AgentGroup
description: Grouping of 6 agents.
agents:
  - key: agent/ostorlab/nmap
    args:
      - name: ports
        type: string
        description: List of ports to scan.
        value: '22,443,80'
  - key: agent/ostorlab/tsunami
    args: [ ]
  - key: agent/ostorlab/openvas
    args: [ ]
  - key: agent/ostorlab/nuclei
    args:
      - name: template_urls
        type: array
        description: List of template urls to run. These will be fetched by the agent
          and passed to Nuclei.
        value: ""
      - name: use_default_templates
        type: boolean
        description: use nuclei's default templates to scan.
        value: true
```

```shell
oxo scan run --install -g agent_group.yaml ip 8.8.8.8 8.8.4.4
```

* Run a web scan:

```yaml
# file: agent_group.yaml
kind: AgentGroup
description: Grouping of 4 agents.
agents:
  - key: agent/ostorlab/zap
    args: [ ]
  - key: agent/ostorlab/whatweb
    args: [ ]
```

```shell
oxo scan run --install -g agent_group.yaml domain-name example.com
```

* List all subdomains, resolve their IP addresses and run a network scan:

```yaml
# file: agent_group.yaml
kind: AgentGroup
description: Grouping of 5 agents.
agents:
  - key: agent/ostorlab/nmap
    args: [ ]
    port_mapping: [ ]
  - key: agent/ostorlab/subfinder
    args: [ ]
  - key: agent/ostorlab/dnsx
    args: [ ]
```

```shell
oxo scan run --install -g agent_group.yaml domain-name example.com
```

* Brute force all TLDs of a subdomain and resolve their IP addresses:

```yaml
# file: agent_group.yaml
kind: AgentGroup
description: Grouping of 5 agents.
agents:
  - key: agent/ostorlab/nmap
    args: [ ]
  - key: agent/ostorlab/all_tlds
    args: [ ]
  - key: agent/ostorlab/subfinder
    args: [ ]
  - key: agent/ostorlab/dnsx
    args: [ ]
```

```shell
oxo scan run --install -g agent_group.yaml domain-name example.com
```

* Scanning a list of assets is supported using a Yaml asset definition file:

Inject domain name assets `example.com` and `example_2.com`

```shell
oxo scan run --install -g agent_group.yaml -a assets_group.yaml
```

```yaml
# file: asset_group.yaml
description: Target group definition
kind: targetGroup
name: master_scan
assets:
  domain:
    - name: "example.com"
    - name: "example_2.com"
```

Inject IOS store assets with the bundle_ids `com.example.app` and `com.example_2.app`

```shell
oxo scan run --install -g agent_group.yaml -a assets_group.yaml
```

```yaml
# file: asset_group.yaml
description: Target group definition
kind: targetGroup
name: master_scan
assets:
  iosStore:
    - bundle_id: "com.example.app"
    - bundle_id: "com.example_2.app"
```

Here is a more encompassing example:

```shell
oxo scan run --install -g agent_group.yaml -a assets_group.yaml
```

```yaml
# file: asset_group.yaml
description: Target group definition for the NSA
kind: targetGroup
name: master_scan
assets:
  androidStore:
      - package_name: "com.example.app"
      - package_name: "com.example_2.app"
  androidApkFile:
      - path: /home/organisation/assets/application.apk
      - path: /home/organisation/assets/application.apk
      - url: https://exanple.storage.com/production.apk
      - url: https://exanple.storage.com/dev.apk
  androidAabFile:
      - path: /home/organisation/assets/application.aab
      - path: /home/organisation/assets/application.aab
      - url: https://exanple.storage.com/production.aab
      - url: https://exanple.storage.com/dev.aab
  iosStore:
      - bundle_id: "com.example.app"
      - bundle_id: "com.example_2.app"
  iosFile:
      - path: /home/organisation/assets/application.ipa
      - path: /home/organisation/assets/application.ipa
      - url: https://exanple.storage.com/production.ipa
      - url: https://exanple.storage.com/dev.ipa
  link:
      - url: "https://www.nasa.gov/humans-in-space/artemis/"
        method: "GET"
      - url: "https://www.nasa.gov/specials/artemis/"
        method: "POST"
  domain:
      - name: "example.co"
      - name: "example_2.dev"
  ip:
      - host: "10.21.11.11"
        mask: 30
      - host: 0.1.2.1
```

## The Pitch

Testing for even the most simple vulnerabilities often requires chaining multiple tools. Take for instance scanning
for a Log4J bug, this requires:

* Crawling
* Path brute forcing
* Request Injection Point fuzzing
* Callback interception

Tools will often re-invent the wheel by poorly re-implementing all of these and then add their detection. These often
results in poor detectors as most of these are complex tasks that requires specialized tools.

OXO offers the ease of chaining the specialized tools to focus on perform the required task, offering increased
detection, faster delivery.

To do that, OXO provides on the following:

* A simple, yet powerful SDK to make simple cases effortless while supporting the complex one, like distributed locking,
  QPS limiting, multiple instance parallelization ...
* A battle-tested framework that has been powering Ostorlab Platform for years and used to perform complex dynamic
  analysis setup and demanding static analysis workloads running on multiple machines.
* Performant and scalable design, thanks to the use of message queue with dynamic routing, binary and compact message
  serialisation with protobuf, universal file format using docker image and resilient deployment thanks to docker swarm.
* A store of agents that makes it easy to use and discover tools to add your toolset.
* An automated builder to take the hassle away of building and publishing.
* A GUI to prepare and write down your tool collection setup.
* Focus on documentation, multiple tutorials and upcoming videos and conference presentations.
* A ready to use one-click template repo to get started.

# Assets

OXO supports scanning of multiple asset types, below is the list of currently supported:

| Asset       | Description                                                                        |
|-------------|------------------------------------------------------------------------------------|
| agent       | Run scan for agent. This is used for agents scanning themselves (meta-scanning :). |
| ip          | Run scan for IP address or an IP range .                                           |
| link        | Run scan for web link accepting a URL, method, headers and request body.           |
| file        | Run scan for a generic file.                                                       |
| android-aab | Run scan for an Android .AAB package file.                                         |
| android-apk | Run scan for an Android .APK package file.                                         |
| ios-ipa     | Run scan for iOS .IPA file.                                                        |
| domain-name | Run scan for Domain Name asset with specifying protocol or port.                   |

# The Store

OXO lists all agents on a public store where you can search and also publish your own agents.

![Store](images/store2.gif)

In addition, the store, a graphical agent group builder is also available to compose multiple agents and see how
they would interact with each other.

![Store](images/store.gif)

The builder also helps with generating the agent group YAML file to set special arguments that can be passed to agents
to control their behavior.

![Build](images/agent_group.gif)

# Publish your first Agent

To write your first agent, you can check out a full
tutorial [here](https://oxo.ostorlab.co/tutorials/write_an_agent).

The steps are basically the following:

* Clone a template agent with all files already setup.
* Change the `template_agent.py` file to add your logic.
* Change the `Dockerfile` adding any extra building steps.
* Change the `ostorlab.yaml` adding selectors, documentation, image, license.
* Publish on the store.
* Profit!

Once you have written your agent, you can publish it on the store for others to use and discover it. The store
will handle agent building and will automatically pick up new releases from your git repo.

![Build](images/build.gif)

## Ideas for Agents to build

Implementation of popular tools like:

* ~~[semgrep](https://github.com/returntocorp/semgrep) for source code scanning.~~
* [nbtscan](http://www.unixwiz.net/tools/nbtscan.html): Scans for open NETBIOS nameservers on your target’s network.
* [onesixtyone](https://github.com/trailofbits/onesixtyone): Fast scanner to find publicly exposed SNMP services.
* [Retire.js](http://retirejs.github.io/retire.js/): Scanner detecting the use of JavaScript libraries with known
  vulnerabilities.
* ~~[snallygaster](https://github.com/hannob/snallygaster): Finds file leaks and other security problems on HTTP servers.~~
* [testssl.sh](https://testssl.sh/): Identify various TLS/SSL weaknesses, including Heartbleed, CRIME and ROBOT.
* ~~[TruffleHog](https://github.com/trufflesecurity/truffleHog): Searches through git repositories for high entropy~~
  strings and secrets, digging deep into commit history.
* [cve-bin-tool](https://github.com/intel/cve-bin-tool): Scan binaries for vulnerable components.
* [XSStrike](https://github.com/s0md3v/XSStrike): XSS web vulnerability scanner with generative payload.
* ~~[Subjack](https://github.com/haccer/subjack): Subdomain takeover scanning tool.~~
* [DnsReaper](https://github.com/punk-security/dnsReaper): Subdomain takeover scanning tool.


======================
Resolwe Bioinformatics
======================

|build| |coverage| |docs| |pypi_version| |pypi_pyversions|

.. |build| image:: https://public.ci.genialis.io/buildStatus/icon/resolwe-bio/master
    :target: https://ci.genialis.io/blue/organizations/jenkins/resolwe-bio/activity
    :alt: Build Status

.. |coverage| image:: https://img.shields.io/codecov/c/github/genialis/resolwe-bio/master.svg
    :target: http://codecov.io/github/genialis/resolwe-bio?branch=master
    :alt: Coverage Status

.. |docs| image:: https://readthedocs.org/projects/resolwe-bio/badge/?version=latest
    :target: http://resolwe-bio.readthedocs.io/
    :alt: Documentation Status

.. |pypi_version| image:: https://img.shields.io/pypi/v/resolwe-bio.svg
    :target: https://pypi.python.org/pypi/resolwe-bio
    :alt: Version on PyPI

.. |pypi_pyversions| image:: https://img.shields.io/pypi/pyversions/resolwe-bio.svg
    :target: https://pypi.python.org/pypi/resolwe-bio
    :alt: Supported Python versions

.. |pypi_downloads| image:: https://img.shields.io/pypi/dm/resolwe-bio.svg
    :target: https://pypi.python.org/pypi/resolwe-bio
    :alt: Number of downloads from PyPI

Bioinformatics pipelines for the Resolwe_ dataflow package for `Django
framework`_.

.. _Resolwe: https://github.com/genialis/resolwe
.. _Django framework: https://www.djangoproject.com/

Docs & Help
===========

Read about getting started and how to write `processes` in the documentation_.

To chat with developers or ask for help, join us on Slack_.

.. _documentation: http://resolwe-bio.readthedocs.io/
.. _Slack: http://resolwe.slack.com/

Install
=======

Prerequisites
-------------

Make sure you have Python_ 3.6 installed on your system. If you don't have it
yet, follow `these instructions
<https://docs.python.org/3/using/index.html>`__.

Resolwe requires PostgreSQL_ (9.4+). Many Linux distributions already include
the required version of PostgreSQL (e.g. Fedora 22+, Debian 8+, Ubuntu 15.04+)
and you can simply install it via distribution's package manager.
Otherwise, follow `these instructions
<https://wiki.postgresql.org/wiki/Detailed_installation_guides>`__.

Additionally, installing some (indirect) dependencies from PyPI_ will require
having a C compiler (e.g. GCC_) as well as Python development files installed
on the system.

Note
^^^^

The preferred way to install the C compiler and Python development files is to
use your distribution's packages, if they exist. For example, on a
Fedora/RHEL-based system, that would mean installing ``gcc`` and
``python3-devel`` packages.

.. _Python: https://www.python.org/
.. _PostgreSQL: http://www.postgresql.org/
.. _PyPi: https://pypi.python.org/
.. _GCC: https://gcc.gnu.org/

Using PyPI_
-----------

.. code::

    pip install resolwe-bio

To install a pre-release, use:

.. code::

    pip install --pre resolwe-bio

Using source on GitHub_
-----------------------

.. code::

   pip install --pre https://github.com/genialis/resolwe-bio/archive/<git-tree-ish>.tar.gz

where ``<git-tree-ish>`` can represent any commit SHA, branch name, tag name,
etc. in `Resolwe Bioinformatics' GitHub repository`_. For example, to install
the latest Resolwe Bioinformatics from the ``master`` branch, use:

.. code::

   pip install --pre https://github.com/genialis/resolwe-bio/archive/master.tar.gz

.. _`Resolwe Bioinformatics' GitHub repository`: https://github.com/genialis/resolwe-bio/
.. _GitHub: `Resolwe Bioinformatics' GitHub repository`_

Contribute
==========

We welcome new contributors. To learn more, read Contributing_ section of the
documentation.

.. _Contributing: http://resolwe-bio.readthedocs.io/en/latest/contributing.html


  Prowler SaaS  and Prowler Open Source are as dynamic and adaptable as the environment they’re meant to protect. Trusted by the leaders in security.

Learn more at prowler.com

Join our Prowler community!

# Description

**Prowler** is an Open Source security tool to perform AWS, Azure, Google Cloud and Kubernetes security best practices assessments, audits, incident response, continuous monitoring, hardening and forensics readiness, and also remediations! We have Prowler CLI (Command Line Interface) that we call Prowler Open Source and a service on top of it that we call Prowler SaaS.

## Prowler CLI

```console
prowler <provider>
```
![Prowler CLI Execution](docs/img/short-display.png)

## Prowler Dashboard

```console
prowler dashboard
```
![Prowler Dashboard](docs/img/dashboard.png)

It contains hundreds of controls covering CIS, NIST 800, NIST CSF, CISA, RBI, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, AWS Well-Architected Framework Security Pillar, AWS Foundational Technical Review (FTR), ENS (Spanish National Security Scheme) and your custom security frameworks.

| Provider | Checks | Services | [Compliance Frameworks](https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/compliance/) | [Categories](https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/misc/#categories) |
|---|---|---|---|---|
| AWS | 360 | 66 -> `prowler aws --list-services` | 28 -> `prowler aws --list-compliance` | 7 -> `prowler aws --list-categories` |
| GCP | 77 | 13 -> `prowler gcp --list-services` | 1 -> `prowler gcp --list-compliance` | 2 -> `prowler gcp --list-categories`|
| Azure | 127 | 16 -> `prowler azure --list-services` | 2 -> `prowler azure --list-compliance` | 2 -> `prowler azure --list-categories` |
| Kubernetes | 83 | 7 -> `prowler kubernetes --list-services` | 1 -> `prowler kubernetes --list-compliance` | 7 -> `prowler kubernetes --list-categories` |

# 💻 Installation

## Pip package
Prowler is available as a project in [PyPI](https://pypi.org/project/prowler-cloud/), thus can be installed using pip with Python >= 3.9, < 3.13:

```console
pip install prowler
prowler -v
```
>More details at [https://docs.prowler.com](https://docs.prowler.com/projects/prowler-open-source/en/latest/)

## Containers

The available versions of Prowler are the following:

- `latest`: in sync with `master` branch (bear in mind that it is not a stable version)
- `v3-latest`: in sync with `v3` branch (bear in mind that it is not a stable version)
- `<x.y.z>` (release): you can find the releases [here](https://github.com/prowler-cloud/prowler/releases), those are stable releases.
- `stable`: this tag always point to the latest release.
- `v3-stable`: this tag always point to the latest release for v3.

The container images are available here:

- [DockerHub](https://hub.docker.com/r/toniblyx/prowler/tags)
- [AWS Public ECR](https://gallery.ecr.aws/prowler-cloud/prowler)

## From Github

Python >= 3.9, < 3.13 is required with pip and poetry:

```
git clone https://github.com/prowler-cloud/prowler
cd prowler
poetry shell
poetry install
python prowler.py -v
```
> If you want to clone Prowler from Windows, use `git config core.longpaths true` to allow long file paths.
# 📐✏ High level architecture

You can run Prowler from your workstation, a Kubernetes Job, a Google Compute Engine, an Azure VM, an EC2 instance, Fargate or any other container, CloudShell and many more.

![Architecture](docs/img/architecture.png)

# Deprecations from v3

## General
- `Allowlist` now is called `Mutelist`.
- The `--quiet` option has been deprecated, now use the `--status` flag to select the finding's status you want to get from PASS, FAIL or MANUAL.
- All `INFO` finding's status has changed to `MANUAL`.
- The CSV output format is common for all the providers.

We have deprecated some of our outputs formats:
- The native JSON is replaced for the JSON [OCSF](https://schema.ocsf.io/) v1.1.0, common for all the providers.

## AWS
- Deprecate the AWS flag --sts-endpoint-region since we use AWS STS regional tokens.
- To send only FAILS to AWS Security Hub, now use either `--send-sh-only-fails` or `--security-hub --status FAIL`.

# 📖 Documentation

Install, Usage, Tutorials and Developer Guide is at https://docs.prowler.com/

# 📃 License

Prowler is licensed as Apache License 2.0 as specified in each file. You may obtain a copy of the License at
<http://www.apache.org/licenses/LICENSE-2.0>