MoaT-MQTT
======

``MoaT-MQTT`` is an open source `MQTT`_ client and broker implementation. It
is a fork of `hbmqtt`_ with support for `anyio`_ and `MoaT-KV`_.

MoaT-MQTT provides a straightforward API based on coroutines, making it easy
to write highly concurrent applications.

.. _anyio: https://github.com/agronholm/anyio
.. _MoaT-KV: https://github.com/M-o-a-T/moat-kv
.. _hbmqtt: https://github.com/beerfactory/hbmqtt

Features
--------

MoaT-MQTT implements the full set of `MQTT 3.1.1`_ protocol specifications and provides the following features:

- Support for QoS 0, QoS 1 and QoS 2 messages flows
- Client auto-reconnection
- Authentication via password file (more methods can be added through a plugin system)
- Basic ``$SYS`` topics
- TCP and websocket support
- SSL support over TCP and websocket
- Plugin system
- Optional: Storage of retained messages in MoaT-KV

Build status
------------

TODO.

Project status
--------------

TODO.

Getting started
---------------

MoaT-MQTT is available on `Pypi <https://pypi.python.org/pypi/moat-mqtt>`_ and can installed simply using ``pip`` :
::

    $ pip install moat-mqtt

Documentation is available on `Read the Docs`_.

Bug reports, patches and suggestions welcome! Just `open an issue`_.

.. image:: https://badges.gitter.im/Join%20Chat.svg
    :target: https://gitter.im/beerfactory/moat-mqtt?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
    :alt: 'Join the chat at https://gitter.im/beerfactory/moat-mqtt'

.. _MQTT: http://www.mqtt.org
.. _MQTT 3.1.1: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
.. _Read the Docs: http://moat-mqtt.readthedocs.org/
.. _open an issue: https://github.com/M-o-a-T/moat-mqtt/issues/new

Moat-MQTT? DistMQTT? MoaT-KV? Whatever?
--------------------------------------

MoaT-MQTT is a Python package that includes a stand-alone MQTT server, as
well as basic client scripts. It is based on hbmqtt and was previously
named "DistMQTT".

`MoaT-KV <https://github.com/M-o-a-T/moat-kv>`_ is a distributed key-value
storage system. It uses the MoaT-MQTT client library as its connector to
an MQTT server, preferably Mosquitto or some other low-latency broker.
It was previously named "DistKV".

A MoaT-MQTT server can hook into MoaT-KV so that some messages are persisted,
translated (i.e. store msgpack messages encoding `True` and `False`, instead of
the strings "on" and "off" (or "ON" and "OFF" or "1" and "0" or …)), filtered
(e.g. the client can only modify existing messages but not add any), et al..


# PySD

[![Maintained](https://img.shields.io/badge/Maintained-Yes-brightgreen.svg)](https://github.com/SDXorg/pysd/pulse)
[![Coverage Status](https://coveralls.io/repos/github/SDXorg/pysd/badge.svg?branch=master)](https://coveralls.io/github/SDXorg/pysd?branch=master)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/pysd/badges/version.svg)](https://anaconda.org/conda-forge/pysd)
[![PyPI version](https://badge.fury.io/py/pysd.svg)](https://badge.fury.io/py/pysd)
[![PyPI status](https://img.shields.io/pypi/status/pysd.svg)](https://pypi.python.org/pypi/pysd/)
[![Py version](https://img.shields.io/pypi/pyversions/pysd.svg)](https://pypi.python.org/pypi/pysd/)
[![JOSS](https://joss.theoj.org/papers/10.21105/joss.04329/status.svg)](https://doi.org/10.21105/joss.04329)
[![Contributions](https://img.shields.io/badge/contributions-welcome-blue.svg)](https://pysd.readthedocs.io/en/latest/development/development_index.html)
[![Docs](https://readthedocs.org/projects/pysd/badge/?version=latest)](https://pysd.readthedocs.io/en/latest/?badge=latest)

![PySD Logo](https://raw.githubusercontent.com/SDXorg/pysd/5cc4fe5dc65e6b5140a00e87a1be9d261570ee8d/docs/images/PySD_Logo_letters.svg?style=centerme)

This project is a library for running [System Dynamics](http://en.wikipedia.org/wiki/System_dynamics) (SD) models in Python, with the purpose of improving integration of *Big Data* and *Machine Learning* into the SD workflow.

**The current version needs to run at least Python 3.9.**

## Resources

See the [project documentation](http://pysd.readthedocs.org/) for information about:

- [Installation](http://pysd.readthedocs.org/en/latest/installation.html)
- [Getting Started](http://pysd.readthedocs.org/en/latest/getting_started.html)
- [Release Notes](http://pysd.readthedocs.org/en/latest/whats_new.html)
- [Citing PySD](https://pysd.readthedocs.io/en/latest/#citing)

For standard methods for data analysis with SD models, see the  [PySD Cookbook](https://github.com/SDXorg/PySD-Cookbook), containing (for example):

- [Model Fitting](http://nbviewer.ipython.org/github/SDXorg/PySD-Cookbook/blob/master/source/analyses/fitting/Fitting_with_Optimization.ipynb)
- [Surrogating model components with machine learning regressions](http://nbviewer.ipython.org/github/SDXorg/PySD-Cookbook/blob/master/source/analyses/surrogating_functions/Surrogating_with_regression.ipynb)
- [Multi-Scale geographic comparison of model predictions](http://nbviewer.ipython.org/github/SDXorg/PySD-Cookbook/blob/master/source/analyses/geo/Exploring_models_across_geographic_scales.ipynb)

## Why create a new SD simulation engine?

There are a number of great SD programs out there ([Vensim](http://vensim.com/), [iThink](http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx), [AnyLogic](http://www.anylogic.com/system-dynamics), [Insight Maker](http://insightmaker.com/), and [others](https://en.wikipedia.org/wiki/Comparison_of_system_dynamics_software)). In order not to waste our effort, or fall victim to the [Not-Invented-Here](http://en.wikipedia.org/wiki/Not_invented_here) fallacy, we should have a very good reason for starting a new project.

That reason is this: There is a whole world of computational tools being developed in the larger data science community. **System dynamicists should directly use the tools that other people are building, instead of replicating their functionality in SD specific software.** The best way to do this is to bring specific SD functionality to the domain where those other tools are being developed.

This approach allows SD modelers to take advantage of the most recent developments in data science, and focus our efforts on improving the part of the stack that is unique to System Dynamics modeling.

## Cloning this repository

If you'd like to work with this repository directly, you'll need to use a recursive git checkout in order to properly load the test suite (sorry..)

The command should be something like:

```shell
git clone --recursive https://github.com/SDXorg/pysd.git
```

## Extensions

You can use PySD in [R](https://www.r-project.org/) via the [PySD2R](https://github.com/JimDuggan/pysd2r) package, also available on [CRAN](https://CRAN.R-project.org/package=pysd2r).

## Contributing

PySD is currently a community-maintained project, any contribution is welcome.

Many people have contributed to developing this project - by [submitting code](https://github.com/SDXorg/pysd/graphs/contributors), bug reports, and advice. Main historic changes in PySD are described in the [About PySD section](https://pysd.readthedocs.io/en/latest/about.html). The [Developer Documentation](https://pysd.readthedocs.io/en/latest/development/development_index.html) could help new developers.

The code for this package is available at: https://github.com/SDXorg/pysd

Join our slack channel in [sd-tools-and-methodology-community](https://slofile.com/slack/sdtoolsandmet-slj3251).


    Supercharge BigQuery
with BigFunctions

    Upgrade your data impact

    with 100+ ready-to-use BigQuery Functions

    (+ build a catalog of functions)

    ❯
      Website |
      GitHub
     ❮

---

## 🔍 1. What is BigFunctions?

BigFunctions is:

✅ **a framework** to build a **governed catalog of powerful BigQuery functions** at YOUR company.

✅ **100+ open-source functions to supercharge BigQuery** that you can call directly (no install) or redeploy in YOUR catalog.

## 💡 2. Why BigFunctions?

**As a data-analyst**

You'll have new powers! *(such as loading data from any source or activating your data through reverse ETL)*.

**As an analytics-engineer**

You'll feel at home with BigFunctions style which imitates the one of dbt *(with a yaml standard and a CLI)*.

You'll love the idea of getting more things done through SQL.

**As a data-engineer**

You'll easily build software-engineering best practices through unit testing, cicd, pull request validation, continuous deployment, etc.

You will love avoiding reinventing the wheel by using functions already developed by the community.

**As a central data-team player in a large company**

You'll be proud of providing a governed catalog of curated functions to your 10000+ employees with mutualized and maintainable effort.

**As a security champion**

You will enjoy the ability to validate the code of functions before deployment thanks to your git validation workflow, CI Testing, binary authorization, etc.

**As an open-source lover**

You'll be able to contribute so that a problem solved for you is solved for everyone.

## 👀 3. Call public BigFunctions without install from your GCP project

All BigFunctions represented by a 'yaml' file in *bigfunctions* folder of the GitHub repo are automatically deployed in public datasets so that you can call them directly without install from your BigQuery project.

Give it a try! Execute this SQL query from your GCP Project 👀:

```sql
select bigfunctions.eu.faker("name", "it_IT")
```

Explore all available bigfunctions **here**.

## 🚀 4. Deploy BigFunctions in your GCP project

You can also deploy any bigfunction in your project! To deploy *my_bigfunction* defined in *bigfunctions/my_bigfunction.yaml* file, simply call:

``` sh
bigfun deploy my_bigfunction
```

Details about `bigfun` command line are given below.

## 💥 5. `bigfun` CLI

`bigfun` CLI (command-line-interface) facilitates BigFunctions development, test, deployment, documentation and monitoring.

### 5.1 Install `bigfun` 🛠

``` sh
pip install bigfunctions
```

### 5.2 Use `bigfun` 🔥

``` sh
$ bigfun --help
Usage: bigfun [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  deploy      Deploy BIGFUNCTION
  docs        Generate, serve and publish documentation
  get         Download BIGFUNCTION yaml file from unytics/bigfunctions...
  test        Test BIGFUNCTION
```

### 5.3 Create you first function 👷

Functions are defined as yaml files under `bigfunctions` folder. To create your first function locally, the easiest is to download an existing yaml file of unytics/bigfunctions Github repo.

For instance to download `is_email_valid.yaml` into bigfunctions folder, do:

```sh
bigfun get is_email_valid
```

You can then update the file to suit your needs.

### 5.4 Deploy you first function 👨💻

> 1. Make sure the `gcloud` command is [installed on your computer](https://cloud.google.com/sdk/docs/install)
> 2. Activate the application-default account with `gcloud auth application-default login`. A browser window should open, and you should be prompted to log into your Google account. Once you've done that, `bigfun` will use your oauth'd credentials to connect to BigQuery through BigQuery python client!
> 3. Get or create a `DATASET` where you have permission to edit data and where the function will be deployed.
> 4. The `DATASET` must belong to a `PROJECT` in which you have permission to run BigQuery queries.

You now can deploy the function `is_email_valid` defined in `bigfunctions/is_email_valid.yaml` yaml file by running:

```sh
bigfun deploy is_email_valid
```

> The first time you run this command it will ask for `PROJECT` and `DATASET`.
>
> Your inputs will be written to `config.yaml` file in current directory so that you won't be asked again (unless you delete the entries in `config.yaml`). You can also override this config at deploy time: `bigfun deploy is_email_valid --project=PROJECT --dataset=DATASET`.

Test it with 👀:

```sql
select PROJECT.DATASET.is_email_valid('paul.marcombes@unytics.io')
```

### 5.5 Deploy you first javascript function which depends on *npm packages* 👽

*To deploy a **javascript** function* which depends on **npm packages** there are additional requirements *in addition to the ones above*.

> 1. You will need to install each *npm package* on your machine and bundle it into one file. For that, you need to [install *nodejs*](https://nodejs.org/en/download/).
> 2. The bundled js file will be uploaded into a cloud storage bucket in which you must have write access. The bucket name must be provided in `config.yaml` file in a variable named `bucket_js_dependencies`. Users of your functions must have read access to the bucket.

You now can deploy the function `render_template` defined in `bigfunctions/render_template.yaml` yaml file by running:

```sh
bigfun deploy render_template
```

Test it with 👀:

```sql
select PROJECT.DATASET.render_template('Hello {{ user }}', json '{"user": "James"}')
```

### 5.6 Deploy you first *remote* function ⚡

*To deploy a **remote** function* (e.g. python function), there are additional requirements *in addition to the ones of **Deploy you first function** section*.

> 1. A *Cloud Run* service will be deployed to host the code ([as seen here](https://cloud.google.com/bigquery/docs/reference/standard-sql/remote-functions)). So you must have [permissions to deploy a *Cloud Run* service](https://cloud.google.com/run/docs/deploying-source-code#permissions_required_to_deploy) in your project `PROJECT`.
> 2. `gcloud` CLI will be used directly to deploy the service (using `gcloud run deploy`). Then, make sure you are logged in with `gcloud` by calling: `gcloud auth login`. A browser window should also open, and you should be prompted to log into your Google account. WARNING: you read correctly: you have to authenticate twice. Once for bigquery python client (to deploy any function including remote as seen above.) and once now to use `gcloud` (to deploy a *Cloud Run* service).
> 3. A *BigQuery Remote Connection* will be created to link BigQuery with the *Cloud Run* service. You then should have permissions to create a remote connection. *[BigQuery Connection Admin](https://cloud.google.com/bigquery/docs/access-control#bigquery.connectionAdmin)* or *[BigQuery Admin](https://cloud.google.com/bigquery/docs/access-control#bigquery.admin)* roles have these permissions.
> 4. A service account will be automatically created by Google along with the *BigQuery Remote Connection*. BigQuery will use this service account of the remote connection to invoke the *Cloud Run* service. You then must have the permission to authorize this service account to invoke the *Cloud Run* service. This permission is provided in the role *[roles/run.admin](https://cloud.google.com/run/docs/reference/iam/roles)*

You now can deploy the function `faker` defined in `bigfunctions/faker.yaml` yaml file by running:

```sh
bigfun deploy faker
```

Test it with 👀:

```sql
select PROJECT.DATASET.faker("name", "it_IT")
```

## 👋 6. Contribute

BigFunctions is fully open-source. Any contribution is more than welcome 🤗!

- Add a ⭐ on the repo to show your support
- [Join our Slack](https://join.slack.com/t/unytics/shared_invite/zt-1gbv491mu-cs03EJbQ1fsHdQMcFN7E1Q) and talk with us
- Suggest a new function [here](https://github.com/unytics/bigfunctions/issues/new?assignees=&labels=new-bigfunction&projects=&template=0_new_bigfunction.yaml&title=%5Bnew%5D%3A+%60function_name%28argument1%2C+argument2%29%60)
- Raise an issue [there](https://github.com/unytics/bigfunctions/issues/new/choose)
- Open a Pull-Request! (See [contributing instructions](https://github.com/unytics/bigfunctions/blob/main/CONTRIBUTING.md)).

**Contributors**


# Welcome to JDBot

The simple easy to use bot that provides a wide range of entertainment and utility for your server.
Originally meant to be a test bot, this soon grew, and now it's at 90+ servers and verified!
This bot is meant to be having all of the features, an "all in one" bot.

## What does JDBot give you?

> Entertainment commands!
>
> > Hug, pat, kiss, all the interactive commands.
> > These are powered by the asuna and some random api. (yes it's actual name)
>
> Utility commands!
>
> > Extremely detailed analysis of user info and server info!
> > This uses the BetterUserConverter and the BetterMemberConverter!
>
> Moderation commands!
>
> > The classic warn, kick, and ban.
> > Warning someone will cause the bot to DM the person warned.
>

### JDBot is always being updated, with many new and useful features coming *in the future*

#### Any Questions about JDBot?

DM `jdjg`!

##### Copyright concerns, including the use of this code

This bot's source code is public so that the people can see how this bot works.
If you were to fork the bot, DO give credit to @JDJGInc and use this responsibly.
Also, if you want to copy the bot's code and run your own instance of JDBot, *do* DM `jdjg` on Discord.
(alternative): jdjg
If you plan to copy anything please let me know on discord or in an issue(this helps keep track of those who use my code, and so I can help you if you need help, please make your own code though.)
If you do copy any code make sure to make a good faith effort to change the code to your own style, that way it's not weird for word. I don't want you to copy every feature I have one for one, if you do want to do this, you can just host your own instance

(make sure to ask me how you can do that, or I can do that for you if you want.)

Though, I would rather you use my bot or make a pull request if you have changes or make an issue, you can just ask, if I deny your request, please understand I have my own reason for denying the change. If you would like to protest please do so in a civil manner.

My Bot also uses a subclassed Bot(if you don't program in discord.py or anyone of the forks, it's not important it just means one session bot wise to be able to do powerful functions without requiring many aiohttp sessions.)

Explanations for developers about the env values:

[Some JDBot Explanations](https://gist.github.com/JDJGInc/5607b3f63f651d2232a25b305aea4d33)

[Main/Bot Support Guild](https://discord.gg/eVQH2ZRYpz)

[Bot and Developer Guild](https://discord.gg/hKn8qgCDzK)

Main/Bot Support Guild is for general contact.

if you need more explicit you can join bot headquarters:

[Headquarters](https://discord.gg/MPNTtFH7pH)

Want A webhook github of this?
You can just DM me on Discord, and I will hook up your webhook to my github, so you will get updates too :D

If you Host your own instance, and call it your own orginal code.
I will take it down.
You Have Been Warned.
Now, if you got permission from me first, then you are fine but It must be done in a public way, I have the right to revoke this.

Also, if need to run it on linux if you want to try it. If you want to host it, just ask a code help server

Thank You,
JDJG


About conda-forge-pinning-feedstock
===================================

Feedstock license: [BSD-3-Clause](https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/LICENSE.txt)

Home: https://conda-forge.org/docs/maintainer/infrastructure.html#conda-forge-pinning

Package license: BSD-3-Clause

Summary: The baseline versions of software for the conda-forge ecosystem

Current build status
====================

All platforms:

Current release info
====================

| Name | Downloads | Version | Platforms |
| --- | --- | --- | --- |
| [![Conda Recipe](https://img.shields.io/badge/recipe-conda--forge--pinning-green.svg)](https://anaconda.org/conda-forge/conda-forge-pinning) | [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/conda-forge-pinning.svg)](https://anaconda.org/conda-forge/conda-forge-pinning) | [![Conda Version](https://img.shields.io/conda/vn/conda-forge/conda-forge-pinning.svg)](https://anaconda.org/conda-forge/conda-forge-pinning) | [![Conda Platforms](https://img.shields.io/conda/pn/conda-forge/conda-forge-pinning.svg)](https://anaconda.org/conda-forge/conda-forge-pinning) |

Installing conda-forge-pinning
==============================

Installing `conda-forge-pinning` from the `conda-forge` channel can be achieved by adding `conda-forge` to your channels with:

```
conda config --add channels conda-forge
conda config --set channel_priority strict
```

Once the `conda-forge` channel has been enabled, `conda-forge-pinning` can be installed with `conda`:

```
conda install conda-forge-pinning
```

or with `mamba`:

```
mamba install conda-forge-pinning
```

It is possible to list all of the versions of `conda-forge-pinning` available on your platform with `conda`:

```
conda search conda-forge-pinning --channel conda-forge
```

or with `mamba`:

```
mamba search conda-forge-pinning --channel conda-forge
```

Alternatively, `mamba repoquery` may provide more information:

```
# Search all versions available on your platform:
mamba repoquery search conda-forge-pinning --channel conda-forge

# List packages depending on `conda-forge-pinning`:
mamba repoquery whoneeds conda-forge-pinning --channel conda-forge

# List dependencies of `conda-forge-pinning`:
mamba repoquery depends conda-forge-pinning --channel conda-forge
```

About conda-forge
=================

[![Powered by
NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org)

conda-forge is a community-led conda channel of installable packages.
In order to provide high-quality builds, the process has been automated into the
conda-forge GitHub organization. The conda-forge organization contains one repository
for each of the installable packages. Such a repository is known as a *feedstock*.

A feedstock is made up of a conda recipe (the instructions on what and how to build
the package) and the necessary configurations for automatic building using freely
available continuous integration services. Thanks to the awesome service provided by
[Azure](https://azure.microsoft.com/en-us/services/devops/), [GitHub](https://github.com/),
[CircleCI](https://circleci.com/), [AppVeyor](https://www.appveyor.com/),
[Drone](https://cloud.drone.io/welcome), and [TravisCI](https://travis-ci.com/)
it is possible to build and upload installable packages to the
[conda-forge](https://anaconda.org/conda-forge) [anaconda.org](https://anaconda.org/)
channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance
[conda-smithy](https://github.com/conda-forge/conda-smithy) has been developed.
Using the ``conda-forge.yml`` within this repository, it is possible to re-render all of
this feedstock's supporting files (e.g. the CI configuration files) with ``conda smithy rerender``.

For more information please check the [conda-forge documentation](https://conda-forge.org/docs/).

Terminology
===========

**feedstock** - the conda recipe (raw material), supporting scripts and CI configuration.

**conda-smithy** - the tool which helps orchestrate the feedstock.
                   Its primary use is in the construction of the CI ``.yml`` files
                   and simplify the management of *many* feedstocks.

**conda-forge** - the place where the feedstock and smithy live and work to
                  produce the finished article (built conda distributions)

Updating conda-forge-pinning-feedstock
======================================

If you would like to improve the conda-forge-pinning recipe or build a new
package version, please fork this repository and submit a PR. Upon submission,
your changes will be run on the appropriate platforms to give the reviewer an
opportunity to confirm that the changes result in a successful build. Once
merged, the recipe will be re-built and uploaded automatically to the
`conda-forge` channel, whereupon the built conda packages will be available for
everybody to install and use from the `conda-forge` channel.
Note that all branches in the conda-forge/conda-forge-pinning-feedstock are
immediately built and any created packages are uploaded, so PRs should be based
on branches in forks and branches in the main repository should only be used to
build distinct package versions.

In order to produce a uniquely identifiable distribution:
 * If the version of a package **is not** being increased, please add or increase
   the [``build/number``](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#build-number-and-string).
 * If the version of a package **is** being increased, please remember to return
   the [``build/number``](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#build-number-and-string)
   back to 0.

Feedstock Maintainers
=====================

* [@conda-forge/Core](https://github.com/conda-forge/Core/)


  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/skypilot-wide-dark-1k.png">

  </picture>

    Run LLMs and AI on Any Cloud

----
:fire: *News* :fire:
- [Jun, 2024] Reproduce **GPT** with [llm.c](https://github.com/karpathy/llm.c/discussions/481) on any cloud: [**guide**](./llm/gpt-2/)
- [Apr, 2024] Serve and finetune [**Llama 3**](https://skypilot.readthedocs.io/en/latest/gallery/llms/llama-3.html) on any cloud or Kubernetes: [**example**](./llm/llama-3/)
- [Apr, 2024] Serve [**Qwen-110B**](https://qwenlm.github.io/blog/qwen1.5-110b/) on your infra: [**example**](./llm/qwen/)
- [Apr, 2024] Using [**Ollama**](https://github.com/ollama/ollama) to deploy quantized LLMs on CPUs and GPUs: [**example**](./llm/ollama/)
- [Feb, 2024] Deploying and scaling [**Gemma**](https://blog.google/technology/developers/gemma-open-models/) with SkyServe: [**example**](./llm/gemma/)
- [Feb, 2024] Serving [**Code Llama 70B**](https://ai.meta.com/blog/code-llama-large-language-model-coding/) with vLLM and SkyServe: [**example**](./llm/codellama/)
- [Dec, 2023] [**Mixtral 8x7B**](https://mistral.ai/news/mixtral-of-experts/), a high quality sparse mixture-of-experts model, was released by Mistral AI! Deploy via SkyPilot on any cloud: [**example**](./llm/mixtral/)
- [Nov, 2023] Using [**Axolotl**](https://github.com/OpenAccess-AI-Collective/axolotl) to finetune Mistral 7B on the cloud (on-demand and spot): [**example**](./llm/axolotl/)
- [Sep, 2023] Case study: [**Covariant**](https://covariant.ai/) transformed AI development on the cloud using SkyPilot, delivering models 4x faster cost-effectively: [**read the case study**](https://blog.skypilot.co/covariant/)
- [Aug, 2023] **Finetuning Cookbook**: Finetuning Llama 2 in your own cloud environment, privately: [**example**](./llm/vicuna-llama-2/), [**blog post**](https://blog.skypilot.co/finetuning-llama2-operational-guide/)
- [June, 2023] Serving LLM 24x Faster On the Cloud [**with vLLM**](https://vllm.ai/) and SkyPilot: [**example**](./llm/vllm/), [**blog post**](https://blog.skypilot.co/serving-llm-24x-faster-on-the-cloud-with-vllm-and-skypilot/)

  Archived

- [Mar, 2024] Serve and deploy [**Databricks DBRX**](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm) on your infra: [**example**](./llm/dbrx/)
- [Feb, 2024] Speed up your LLM deployments with [**SGLang**](https://github.com/sgl-project/sglang) for 5x throughput on SkyServe: [**example**](./llm/sglang/)
- [Dec, 2023] Using [**LoRAX**](https://github.com/predibase/lorax) to serve 1000s of finetuned LLMs on a single instance in the cloud: [**example**](./llm/lorax/)
- [Sep, 2023] [**Mistral 7B**](https://mistral.ai/news/announcing-mistral-7b/), a high-quality open LLM, was released! Deploy via SkyPilot on any cloud: [**Mistral docs**](https://docs.mistral.ai/self-deployment/skypilot)
- [July, 2023] Self-Hosted **Llama-2 Chatbot** on Any Cloud: [**example**](./llm/llama-2/)
- [April, 2023] [SkyPilot YAMLs](./llm/vicuna/) for finetuning & serving the [Vicuna LLM](https://lmsys.org/blog/2023-03-30-vicuna/) with a single command!

----

SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution.

SkyPilot **abstracts away cloud infra burdens**:
- Launch jobs & clusters on any cloud
- Easy scale-out: queue and run many jobs, automatically managed
- Easy access to object stores (S3, GCS, R2)

SkyPilot **maximizes GPU availability for your jobs**:
* Provision in all zones/regions/clouds you have access to ([the _Sky_](https://arxiv.org/abs/2205.07147)), with automatic failover

SkyPilot **cuts your cloud costs**:
* [Managed Spot](https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html): 3-6x cost savings using spot VMs, with auto-recovery from preemptions
* [Optimizer](https://skypilot.readthedocs.io/en/latest/examples/auto-failover.html): 2x cost savings by auto-picking the cheapest VM/zone/region/cloud
* [Autostop](https://skypilot.readthedocs.io/en/latest/reference/auto-stop.html): hands-free cleanup of idle clusters

SkyPilot supports your existing GPU, TPU, and CPU workloads, with no code changes.

Install with pip (we recommend the nightly build for the latest features or [from source](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)):
```bash
pip install "skypilot-nightly[aws,gcp,azure,oci,lambda,runpod,fluidstack,paperspace,cudo,ibm,scp,kubernetes]"  # choose your clouds
```
To get the last release, use:
```bash
pip install -U "skypilot[aws,gcp,azure,oci,lambda,runpod,fluidstack,paperspace,cudo,ibm,scp,kubernetes]"  # choose your clouds
```

Current supported providers (AWS, Azure, GCP, OCI, Lambda Cloud, RunPod, Fluidstack, Paperspace, Cudo, IBM, Samsung, Cloudflare, any Kubernetes cluster):

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-dark.png">

  </picture>

## Getting Started
You can find our documentation [here](https://skypilot.readthedocs.io/en/latest/).
- [Installation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)
- [Quickstart](https://skypilot.readthedocs.io/en/latest/getting-started/quickstart.html)
- [CLI reference](https://skypilot.readthedocs.io/en/latest/reference/cli.html)

## SkyPilot in 1 Minute

A SkyPilot task specifies: resource requirements, data to be synced, setup commands, and the task commands.

Once written in this [**unified interface**](https://skypilot.readthedocs.io/en/latest/reference/yaml-spec.html) (YAML or Python API), the task can be launched on any available cloud.  This avoids vendor lock-in, and allows easily moving jobs to a different provider.

Paste the following into a file `my_task.yaml`:

```yaml
resources:
  accelerators: V100:1  # 1x NVIDIA V100 GPU

num_nodes: 1  # Number of VMs to launch

# Working directory (optional) containing the project codebase.
# Its contents are synced to ~/sky_workdir/ on the cluster.
workdir: ~/torch_examples

# Commands to be run before executing the job.
# Typical use: pip install -r requirements.txt, git clone, etc.
setup: |
  pip install "torch<2.2" torchvision --index-url https://download.pytorch.org/whl/cu121

# Commands to run as a job.
# Typical use: launch the main program.
run: |
  cd mnist
  python main.py --epochs 1
```

Prepare the workdir by cloning:
```bash
git clone https://github.com/pytorch/examples.git ~/torch_examples
```

Launch with `sky launch` (note: [access to GPU instances](https://skypilot.readthedocs.io/en/latest/cloud-setup/quota.html) is needed for this example):
```bash
sky launch my_task.yaml
```

SkyPilot then performs the heavy-lifting for you, including:
1. Find the lowest priced VM instance type across different clouds
2. Provision the VM, with auto-failover if the cloud returned capacity errors
3. Sync the local `workdir` to the VM
4. Run the task's `setup` commands to prepare the VM for running the task
5. Run the task's `run` commands

Refer to [Quickstart](https://skypilot.readthedocs.io/en/latest/getting-started/quickstart.html) to get started with SkyPilot.

## More Information
To learn more, see our [Documentation](https://skypilot.readthedocs.io/en/latest/) and [Tutorials](https://github.com/skypilot-org/skypilot-tutorial).

Runnable examples:
- LLMs on SkyPilot
  - [GPT-2 via `llm.c`](./llm/gpt-2/)
  - [Llama 3](./llm/llama-3/)
  - [Qwen](./llm/qwen/)
  - [Databricks DBRX](./llm/dbrx/)
  - [Gemma](./llm/gemma/)
  - [Mixtral 8x7B](./llm/mixtral/); [Mistral 7B](https://docs.mistral.ai/self-deployment/skypilot/) (from official Mistral team)
  - [Code Llama](./llm/codellama/)
  - [vLLM: Serving LLM 24x Faster On the Cloud](./llm/vllm/) (from official vLLM team)
  - [SGLang: Fast and Expressive LLM Serving On the Cloud](./llm/sglang/) (from official SGLang team)
  - [Vicuna chatbots: Training & Serving](./llm/vicuna/) (from official Vicuna team)
  - [Train your own Vicuna on Llama-2](./llm/vicuna-llama-2/)
  - [Self-Hosted Llama-2 Chatbot](./llm/llama-2/)
  - [Ollama: Quantized LLMs on CPUs](./llm/ollama/)
  - [LoRAX](./llm/lorax/)
  - [QLoRA](https://github.com/artidoro/qlora/pull/132)
  - [LLaMA-LoRA-Tuner](https://github.com/zetavg/LLaMA-LoRA-Tuner#run-on-a-cloud-service-via-skypilot)
  - [Tabby: Self-hosted AI coding assistant](https://github.com/TabbyML/tabby/blob/bed723fcedb44a6b867ce22a7b1f03d2f3531c1e/experimental/eval/skypilot.yaml)
  - [LocalGPT](./llm/localgpt)
  - [Falcon](./llm/falcon)
  - Add yours here & see more in [`llm/`](./llm)!
- Framework examples: [PyTorch DDP](https://github.com/skypilot-org/skypilot/blob/master/examples/resnet_distributed_torch.yaml), [DeepSpeed](./examples/deepspeed-multinode/sky.yaml), [JAX/Flax on TPU](https://github.com/skypilot-org/skypilot/blob/master/examples/tpu/tpuvm_mnist.yaml), [Stable Diffusion](https://github.com/skypilot-org/skypilot/tree/master/examples/stable_diffusion), [Detectron2](https://github.com/skypilot-org/skypilot/blob/master/examples/detectron2_docker.yaml), [Distributed](https://github.com/skypilot-org/skypilot/blob/master/examples/resnet_distributed_tf_app.py) [TensorFlow](https://github.com/skypilot-org/skypilot/blob/master/examples/resnet_app_storage.yaml), [Ray Train](examples/distributed_ray_train/ray_train.yaml), [NeMo](https://github.com/skypilot-org/skypilot/blob/master/examples/nemo/nemo.yaml), [programmatic grid search](https://github.com/skypilot-org/skypilot/blob/master/examples/huggingface_glue_imdb_grid_search_app.py), [Docker](https://github.com/skypilot-org/skypilot/blob/master/examples/docker/echo_app.yaml), [Cog](https://github.com/skypilot-org/skypilot/blob/master/examples/cog/), [Unsloth](https://github.com/skypilot-org/skypilot/blob/master/examples/unsloth/unsloth.yaml), [Ollama](https://github.com/skypilot-org/skypilot/blob/master/llm/ollama), [llm.c](https://github.com/skypilot-org/skypilot/tree/master/llm/gpt-2) and [many more (`examples/`)](./examples).

Follow updates:
- [Twitter](https://twitter.com/skypilot_org)
- [Slack](http://slack.skypilot.co)
- [SkyPilot Blog](https://blog.skypilot.co/) ([Introductory blog post](https://blog.skypilot.co/introducing-skypilot/))

Read the research:
- [SkyPilot paper](https://www.usenix.org/system/files/nsdi23-yang-zongheng.pdf) and [talk](https://www.usenix.org/conference/nsdi23/presentation/yang-zongheng) (NSDI 2023)
- [Sky Computing whitepaper](https://arxiv.org/abs/2205.07147)
- [Sky Computing vision paper](https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s02-stoica.pdf) (HotOS 2021)
- [Policy for Managed Spot Jobs](https://www.usenix.org/conference/nsdi24/presentation/wu-zhanghao)  (NSDI 2024)

## Support and Questions
We are excited to hear your feedback!
* For issues and feature requests, please [open a GitHub issue](https://github.com/skypilot-org/skypilot/issues/new).
* For questions, please use [GitHub Discussions](https://github.com/skypilot-org/skypilot/discussions).

For general discussions, join us on the [SkyPilot Slack](http://slack.skypilot.co).

## Contributing
We welcome and value all contributions to the project! Please refer to [CONTRIBUTING](CONTRIBUTING.md) for how to get involved.


![logo](./assets/identity/graphnet-logo-and-wordmark.png)

| Usage                                                                                | Development |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- |
| [![status](https://joss.theoj.org/papers/eecab02fb1ecd174a5273750c1ea0baf/status.svg)](https://joss.theoj.org/papers/eecab02fb1ecd174a5273750c1ea0baf)             | ![build](https://github.com/graphnet-team/graphnet/actions/workflows/build.yml/badge.svg) |
| [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6720188.svg)](https://doi.org/10.5281/zenodo.6720188)                                                          | ![code-quality](https://github.com/graphnet-team/graphnet/actions/workflows/code-quality.yml/badge.svg) |
| [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)                                               | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) |
| ![Supported python versions](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)                                                   | [![Maintainability](https://api.codeclimate.com/v1/badges/b273a774112e32643162/maintainability)](https://codeclimate.com/github/graphnet-team/graphnet/maintainability) |
| [![Docker image](https://img.shields.io/docker/v/asogaard/graphnet?color=blue&logo=docker&sort=semver)](https://hub.docker.com/repository/docker/asogaard/graphnet) | [![Test Coverage](https://api.codeclimate.com/v1/badges/b273a774112e32643162/test_coverage)](https://codeclimate.com/github/graphnet-team/graphnet/test_coverage) |

## :rocket: About

**GraphNeT** is an open-source Python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using deep learning (DL). GraphNeT makes it fast and easy to train complex models that can provide event reconstruction with state-of-the-art performance, for arbitrary detector configurations, with inference times that are orders of magnitude faster than traditional reconstruction techniques.

Feel free to join the [GraphNeT Slack group](https://join.slack.com/t/graphnet-team/signup)!

### Publications using GraphNeT

| Type | Title | DOI |
| --- | --- | --- |
| Proceeding | Extending the IceCube search for neutrino point sources in the Northern sky with additional years of data | [![PoS](https://img.shields.io/badge/PoS-ICRC2023.1060-blue)]([https://doi.org/10.1088/1748-0221/17/11/P11003]([https://pos.sissa.it/444/1036/pdf](https://pos.sissa.it/444/1060/pdf))) |
| Proceeding | Sensitivity of the IceCube Upgrade to Atmospheric Neutrino Oscillations | [![PoS](https://img.shields.io/badge/PoS-ICRC2023.1036-blue)]([https://doi.org/10.1088/1748-0221/17/11/P11003](https://pos.sissa.it/444/1036/pdf)) |
| Paper | GraphNeT: Graph neural networks for neutrino telescope event reconstruction | [![status](https://joss.theoj.org/papers/eecab02fb1ecd174a5273750c1ea0baf/status.svg)](https://joss.theoj.org/papers/eecab02fb1ecd174a5273750c1ea0baf) |
| Paper | Graph Neural Networks for low-energy event classification & reconstruction in IceCube | [![JINST](https://img.shields.io/badge/JINST-10.1088%2F1748--0221%2F17%2F11%2FP11003-blue)](https://doi.org/10.1088/1748-0221/17/11/P11003) |

## :gear:  Install

GraphNeT is compatible with Python 3.8 - 3.11, Linux and macOS, and we recommend installing `graphnet` in a separate virtual environment. To install GraphNeT, please follow the [installation instructions](https://graphnet-team.github.io/graphnet/installation/install.html#quick-start)

## :ringed_planet:  Use cases

Below is an incomplete list of potential use cases for Deep Learning in neutrino telescopes.
These are categorised as either "Reconstruction challenges" that are considered common and that may benefit several experiments physics analyses; and those same "Experiments" and "Physics analyses".

Reconstruction challenges

| Title | Status | People | Materials |
| --- | --- | --- | --- |
| Low-energy neutrino classification and reconstruction | Done | Rasmus Ørsøe | https://arxiv.org/abs/2209.03042 |
| High-energy neutrino classification and reconstruction | Active | Rasmus Ørsøe | |
| Pulse noise cleaning | Paused | Rasmus Ørsøe, Kaare Iversen (past), Morten Holm | |
| (In-)elasticity reconstruction | Paused | Marc Jacquart (past) | |
| Multi-class event classification | Active | Morten Holm, Peter Andresen | |
| Data/MC difference mitigation |  | | |
| Systematic uncertainty mitigation |  | | |

Experiments

| Title | Status | People | Materials |
| --- | --- | --- | --- |
| IceCube | Active | (...) | |
| IceCube-Upgrade | Active | (...) | |
| IceCube-Gen2 | Active | (...) | |
| P-ONE | | (...) | |
| KM3NeT-ARCA | | (...) | |
| KM3NeT-ORCA | | (...) | |

Physics analyses

| Title | Status | People | Materials |
| --- | --- | --- | --- |
| Neutrino oscillations | | | |
| Point source searches | | | |
| Low-energy cosmic alerts | | | |
| High-energy cosmic alerts | | | |
| Moon pointing | | | |
| Muon decay asymmetry | | | |
| Spectra measurements | | | |

## :handshake:  Contributing

To make sure that the process of contributing is as smooth and effective as possible, we provide a few guidelines in the [contributing guide](CONTRIBUTING.md) that we encourage contributors to follow.

In short, everyone who wants to contribute to this project is more than welcome to do so! Contributions are handled through pull requests, that should be linked to a [GitHub issue](https://github.com/graphnet-team/graphnet/issues) describing the feature to be added or bug to be fixed. Pull requests will be reviewed by the project maintainers and merged into the main branch when accepted.

## :memo: License

GraphNeT has an Apache 2.0 license, as found in the [LICENSE](LICENSE) file.

## :raised_hands: Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 890778, and the PUNCH4NFDI consortium via DFG fund “NFDI39/1”.

[^1]: Examples of this are shown in the [examples/01_icetray/01_convert_i3_files.py](./examples/01_icetray/01_convert_i3_files.py) script


# QUTMS Driverless

## A Driverless FSAE/FS racecar
The Driverless team at QUTMS has been in development of an autonomous package for an electric racecar since 2020. Many simulations have been designed or adapted to develop perception and control algorithms while physical hardware was unavailable.

QEV-3, *Lando*, has been the first testbench for the driverless team and was used for hardware testing throughout 2021. As of November 2021, the team had successfully tested perception (LiDAR and camera) and control (steering actuation) on Lando. Additionally, the team took the autonomous package on QEV-3 to track the following month, testing these components with a safety driver to control throttle and brake only, while also gathering an abundance of real-world™ data. This has allowed further development for real-time pipelines, using machine learning and clustering models to detect cones outlining the track circuit.

In 2022, the team was determined to race QEV-3 as one of the first autonomous racecars in Australia during the 2022 FSAE Australasian Competition. After months of hardware and software debugging, ensuring algorithms were reliable, and redesigning some critical systems, QEV-3 was able to drive itself in October. This was remote controlled at first, then operated under its own processes. Soon after, the team was on the road to the competition in December, where QEV-3 passed all technical inspections to be the second competition-ready FSAE Autonomous Vehicle.

As we begin 2023, the Driverless team is aiming to improve the reliability of navigation processes and streamline our software stack for faster and more effective development into the future.

The Driverless System is developed using the Robot Operating System (ROS2), for efficient data transfer and package management. We are constantly pushing the boundaries of cutting-edge to create a sophisticated and future-proof package for years to come.


[![CI](https://github.com/basxsoftwareassociation/bread/actions/workflows/main.yml/badge.svg)](https://github.com/basxsoftwareassociation/bread/actions/workflows/main.yml)
[![Documentation Status](https://readthedocs.org/projects/basx-bread/badge/?version=latest)](https://basx-bread.readthedocs.io/en/latest/?badge=latest)
[![Translation status](https://hosted.weblate.org/widgets/basxconnect/-/bread/svg-badge.svg)](https://hosted.weblate.org/engage/basxconnect/)

Documentation
-------------

Documentation is hosted at [readthedocs.io](https://basx-bread.readthedocs.io/en/latest/)

BasxBread Engine
------------

This package provides functionality to create Django-based database
applications in a quick and customizable way. Similar concepts are CRUD
(create-read-update-delete) frameworks or RAD (rapid application development)
tools.

BasxBread relies in many regards on the Django web framework. Familiarity with
Django is highly recommended and assumed for readers of the documentation.

Installation
------------

```bash
pip install basx-bread
```

Quickstart
----------

The following are the required step to get a new project quickly up and running.
For seasoned Django users there should be nothing new for the most parts.
In that case only the section *Registering the UI* might be worth reading.

### Setup

```bash
    python3 -m venv .venv
    . .venv/bin/activate # this is for bash, for windows use the script .venv/bin/Activate.ps1, there are also scripts for csh and fish
    pip install basx-bread # should run without problems, but users reported problems in some Mac setups due to native libraries missing
    django-admin startproject --template $( pip show basx-bread | grep '^Location: ' | cut -c 11- )/basxbread/resources/project_template/ myproject . # template-project for basxbread

    # adding a Django "app", Django projects consist of different apps with different models, pretty standard
    # can also be achieved with "python manage.py startapp mymodels" but it would create a few unnecessary files
    mkdir mymodels mymodels/migrations
    touch mymodels/__init__.py mymodels/migrations/__init__.py
    echo -e 'from django.apps import AppConfig\n\n\nclass Mymodels(AppConfig):\n    name = "mymodels"' > mymodels/apps.py
```

After this the file ```mymodels/models.py``` needs to be created and filled with your database models. Then add ```"mymodels"``` to the list of ```INSTALLED_APPS``` inside ```myproject/settings/base.py```.

### Registering the UI

In order to get started with the UI quickly the following code can be put into ```mymodels/urls.py```.
The code below assumes there exists a single model inside ```mymodels/models.py``` called ```MyModel```.

```python
    from basxbread.utils import quickregister
    from . import models

    urlpatterns = []
    quickregister(urlpatterns, models.MyModel)
```

The root URL list in ```myproject/urls.py``` needs to be extended with an item ```path("myapp", include("mymodels.urls"))```.

### Running the application

Finally run the following commands to initialize the database and start the development server.

```bash
    python manage.py makemigrations
    python manage.py migrate
    python manage.py createsuperuser
    python manage.py runserver
```

The application can now be accessed via http://127.0.0.1:8000.

Notes on maintenance
--------------------

Most of the code that is being written for BasxBread is developed while doing
payed work. That way the development and maintenance of the framework can be
done in a sustainable manner.

Future plans
------------

BasxBread is currently running on a range of custom database applications that
are used in production. Most parts of the framework are now on a level that we
consider production ready. However, there are some additions and improvments
that we still would like to work on. Those are listed here.

Refactorings:

(Maybe move this stuff into Github issues)

- [ ] We should really go over our documentation...
- [ ] Change implementation of some things where not the HTML/REST-paradigm is
  used but a custom javascript hack. This might reduce some of the "asthetical"
  behaviour, like clickable table rows. However, for customization and
  composition of different UI elements it is always preferable to use standard
  HTML behaviour and features where possibel.
- [ ] Current-menu-item-selection. Right now the currently active menu item is
  only detected via prefix-matching of the current URL. This can lead sometimes
  to incorrect behaviour, if two menu items have a similar prefix. There should
  be an unambigous way to determine the currently active menu item, maybe even
  for pages that are deeper in the navigation hierarchy
- [ ] Pre-defined querysets with direct links for BrowseViews. BrowseViews
  would benefit of a short-cut system, where frequently used filters can be
  defined in the code and are displayed as links in the datatable header.
- [ ] Improve definition of URL patterns and navigation hierarchy. No idea what
  would be a good, generic approach, maybe take some stuff from DRF.

New features

- [ ] Data analytics with graphs [analytics]: In order to allow producing nice
  graphs, exploring all the data and getting statistical insights a data
  analytics-tool would be really nice to have.
- [ ] Editing models via the web UI [modeledit]: This is a feature which
  almost all of the bigger database frameworks and CRM support. But BasxBread
  main goal is to empower the developer and end-user accessible model
  definitions increase developer friction substantialy. However, there are
  many cases where a project would benefit if certain changes can be done
  without having to update code and make a new deployment. Therefore we are
  experimenting with a feature that would allow users to add and modify
  certain models via the web-interface. The implementation would still rely on
  code-defined models. The web-UI would translate the desired changes into an
  automated version of what the developer would normally do, i.e. updating the
  model definition in the source code, creating migrations, running migrations
  and restart the application server.
- [ ] Customizable menu [custommenu]: Currently the navigation menu must be
  defined via code. Allowing the configuration of the menu via database would
  be an advantage for users. Two things to keep in mind for this are: It must
  still be easy for developers to quickly add menus to test models and custom
  pages. And also, the current feature set (e.g. icons and permissions)
  for menu items should be supported in a database implementation.
- [ ] More/better customization of custom forms.
- [ ] Add feature do define custom layouts.


[![License Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
![Python 3.6, 3.7, 3.8, 3.9, 3.10, 3.11](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-green.svg)
[![Downloads](https://pepy.tech/badge/deeppavlov)](https://pepy.tech/project/deeppavlov)
<img align="right" height="27%" width="27%" src="docs/_static/deeppavlov_logo.png"/>

DeepPavlov is an open-source conversational AI library built on [PyTorch](https://pytorch.org/).

DeepPavlov is designed for
* development of production ready chat-bots and complex conversational systems,
* research in the area of NLP and, particularly, of dialog systems.

## Quick Links

* Demo [*demo.deeppavlov.ai*](https://demo.deeppavlov.ai/)
* Documentation [*docs.deeppavlov.ai*](http://docs.deeppavlov.ai/)
    * Model List [*docs:features/*](http://docs.deeppavlov.ai/en/master/features/overview.html)
    * Contribution Guide [*docs:contribution_guide/*](http://docs.deeppavlov.ai/en/master/devguides/contribution_guide.html)
* Issues [*github/issues/*](https://github.com/deeppavlov/DeepPavlov/issues)
* Forum [*forum.deeppavlov.ai*](https://forum.deeppavlov.ai/)
* Blogs [*medium.com/deeppavlov*](https://medium.com/deeppavlov)
* [Extended colab tutorials](https://github.com/deeppavlov/dp_tutorials)
* Docker Hub [*hub.docker.com/u/deeppavlov/*](https://hub.docker.com/u/deeppavlov/)
    * Docker Images Documentation [*docs:docker-images/*](http://docs.deeppavlov.ai/en/master/intro/installation.html#docker-images)

Please leave us [your feedback](https://forms.gle/i64fowQmiVhMMC7f9) on how we can improve the DeepPavlov framework.

**Models**

[Named Entity Recognition](http://docs.deeppavlov.ai/en/master/features/models/NER.html) | [Intent/Sentence Classification](http://docs.deeppavlov.ai/en/master/features/models/classification.html) |

[Question Answering over Text (SQuAD)](http://docs.deeppavlov.ai/en/master/features/models/SQuAD.html) | [Knowledge Base Question Answering](http://docs.deeppavlov.ai/en/master/features/models/KBQA.html)

[Sentence Similarity/Ranking](http://docs.deeppavlov.ai/en/master/features/models/neural_ranking.html) | [TF-IDF Ranking](http://docs.deeppavlov.ai/en/master/features/models/tfidf_ranking.html)

[Syntactic Parsing](http://docs.deeppavlov.ai/en/master/features/models/syntax_parser.html) | [Morphological Tagging](http://docs.deeppavlov.ai/en/master/features/models/morpho_tagger.html)

[Automatic Spelling Correction](http://docs.deeppavlov.ai/en/master/features/models/spelling_correction.html) | [Entity Extraction](http://docs.deeppavlov.ai/en/master/features/models/entity_extraction.html)

[Open Domain Questions Answering](http://docs.deeppavlov.ai/en/master/features/models/ODQA.html) | [Russian SuperGLUE](http://docs.deeppavlov.ai/en/master/features/models/superglue.html)

[Relation Extraction](http://docs.deeppavlov.ai/en/master/features/models/relation_extraction.html)

**Embeddings**

[BERT embeddings for the Russian, Polish, Bulgarian, Czech, and informal English](http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html#bert)

[ELMo embeddings for the Russian language](http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html#elmo)

[FastText embeddings for the Russian language](http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html#fasttext)

**Auto ML**

[Tuning Models](http://docs.deeppavlov.ai/en/master/features/hypersearch.html)

**Integrations**

[REST API](http://docs.deeppavlov.ai/en/master/integrations/rest_api.html) | [Socket API](http://docs.deeppavlov.ai/en/master/integrations/socket_api.html)

[Amazon AWS](http://docs.deeppavlov.ai/en/master/integrations/aws_ec2.html)

## Installation

0. DeepPavlov supports `Linux`, `Windows 10+` (through WSL/WSL2), `MacOS` (Big Sur+) platforms, `Python 3.6`, `3.7`, `3.8`, `3.9` and `3.10`.
    Depending on the model used, you may need from 4 to 16 GB RAM.

1. Create and activate a virtual environment:
    * `Linux`
    ```
    python -m venv env
    source ./env/bin/activate
    ```
2. Install the package inside the environment:
    ```
    pip install deeppavlov
    ```

## QuickStart

There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is
determined by its config file.

List of models is available on
[the doc page](http://docs.deeppavlov.ai/en/master/features/overview.html) in
the `deeppavlov.configs` (Python):

```python
from deeppavlov import configs
```

When you're decided on the model (+ config file), there are two ways to train,
evaluate and infer it:

* via [Command line interface (CLI)](#command-line-interface-cli) and
* via [Python](#python).

#### GPU requirements

By default, DeepPavlov installs models requirements from PyPI. PyTorch from PyPI could not support your device CUDA
capability. To run supported DeepPavlov models on GPU you should have [CUDA](https://developer.nvidia.com/cuda-toolkit)
compatible with used GPU and [PyTorch version](deeppavlov/requirements/pytorch.txt) required by DeepPavlov models.
See [docs](https://docs.deeppavlov.ai/en/master/intro/quick_start.html#using-gpu) for details.
GPU with Pascal or newer architecture and 4+ GB VRAM is recommended.

### Command line interface (CLI)

To get predictions from a model interactively through CLI, run

```bash
python -m deeppavlov interact <config_path> [-d] [-i]
```

* `-d` downloads required data - pretrained model files and embeddings (optional).
* `-i` installs model requirements (optional).

You can train it in the same simple way:

```bash
python -m deeppavlov train <config_path> [-d] [-i]
```

Dataset will be downloaded regardless of whether there was `-d` flag or not.

To train on your own data you need to modify dataset reader path in the
[train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).
The data format is specified in the corresponding model doc page.

There are even more actions you can perform with configs:

```bash
python -m deeppavlov <action> <config_path> [-d] [-i]
```

* `<action>` can be
    * `install` to install model requirements (same as `-i`),
    * `download` to download model's data (same as `-d`),
    * `train` to train the model on the data specified in the config file,
    * `evaluate` to calculate metrics on the same dataset,
    * `interact` to interact via CLI,
    * `riseapi` to run a REST API server (see
    [doc](http://docs.deeppavlov.ai/en/master/integrations/rest_api.html)),
    * `predict` to get prediction for samples from *stdin* or from
      *<file_path>* if `-f <file_path>` is specified.
* `<config_path>` specifies path (or name) of model's config file
* `-d` downloads required data
* `-i` installs model requirements

### Python

To get predictions from a model interactively through Python, run

```python
from deeppavlov import build_model

model = build_model(<config_path>, install=True, download=True)

# get predictions for 'input_text1', 'input_text2'
model(['input_text1', 'input_text2'])
```
where
* `install=True` installs model requirements (optional),
* `download=True` downloads required data from web - pretrained model files and embeddings (optional),
* `<config_path>` is model name (e.g. `'ner_ontonotes_bert_mult'`), path to the chosen model's config file (e.g.
  `"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"`),  or `deeppavlov.configs` attribute (e.g.
  `deeppavlov.configs.ner.ner_ontonotes_bert_mult` without quotation marks).

You can train it in the same simple way:

```python
from deeppavlov import train_model

model = train_model(<config_path>, install=True, download=True)
```

To train on your own data you need to modify dataset reader path in the
[train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).
The data format is specified in the corresponding model doc page.

You can also calculate metrics on the dataset specified in your config file:

```python
from deeppavlov import evaluate_model

model = evaluate_model(<config_path>, install=True, download=True)
```

DeepPavlov also [allows](https://docs.deeppavlov.ai/en/master/intro/python.html) to build a model from components for
inference using Python.

## License

DeepPavlov is Apache 2.0 - licensed.


Adaptive experiments for neuroscience
---------

[![PyPI](https://img.shields.io/pypi/v/improv?style=flat-square?style=flat-square)](https://pypi.org/project/improv)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/improv?style=flat-square)](https://pypi.org/project/improv)
[![docs](https://github.com/project-improv/improv/actions/workflows/docs.yaml/badge.svg?style=flat-square)](https://project-improv.github.io/)
[![tests](https://github.com/project-improv/improv/actions/workflows/CI.yaml/badge.svg?style=flat-square)](https://github.com/project-improv/improv/actions/workflows/CI.yaml)
[![Coverage Status](https://coveralls.io/repos/github/project-improv/improv/badge.svg?branch=main)](https://coveralls.io/github/project-improv/improv?branch=main)
[![PyPI - License](https://img.shields.io/pypi/l/improv?style=flat-square)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square)](https://github.com/psf/black)

_improv_ is a streaming software platform designed to enable adaptive experiments. By analyzing data as they arrive, we can obtain information about the current brain state in real time and use it to adaptively modify an experiment as data collection is ongoing.

![](https://dibs-files.cloud.duke.edu/sites/default/files/improvGif.gif)

This video shows raw 2-photon calcium imaging data in zebrafish, with cells detected in real time by [CaImAn](https://github.com/flatironinstitute/CaImAn), and directional tuning curves (shown as colored neurons) and functional connectivity (lines) estimated online, during a live experiment. Here only a few minutes of data have been acquired, and neurons are colored by their strongest response to visual simuli shown so far.
We also provide up-to-the-moment estimates of the functional connectivity by fitting linear-nonlinear-Poisson models online, as each new piece of data is acquired. Simple visualizations offer real-time insights, allowing for adaptive experiments that change in response to the current state of the brain.

### How improv works

improv allows users to flexibly specify and manage adaptive experiments to integrate data collection, preprocessing, visualization, and user-defined analytics. All kinds of behavioral, neural, or modeling data can be incorporated, and input and output data streams are managed independently and asynchronously. With this design, streaming analyses and real-time interventions can be easily integrated into various experimental setups. improv manages the backend engineering of data flow and task execution for all steps in an experimental pipeline in real time, without requiring user oversight. Users need only define their particular processing pipeline with simple text files and are free to define their own streaming analyses via Python classes, allowing for rapid prototyping of adaptive experiments.

_improv_'s design is based on a streamlined version of the actor model for concurrent computation. Each component of the system (experimental pipeline) is considered an 'actor' and has a unique role. They interact via message passing, without the need for a central broker. Actors are implemented as user-defined classes that inherit from _improv_'s `Actor` class, which supplies all queues for message passing and orchestrates process execution and error handling. Messages between actors are composed of keys that correspond to items in a shared, in-memory data store. This both minimizes communication overhead and data copying between processes.

## Installation

For installation instructions, please consult the [documentation](https://project-improv.github.io/improv/installation.html).

### Contact
To get in touch, feel free to reach out on Twitter @annedraelos or @jmxpearson.


Massive Text Embedding Benchmark

        Installation |
        Usage |
        Leaderboard |
        Documentation |
        Citing

## Installation

```bash
pip install mteb
```

## Usage

* Using a python script (see [scripts/run_mteb_english.py](https://github.com/embeddings-benchmark/mteb/blob/main/scripts/run_mteb_english.py) and [mteb/mtebscripts](https://github.com/embeddings-benchmark/mtebscripts) for more):

```python
import mteb
from sentence_transformers import SentenceTransformer

# Define the sentence-transformers model name
model_name = "average_word_embeddings_komninos"
# or directly from huggingface:
# model_name = "sentence-transformers/all-MiniLM-L6-v2"

model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(tasks=["Banking77Classification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")
```

* Using CLI

```bash
mteb available_tasks

mteb run -m sentence-transformers/all-MiniLM-L6-v2 \
    -t Banking77Classification  \
    --verbosity 3

# if nothing is specified default to saving the results in the results/{model_name} folder
```

* Using multiple GPUs in parallel can be done by just having a custom encode function that distributes the inputs to multiple GPUs like e.g. [here](https://github.com/microsoft/unilm/blob/b60c741f746877293bb85eed6806736fc8fa0ffd/e5/mteb_eval.py#L60) or [here](https://github.com/ContextualAI/gritlm/blob/09d8630f0c95ac6a456354bcb6f964d7b9b6a609/gritlm/gritlm.py#L75).

   Advanced Usage (click to unfold)

## Advanced Usage

### Dataset selection

Datasets can be selected by providing the list of datasets, but also

* by their task (e.g. "Clustering" or "Classification")

```python
tasks = mteb.get_tasks(task_types=["Clustering", "Retrieval"]) # Only select clustering and retrieval tasks
```

* by their categories e.g. "s2s" (sentence to sentence) or "p2p" (paragraph to paragraph)

```python
tasks = mteb.get_tasks(categories=["s2s", "p2p"]) # Only select sentence2sentence and paragraph2paragraph datasets
```

* by their languages

```python
tasks = mteb.get_tasks(languages=["eng", "deu"]) # Only select datasets which contain "eng" or "deu" (iso 639-3 codes)
```

You can also specify which languages to load for multilingual/cross-lingual tasks like below:

```python
import mteb

tasks = [
    mteb.get_task("AmazonReviewsClassification", languages = ["eng", "fra"]),
    mteb.get_task("BUCCBitextMining", languages = ["deu"]), # all subsets containing "deu"
]

# or you can select specific huggingface subsets like this:
from mteb.tasks import AmazonReviewsClassification, BUCCBitextMining

evaluation = mteb.MTEB(tasks=[
        AmazonReviewsClassification(hf_subsets=["en", "fr"]) # Only load "en" and "fr" subsets of Amazon Reviews
        BUCCBitextMining(hf_subsets=["de-en"]), # Only load "de-en" subset of BUCC
])
# for an example of a HF subset see "Subset" in the dataset viewer at: https://huggingface.co/datasets/mteb/bucc-bitext-mining
```

There are also presets available for certain task collections, e.g. to select the 56 English datasets that form the "Overall MTEB English leaderboard":

```python
from mteb import MTEB_MAIN_EN
evaluation = mteb.MTEB(tasks=MTEB_MAIN_EN, task_langs=["en"])
```

### Evaluation split
You can evaluate only on `test` splits of all tasks by doing the following:

```python
evaluation.run(model, eval_splits=["test"])
```

Note that the public leaderboard uses the test splits for all datasets except MSMARCO, where the "dev" split is used.

### Using a custom model

Models should implement the following interface, implementing an `encode` function taking as inputs a list of sentences, and returning a list of embeddings (embeddings can be `np.array`, `torch.tensor`, etc.). For inspiration, you can look at the [mteb/mtebscripts repo](https://github.com/embeddings-benchmark/mtebscripts) used for running diverse models via SLURM scripts for the paper.

```python
class MyModel():
    def encode(
        self, sentences: list[str], **kwargs: Any
    ) -> torch.Tensor | np.ndarray:
        """Encodes the given sentences using the encoder.

        Args:
            sentences: The sentences to encode.
            **kwargs: Additional arguments to pass to the encoder.

        Returns:
            The encoded sentences.
        """
        pass

model = MyModel()
tasks = mteb.get_task("Banking77Classification")
evaluation = MTEB(tasks=tasks)
evaluation.run(model)
```

If you'd like to use different encoding functions for query and corpus when evaluating on Retrieval or Reranking tasks, you can add separate methods for `encode_queries` and `encode_corpus`. If these methods exist, they will be automatically used for those tasks. You can refer to the `DRESModel` at `mteb/evaluation/evaluators/RetrievalEvaluator.py` for an example of these functions.

```python
class MyModel():
    def encode_queries(self, queries: list[str], **kwargs) -> list[np.ndarray] | list[torch.Tensor]:
        """
        Returns a list of embeddings for the given sentences.
        Args:
            queries: List of sentences to encode

        Returns:
            List of embeddings for the given sentences
        """
        pass

    def encode_corpus(self, corpus: list[str] | list[dict[str, str]], **kwargs) -> list[np.ndarray] | list[torch.Tensor]:
        """
        Returns a list of embeddings for the given sentences.
        Args:
            corpus: List of sentences to encode
                or list of dictionaries with keys "title" and "text"

        Returns:
            List of embeddings for the given sentences
        """
        pass
```

### Evaluating on a custom dataset

To evaluate on a custom task, you can run the following code on your custom task. See [how to add a new task](docs/adding_a_dataset.md), for how to create a new task in MTEB.

```python
from mteb import MTEB
from mteb.abstasks.AbsTaskReranking import AbsTaskReranking
from sentence_transformers import SentenceTransformer

class MyCustomTask(AbsTaskReranking):
    ...

model = SentenceTransformer("average_word_embeddings_komninos")
evaluation = MTEB(tasks=[MyCustomTask()])
evaluation.run(model)
```

## Documentation

| Documentation                  |                        |
| ------------------------------ | ---------------------- |
| 📋 [Tasks] | Overview of available tasks |
| 📈 [Leaderboard] | The interactive leaderboard of the benchmark |
| 🤖 [Adding a model] | Information related to how to submit a model to the leaderboard |
| 👩🔬 [Reproducible workflows] | Information related to how to reproduce and create reproducible workflows with MTEB |
| 👩💻 [Adding a dataset] | How to add a new task/dataset to MTEB |
| 👩💻 [Adding a leaderboard tab] | How to add a new leaderboard tab to MTEB |
| 🤝 [Contributing] | How to contribute to MTEB and set it up for development |
| 🌐 [MMTEB] | An open-source effort to extend MTEB to cover a broad set of languages |

[Tasks]: docs/tasks.md
[Contributing]: CONTRIBUTING.md
[Adding a model]: docs/adding_a_model.md
[Adding a dataset]: docs/adding_a_dataset.md
[Adding a leaderboard tab]: docs/adding_a_leaderboard_tab.md
[Leaderboard]: https://huggingface.co/spaces/mteb/leaderboard
[MMTEB]: docs/mmteb/readme.md
[Reproducible workflows]: docs/reproducible_workflow.md

## Citing

MTEB was introduced in "[MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316)", feel free to cite:

```bibtex
@article{muennighoff2022mteb,
  doi = {10.48550/ARXIV.2210.07316},
  url = {https://arxiv.org/abs/2210.07316},
  author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"\i}c and Reimers, Nils},
  title = {MTEB: Massive Text Embedding Benchmark},
  publisher = {arXiv},
  journal={arXiv preprint arXiv:2210.07316},
  year = {2022}
}
```

You may also want to read and cite the amazing work that has extended MTEB & integrated new datasets:
- Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff. "[C-Pack: Packaged Resources To Advance General Chinese Embedding](https://arxiv.org/abs/2309.07597)" arXiv 2023
- Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, Han Xiao. "[Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://arxiv.org/abs/2310.19923)" arXiv 2023
- Silvan Wehrli, Bert Arnrich, Christopher Irrgang. "[German Text Embedding Clustering Benchmark](https://arxiv.org/abs/2401.02709)" arXiv 2024
- Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini. "[FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions](https://arxiv.org/abs/2403.15246)" arXiv 2024
- Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li. "[LongEmbed: Extending Embedding Models for Long Context Retrieval](https://arxiv.org/abs/2404.12096)" arXiv 2024
- Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, Kristoffer Laigaard Nielbo. "[The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://arxiv.org/abs/2406.02396)" arXiv 2024

For works that have used MTEB for benchmarking, you can find them on the [leaderboard](https://huggingface.co/spaces/mteb/leaderboard).


# SingularityCE User Docs

This repository holds user-facing documentation for the
[SingularityCE](https://github.com/sylabs/singularity) container runtime.

This is a community project led by [Sylabs](https://sylabs.io), and
contributions are always welcome! If you'd like to update or improve
SingularityCE's documentation please follow the instructions below, and submit a
PR on GitHub.

## Setting up an environment to contribute

The Singularity user documentation is written in [reStructured Text (RST)
format](http://docutils.sourceforge.net/rst.html) and generated using
[Sphinx](https://pypi.org/project/Sphinx/). The
[ReadTheDocs](https://readthedocs.org/) theme for Sphinx is used.

We use RST instead of markdown as it's better at handling large documents with
many linked sections, and Sphinx makes it easy to produce online documentation
as well as PDFs.

Sphinx is written in Python. To get setup to contribute:

- Install Python 3.5 or newer, from your OS package manager or the [Python download
  site](https://www.python.org/downloads/)
- Use `pip3`to install Sphinx, RTD theme package, and extensions / linters into
  your home directory:

```sh
pip3 install --user sphinx sphinx-rtd-theme rstcheck pygments m2r2
```

If your version of python 3 does not come with `pip` / `pip3`, you may need to
install a `python3-pip`package with `apt` or `yum`, or you can install pip
following [the instructions here](https://pip.pypa.io/en/stable/installing/).

You're all set! After this you will only need to use your favorite editor to
work with the RST files.

## How to edit & write RST

A Sphinx documentation project has some structure that it's good to know before
you dive in and start editing or writing content.

### Structure of the project

This project maintains the following structure:

1. `index.rst` : contains the front page of the online documentation and the
   initial table of contents tree. Every documentation section is referenced by
   a tag next to its name. (e.g. ``Quick Start <quick_start>``)
2. All other `.rst`files are sections named to match reference tags described
   before. So, for the `<quick_start>` reference in `index.rst` you'll find
   a `quick_start.rst` file with the content for that section.
3. The configuration used to build the final documentations from the `.rst`
   files is set in the `conf.py` file.

### The conf.py file

This file sets the themes, extensions, variables and naming scheme for output
created when building the documentation with Sphinx. Some important elements
include:

- `version` : Describes the current version of `SingularityCE` that the
  documentation is for. We set version to the `major.minor` values, e.g. `3.5`,
  as we are not creating separate documentation for each patch release.
- `release`: Would be used to specify the current release of the software being
  documented, including patch number, alpha tags etc. We leave this the same as
  `version` as we only generate documentation for each `major.minor` version of
  singularity.
- `html_theme`: Sets the theme to be used for HTML output. We are using the RTD
  or Read The Docs theme.
- `html_context`: Options here control links back to our GitHub repository.
- `html_logo`: The logo for the sidebar
- `html_favicon`: The `favicon`for the entire project

### Cheatsheet to get started with reStructured (RST) Text

RST is similar to Markdown, but has enough differences that you are likely to be
caught out a few times until you are familiar with it. Let's look at some of the
common things you need to do when writing or editing RST.

#### 1. Create a section/subsection/subsubsection title

Section titles are defined by surrounding or underlining them with different
characters. Each combination of overline/underline and character used represents
a different level section. We follow the conventions used by the python
documentation for headers:

```rst
##################
H1: document title
##################

*********
Sample H2
*********

Sample H3
=========

Sample H4
---------

Sample H5
^^^^^^^^^

Sample H6
"""""""""
```

#### 2. Referencing sections

To reference a section in an RST file you need to first create the reference
above the title you need to reference, and second to reference it where you need
the link to the reference section. When you build HTML or PDF output with Sphinx
it will create the links for you, so the reader can jump around the
documentation easily.

##### Step 1: Create the reference

To create the reference on the section you need to link, you must specify a
tag:

```rst
.. _build-docker:

-------------------------
Build from a Docker Image
-------------------------
```

This example will let you refer to the section titled "Build from a Docker
Image" with the tag `build-docker`. Note that the tag here doesn't include the
`_` that you have to prefix it with when creating the reference.

##### Step 2: Reference it

To reference a section you need to use this syntax:

```rst
:ref:`read the section covering docker images <build-docker>`
```

`:ref:` tells Sphinx that this is a reference. In between the `` ` ``you should
provide text for the link, and then the tag you created above between `< >`.

### Further Reading

These are some of the basics of RST. For a more complete introduction, see the
[Sphinx documentation](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html)

### Building HTML documentation

To build the HTML documentation, make sure you are in the top level of the
`singularity-userdocs` repository and run:

```sh
make html SKIPCLI=1
```

This will generate a folder called `_build/html` with the output. Open
`index.html` to browse the documentation.

The `SKIPCLI=1` option tells `make` not to generate the CLI reference, which is
created automatically from the SingularityCE source code. You can generate the CLI
documentation by running `make html` alone. This requires a Go build
environment (see below).

### Generating PDF files from RST

This is very similar to the previous step, you will need to run:

```sh
make latexpdf SKIPCLI=1
```

Output is written into `_build/latex` and the final PDF will be named
`singularity-userdocs.pdf`

### Generating EPUB from RST

Very similar to the previous command, you will just need to run:

```sh
make epub SKIPCLI=1
```

Output is written into `_build/epub` and the final EPUB will be named
`singularity-userdocs.epub`

## Generating CLI docs

The SingularityCE CLI docs are generated using the actual code from SingularityCE.
To do this, we include SingularityCE as a git submodule, and whenever a Makefile
target (like `make html`) is run, `singularity` itself is compiled and used to
generate the CLI docs.

However, you might not want to compile `singularity`, either because you can't on
your machine, or because you want to test out a quick change to the docs.  If
this is the case, you can skip the CLI doc generation using the `SKIPCLI`
argument.  For example, to rebuild the HTML docs without including the CLI docs,
just run `make html SKIPCLI=1`.

If SingularityCE has been updated and you want to synchronize the CLI docs with
the new version of SingularityCE, you'll have to update the submodule.  To do
this, just run:

```bash
git submodule update --remote --merge
git add vendor/src/github.com/sylabs/singularity
git commit
```

This will update the submodule to the latest version of the main
branch.


[![Python](https://img.shields.io/pypi/pyversions/gymnasium.svg)](https://badge.fury.io/py/gymnasium)
[![PyPI](https://badge.fury.io/py/gymnasium.svg)](https://badge.fury.io/py/gymnasium)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8127025.svg)](https://doi.org/10.5281/zenodo.8127025)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://pre-commit.com/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

Gymnasium is an open source Python library for developing and comparing reinforcement learning algorithms by providing a standard API to communicate between learning algorithms and environments, as well as a standard set of environments compliant with that API. This is a fork of OpenAI's [Gym](https://github.com/openai/gym) library by its maintainers (OpenAI handed over maintenance a few years ago to an outside team), and is where future maintenance will occur going forward.

The documentation website is at [gymnasium.farama.org](https://gymnasium.farama.org), and we have a public discord server (which we also use to coordinate development work) that you can join here: https://discord.gg/bnJ6kubTg6

## Environments

Gymnasium includes the following families of environments along with a wide variety of third-party environments
* [Classic Control](https://gymnasium.farama.org/environments/classic_control/) - These are classic reinforcement learning based on real-world problems and physics.
* [Box2D](https://gymnasium.farama.org/environments/box2d/) - These environments all involve toy games based around physics control, using box2d based physics and PyGame-based rendering
* [Toy Text](https://gymnasium.farama.org/environments/toy_text/) - These environments are designed to be extremely simple, with small discrete state and action spaces, and hence easy to learn. As a result, they are suitable for debugging implementations of reinforcement learning algorithms.
* [MuJoCo](https://gymnasium.farama.org/environments/mujoco/) - A physics engine based environments with multi-joint control which are more complex than the Box2D environments.
* [Atari](https://gymnasium.farama.org/environments/atari/) - A set of 57 Atari 2600 environments simulated through Stella and the Arcade Learning Environment that have a high range of complexity for agents to learn.
* [Third-party](https://gymnasium.farama.org/environments/third_party_environments/) - A number of environments have been created that are compatible with the Gymnasium API. Be aware of the version that the software was created for and use the `apply_env_compatibility` in `gymnasium.make` if necessary.

## Installation

To install the base Gymnasium library, use `pip install gymnasium`

This does not include dependencies for all families of environments (there's a massive number, and some can be problematic to install on certain systems). You can install these dependencies for one family like `pip install "gymnasium[atari]"` or use `pip install "gymnasium[all]"` to install all dependencies.

We support and test for Python 3.8, 3.9, 3.10, 3.11 on Linux and macOS. We will accept PRs related to Windows, but do not officially support it.

## API

The Gymnasium API models environments as simple Python `env` classes. Creating environment instances and interacting with them is very simple- here's an example using the "CartPole-v1" environment:

```python
import gymnasium as gym
env = gym.make("CartPole-v1")

observation, info = env.reset(seed=42)
for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)

    if terminated or truncated:
        observation, info = env.reset()
env.close()
```

## Notable Related Libraries

Please note that this is an incomplete list, and just includes libraries that the maintainers most commonly point newcommers to when asked for recommendations.

* [CleanRL](https://github.com/vwxyzjn/cleanrl) is a learning library based on the Gymnasium API. It is designed to cater to newer people in the field and provides very good reference implementations.
* [PettingZoo](https://github.com/Farama-Foundation/PettingZoo) is a multi-agent version of Gymnasium with a number of implemented environments, i.e. multi-agent Atari environments.
* The Farama Foundation also has a collection of many other [environments](https://farama.org/projects) that are maintained by the same team as Gymnasium and use the Gymnasium API.
* [Comet](https://www.comet.com/site/?utm_source=gymnasium&utm_medium=partner&utm_campaign=partner_gymnasium_2023&utm_content=github) is a free ML-Ops tool that tracks rewards, metrics, hyperparameters, and code for ML training runs. Comet has an easy-to use integration with Gymnasium, here's a [tutorial](https://bit.ly/CometGymnasiumIntegration) on how to use them together! Comet is a sponsor of the Farama Foundation!

## Environment Versioning

Gymnasium keeps strict versioning for reproducibility reasons. All environments end in a suffix like "-v0".  When changes are made to environments that might impact learning results, the number is increased by one to prevent potential confusion. These inherit from Gym.

## Development Roadmap

We have a roadmap for future development work for Gymnasium available here: https://github.com/Farama-Foundation/Gymnasium/issues/12

## Support Gymnasium's Development

If you are financially able to do so and would like to support the development of Gymnasium, please join others in the community in [donating to us](https://github.com/sponsors/Farama-Foundation).

## Citation

You can cite Gymnasium as:

```
@misc{towers_gymnasium_2023,
        title = {Gymnasium},
        url = {https://zenodo.org/record/8127025},
        abstract = {An API standard for single-agent reinforcement learning environments, with popular reference environments and related utilities (formerly Gym)},
        urldate = {2023-07-08},
        publisher = {Zenodo},
        author = {Towers, Mark and Terry, Jordan K. and Kwiatkowski, Ariel and Balis, John U. and Cola, Gianluca de and Deleu, Tristan and Goulão, Manuel and Kallinteris, Andreas and KG, Arjun and Krimmel, Markus and Perez-Vicente, Rodrigo and Pierré, Andrea and Schulhoff, Sander and Tai, Jun Jet and Shen, Andrew Tan Jin and Younis, Omar G.},
        month = mar,
        year = {2023},
        doi = {10.5281/zenodo.8127026},
}
```


![E.ON EBC RWTH Aachen University](https://raw.githubusercontent.com/RWTH-EBC/FiLiP/master/docs/logos/EBC_Logo.png)

# FiLiP

[![pylint](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/pylint/pylint.svg)](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/pylint/pylint.html)
[![Documentation](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/docs/doc.svg)](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/docs/index.html)
[![coverage](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/coverage/badge.svg)](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/coverage)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![build](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/build/build.svg)](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/build/build.svg)

FiLiP (Fiware Library for Python) is a python software development kit (SDK) for
accelerating the development of web services that use Fiware's Generic
Enablers (GEs) as backend.

It is mainly based on the [Pydantic](https://pydantic-docs.helpmanual.io/)
package which is a sophisticated library for data validation and settings
management using python type annotations.
Pydantic enforces type hints at runtime, and provides user friendly errors
when data is invalid.
We mainly use the Pydantic model to build our own data model structure required
for efficient data model parsing and validation and interaction with FIWARE
services' RestAPIs.

For API interaction, FiLiP relies on the well-known
[requests](https://docs.python-requests.org/en/latest/) package.
It is important to understand that we do not in any way restrict any
features of requests.

Furthermore, FiLiP is designed to help with the fast development of FIWARE-based
applications and avoid hundreds of lines of boilerplate, but it cannot
substitute learning the basic concepts behind the used FIWARE components.

## General Motivation

Why implement a client library when clients can be auto-generated
from openapi documentation?
A general prerequisite to do so is that the documentation is in depth and of
good quality.
While FIWARE generally provides
[openapi documentation](https://github.com/FIWARE/specifications),
here are some thoughts on the challenges of auto-generating client code from
these documents:

- Auto-generated code tends to become rather bulky and its quality strongly
  depends on the provided input data.
- Manipulating generated code can result in a big hassle for maintenance if
  additional features need to be integrated.
- The underlying NGSI (Next Generation Service Interface) for FIWARE is a
  rather generic specification.
  Hence, generated models may also be of generic types as lists
  and dicts in Python. So there is no real benefit.
  Furthermore, there is no chance for reasonable validation and error handling.

## Getting started

The following section shortly describes how to use the library.

### Prerequisites

Since FiLiP is designed as a client library, it requires a server that provides
the target Service-APIs.
Hence, if you do not yet have a running instance of a FIWARE based platform,
using docker is the most convenient way to set it up.
Please check [here](https://github.com/N5GEH/n5geh.platform) for a tutorial
on this.
If this is not an option for you, FIWARE also provides a testing server.
You can register for a testing account
[here](https://www.fiware.org/developers/fiware-lab/).
> **Note**: FiLiP is now compatible to [Pydantic V2](https://docs.pydantic.dev/latest/migration/). If your program still require Pydantic V1.x for some reason, please use release [v0.2.5](https://github.com/RWTH-EBC/FiLiP/releases/tag/v0.2.5) or earlier version of FiLiP. Besides, we recommended to use `pydantic~=1.10` in the `requirements.txt`

### Installation

The easiest way to install the library is via pip:

````
pip install -U filip
````

If you want to benefit from the latest changes, use the following command
(This will install the current master branch from this repository):

```
pip install -U git+git://github.com/RWTH-EBC/filip
```

#### Install semantics module (optional)

If you want to use the optional [semantics module](filip/semantics), use the following command (This will install the libraries that only required for the semantics module):
````
pip install -U filip[semantics]
````

### Introduction to FIWARE

The following section introduces FIWARE. If you are already familiar with
FIWARE, you can skip this section and go straight to [Getting Started](#getting-started).

#### What is FIWARE?

FIWARE is a framework of open-source cloud platform components, created
to facilitate the development of smart solutions within various application
domains.
At the moment, the FIWARE
[catalogue](https://www.fiware.org/developers/catalogue/) contains over 30
interoperable software modules, so-called Generic Enablers
(GE) for developing and providing customized IoT platform solutions.

To get familiar with the APIs of the different modules we highly recommend
checking the
[step-by-step tutorial](https://fiware-tutorials.readthedocs.io/en/latest/).
It provides a good overview on FIWARE and its basic usage.
Whereas the tutorial helps to understand most of the general concepts,
for a deep dive, where you can learn about the components in more detail,
FIWARE also offers extended lessons through their
[academy](https://fiware-academy.readthedocs.io/en/latest/index.html/).

However, usually one only requires a small set of components.
Hence, we recommend using the cited pages only as needed.

#### How to set up a FIWARE platform?

The easiest way to set up a FIWARE platform is by using docker as all GEs are
open-source and distributed as docker containers on dockerhub.

However, as mentioned before, for most use cases only a subset of GEs is required.
Hence, we wrote a small [tutorial](https://github.com/N5GEH/n5geh.platform)
explaining how to set up a platform suited for most use cases within the energy
domain.

#### FIWARE GEs covered by FiLiP

FiLiP is a library developed on demand.
Hence, we do not aim to cover the APIs of all GEs that are included in the
[catalogue](https://www.fiware.org/developers/catalogue/).
This would mean an unnecessary development overhead.
Therefore, FiLiP currently only covers the APIs of the following GEs:

- NGSIv2 Context Broker for managing context data. We use its
  reference implementation ORION for testing.
    - [documentation](https://fiware-orion.readthedocs.io/en/master/)
    - [github](https://github.com/telefonicaid/fiware-orion)
    - [swagger](https://swagger.lab.fiware.org/)
    - [NGSI v2 specifications](https://github.com/FIWARE/specifications/tree/master/OpenAPI/ngsiv2)

- IoT-Agents for managing IoT Devices. IoT agents are implemented using
  the FIWARE IoT Agent Node Lib as a common framework.
    - [documentation](https://iotagent-node-lib.readthedocs.io/en/latest/)
    - [github](https://github.com/telefonicaid/iotagent-node-lib)

- IoT-Agent-JSON for managing devices using a JSON message payload protocol
  format.
    - [documentation](https://fiware-iotagent-json.readthedocs.io/en/latest/)
    - [github](https://github.com/telefonicaid/iotagent-json)
    - [apiary](https://telefonicaiotiotagents.docs.apiary.io/)
    (*partly deprecated*)

  Example payload:

        {
            "humidity": "45%",
            "temperature": "23",
            "luminosity": "1570"
        }

- IoT-Agent-Ultralight for managing devices using an Ultralight 2.0 message
  payload protocol.

    - [documentation](https://fiware-iotagent-ul.readthedocs.io/en/latest/)
    - [github](https://github.com/telefonicaid/iotagent-ul)
    - [apiary](https://telefonicaiotiotagents.docs.apiary.io/)
      (*partly deprecated*)

    Example payload:

        humidity|45%|temperature|23|luminosity|1570

- QuantumLeap for the management of time series data

    - [documentation](https://quantumleap.readthedocs.io/en/latest/)
    - [github](https://github.com/FIWARE-GEs/quantum-leap)
    - [swagger](https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb/0.8.3)

## Structure of FiLiP

![Library Structure](https://raw.githubusercontent.com/RWTH-EBC/FiLiP/master/docs/diagrams/out/architecture.png)

## Documentation

We are still working on the documentation.
You can find our current documentation
[here](https://ebc.pages.rwth-aachen.de/EBC_all/github_ci/FiLiP/master/docs/index.html).

## Running examples

Once you have installed the library, you can check the [examples](/examples)
to learn how to use the different components.

Currently, we provide basic examples for the usage of FiLiP for the FIWARE
GEs mentioned above.
We suggest to start in the right order to first understand the
configuration of clients.
Afterwards, you can start modelling context data and interacting with the context
broker and use its functionalities before you learn how to connect
IoT Devices and store historic data.

## Testing

We use unittests to write our test cases.
To test the source code of the library in our CI workflow, the CI
executes all tests located in the `tests`-directory and prefixed with `test_` .

## How to contribute

Please see our [contribution guide](./CONTRIBUTING.md) for more details on
how you can contribute to this project.

## Authors

* [Thomas Storek](https://github.com/tstorek)
* [Junsong Du](https://www.ebc.eonerc.rwth-aachen.de/cms/E-ON-ERC-EBC/Das-Institut/Mitarbeiter/Digitale-Energie-Quartiere/~trcib/Du-Junsong/lidx/1/) (corresponding)
* [Saira Bano](https://www.ebc.eonerc.rwth-aachen.de/cms/E-ON-ERC-EBC/Das-Institut/Mitarbeiter/Systemadministration/~ohhca/Bano-Saira/)
* [Sebastian Blechmann](https://www.ebc.eonerc.rwth-aachen.de/cms/E-ON-ERC-EBC/Das-Institut/Mitarbeiter/Team2/~carjd/Blechmann-Sebastian/)

## Alumni

* Jeff Reding
* Felix Rehmann
* Daniel Nikolay

## References

We presented or applied the library in the following publications:

- S. Blechmann, I. Sowa, M. H. Schraven, R. Streblow, D. Müller & A. Monti. Open source platform application for smart building and smart grid controls. Automation in Construction 145 (2023), 104622. ISSN: 0926-5805. https://doi.org/10.1016/j.autcon.2022.104622

- Haghgoo, M., Dognini, A., Storek, T., Plamanescu, R, Rahe, U.,
  Gheorghe, S, Albu, M., Monti, A., Müller, D. (2021) A cloud-based service-oriented architecture to unlock smart energy services
  https://www.doi.org/10.1186/s42162-021-00143-x

- Baranski, M., Storek, T. P. B., Kümpel, A., Blechmann, S., Streblow, R.,
Müller, D. et al.,
(2020). National 5G Energy Hub : Application of the Open-Source Cloud Platform
FIWARE for Future Energy Management Systems.
https://doi.org/10.18154/RWTH-2020-07876

- T. Storek, J. Lohmöller, A. Kümpel, M. Baranski & D. Müller (2019).
Application of the open-source cloud platform FIWARE for future building
energy management systems.
Journal of Physics:
Conference Series, 1343, 12063. https://doi.org/10.1088/1742-6596/1343/1/012063

## License

This project is licensed under the BSD License - see the [LICENSE](LICENSE) file for details.

## Copyright

2021-2024, RWTH Aachen University, E.ON Energy Research Center, Institute for Energy
Efficient Buildings and Indoor Climate

[Institute for Energy Efficient Buildings and Indoor Climate (EBC)](https://www.ebc.eonerc.rwth-aachen.de)
[E.ON Energy Research Center (E.ON ERC)](https://www.eonerc.rwth-aachen.de)
[RWTH University Aachen, Germany](https://www.rwth-aachen.de)

## Disclaimer

This project is part of the cooperation between the RWTH Aachen University and
the Research Centre Jülich.

## Related projects

## Acknowledgments

We gratefully acknowledge the financial support of the Federal Ministry

for Economic Affairs and Climate Action (BMWK), promotional references
03ET1495A, 03ET1551A, 0350018A, 03ET1561B.

This project has received funding in the framework of the joint programming initiative ERA-Net Smart Grids Plus, with support from the European Union’s Horizon 2020 research and innovation programme.