# Meet Fides: Privacy as Code

[![Latest Release Version][release-image]][release-url]
[![Docker][docker-workflow-image]][docker-actions-url]
[![Docs][docs-workflow-image]][docs-actions-url]
[![Package][release-workflow-image]][publish-actions-url]
[![License][license-image]][license-url]
[![Code style: black][black-image]][black-url]
[![Checked with mypy][mypy-image]][mypy-url]
[![Twitter][twitter-image]][twitter-url]
[![Coverage](https://codecov.io/github/ethyca/fides/coverage.svg?branch=main)](https://codecov.io/gh/ethyca/fides)

![Fides banner](docs/fides/docs/img/fides-banner.png "Fides banner")

## :zap: Overview

Fides (pronounced */fee-dhez/*, from Latin: Fidēs) is an open-source privacy engineering platform for managing the fulfillment of data privacy requests in your runtime environment, and the enforcement of privacy regulations in your code.

## :rocket: Quick Start

### Getting Started

In order to get started quickly with Fides, a sample project is bundled within the Fides CLI that will set up a server, privacy center, and a sample application for you to experiment with.

#### Minimum requirements (for all platforms)

* [Docker](https://www.docker.com/products/docker-desktop) (version 20.10.11 or later)
* [Python](https://www.python.org/downloads/) (version 3.8 through 3.10)

#### Download and install Fides

> [!TIP]
> We highly recommend setting up a Python virtual environment such as `venv` to install Fides into. For example:
>
> ```sh
> mkdir ~/fides
> cd ~/fides
> python3 -m venv venv
> source venv/bin/activate
> ```

Once your virtual environment is ready, you can easily download and install Fides using `pip`. Run the following command to get started:

```
pip install ethyca-fides
```

#### Deploy the Fides sample project

By default, Fides ships with a small project belonging to a fictional e-commerce store. Running the `fides deploy up` command builds a Fides project with all you need to run your first Data Subject Request against real databases.

```sh
fides deploy up
```

#### Explore the sample project

When your deployment finishes, a welcome screen will explain the key components of Fides and the sample "Cookie House" project.

If your browser does not open automatically, you should navigate to <http://localhost:3000/landing>.

The project contains:

* The Fides Admin UI for managing privacy requests
* The Fides [Privacy Center](https://docs.ethyca.com/fides/dsr_quickstart/privacy_center) for submitting requests
* The sample "Cookie House" eCommerce site for testing
* A DSR Directory on your computer to view results (`./fides_uploads`)

#### Run your first Privacy Access Request

Navigate to the Fides Privacy Center (<http://localhost:3001>), submit a "Download your data" request, provide the email address for the sample user (`jane@example.com`), and submit the request.

Then, navigate to the Fides Admin UI (<http://localhost:8080>) to review the pending privacy request.

Use username `root_user` and password `Testpassword1!` to login, approve the request, and review the resulting package in your `./fides_uploads` folder!

### Next Steps

Congratulations! You've just run an entire privacy request in under 5 minutes! Fides offers many more tools to help take control of your data privacy. To find out more, you can run a privacy request on [your own infrastructure](https://docs.ethyca.com/fides/dsr_quickstart/overview), discover [data mapping](https://docs.ethyca.com/fides/data_mapping/overview), or learn about the [Fides Taxonomy](https://ethyca.github.io/fideslang/).

## :book: Learn More

The Fides core team is committed to providing a variety of documentation to help get you started using Fides.  As such, all interactions are governed by the [Fides Code of Conduct](https://docs.ethyca.com/fides/community/code_of_conduct).

### Documentation

For more information on getting started with Fides, how to configure and set up Fides, and more about the Fides ecosystem of open source projects:

* Documentation: <https://docs.ethyca.com>
* Website: www.ethyca.com/fides

### Support

Join the conversation on:

* [Slack](https://fid.es/join-slack)
* [Twitter](https://twitter.com/ethyca)
* [Discussions](https://github.com/ethyca/fides/discussions)

### Contributing

We welcome and encourage all types of contributions and improvements!  Please see our [contribution guide](https://docs.ethyca.com/fides/community/overview) to opening issues for bugs, new features, and security or experience enhancements.

Read about the [Fides community](https://docs.ethyca.com/fides/community/hints_tips) or dive into the [contributor guides](https://docs.ethyca.com/fides/community/development/overview) for information about contributions, documentation, code style, testing and more. Ethyca is committed to fostering a safe and collaborative environment, such that all interactions are governed by the [Fides Code of Conduct](https://docs.ethyca.com/fides/community/code_of_conduct).

## :balance_scale: License

The [Fides](https://github.com/ethyca/fides) ecosystem of tools are licensed under the [Apache Software License Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
Fides tools are built on [fideslang](https://github.com/ethyca/privacy-taxonomy), the Fides language specification, which is licensed under [CC by 4](https://github.com/ethyca/privacy-taxonomy/blob/main/LICENSE).

Fides is created and sponsored by Ethyca: a developer tools company building the trust infrastructure of the internet. If you have questions or need assistance getting started, let us know at fides@ethyca.com!

[release-image]: https://img.shields.io/github/release/ethyca/fides.svg
[release-url]: https://github.com/ethyca/fides/releases
[docker-workflow-image]: https://github.com/ethyca/fides/workflows/Docker%20Build%20&%20Push/badge.svg
[docs-workflow-image]: https://github.com/ethyca/fides/workflows/Publish%20Docs/badge.svg
[release-workflow-image]: https://github.com/ethyca/fides/actions/workflows/publish_package.yaml/badge.svg
[docker-actions-url]: https://github.com/ethyca/fides/actions/workflows/publish_docker.yaml
[docs-actions-url]: https://github.com/ethyca/fides/actions/workflows/publish_docs.yaml
[publish-actions-url]: https://github.com/ethyca/fides/actions/workflows/publish_package.yaml
[license-image]: https://img.shields.io/:license-Apache%202-blue.svg
[license-url]: https://www.apache.org/licenses/LICENSE-2.0.txt
[black-image]: https://img.shields.io/badge/code%20style-black-000000.svg
[black-url]: https://github.com/psf/black/
[mypy-image]: http://www.mypy-lang.org/static/mypy_badge.svg
[mypy-url]: http://mypy-lang.org/
[twitter-image]: https://img.shields.io/twitter/follow/ethyca?style=social
[twitter-url]: https://twitter.com/ethyca

## ⚠ Advanced Setup for Microsoft SQL Server (MSSQL) Support

By default, running `pip install ethyca-fides` locally will not install the optional Python libraries needed for Microsoft SQL Server, since these rely on additional system dependencies (`freetds`)! However, if you *do* want to connect to MSSQL, you have two options:
1. Use our pre-built Docker images which install these optional dependencies automatically: [`ethyca/fides`](https://hub.docker.com/r/ethyca/fides). See our [Deployment Guide](https://ethyca.com/docs/dev-docs/configuration/deployment) for more!
2. Install the required dependencies on your local development machine and run `pip install ethyca-fides[all]` to include "all" the optional libraries. Keep reading to learn more about this!

For local development setup on macOS, follow these steps:
1. Install the required development libraries from Homebrew:
```bash
brew install freetds openssl
```
2. Add the following to your shell (i.e. `.zshrc`) to ensure your compiler can access the `freetds` and `openssl` libraries, updating the paths & versions to match your local install:
```bash
export LDFLAGS="-L/opt/homebrew/Cellar/freetds/1.3.18/lib -L/opt/homebrew/Cellar/openssl@1.1/1.1.1u/lib"
export CFLAGS="-I/opt/homebrew/Cellar/freetds/1.3.18/include"
```
3. Reinstall Fides with MSSQL support by including the `all` extra requirement:
```bash
pip install ethyca-fides[all]
```


![Arkwork](https://github.com/vertyco/Vertyco/blob/main/.github/resources/vrt-cogs-01.png)

Various useful and fun cogs for Red V3.

![Py](https://img.shields.io/badge/python-v3.11-orange?style=for-the-badge)
![black](https://img.shields.io/badge/style-black-000000?style=for-the-badge&?link=https://github.com/psf/black)
![license](https://img.shields.io/github/license/Vertyco/Vrt-Cogs?style=for-the-badge)

![Red-DiscordBot](https://img.shields.io/badge/Red%20DiscordBot-V3.5-red?style=for-the-badge)
![GitHub repo size](https://img.shields.io/github/repo-size/Vertyco/Vrt-Cogs?color=blueviolet&style=for-the-badge)

I'm a full time software developer that hosts Ark servers as a hobby.
I enjoy working with APIs and making tools that are useful to myself and others.
Check out some of my projects below!

| Cog                                    | Description                                                                                |
|----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Assistant](assistant/README.md)       | Get your own AI assistant in Discord Automate your server with OpenAI's ChatGPT language model (or [self-host your own!](https://github.com/vertyco/gpt-api)). Configure a prompt with condensed information about your server, along with other options to make answering questions easier. Setup embeddings to optimize token usage, and even write custom functions to extend the functionality of what the assistant can do! |
| [AutoDocs](autodocs/README.md)         | Document your cogs with ease! Easily generate documentation for any cog in Markdown format.                                                                                |
| [BankBackup](bankbackup/README.md)     | Backup and restore bank balances for all members in a server. Backup/Restore for server bank balances.                                                                                |
| [BankEvents](bankevents/README.md)     | Bank transaction listener events for 3rd party cogs. Dispatches events when different bank transactions occur, such as when a user deposits credits, withdraws credits, or transfers credits.                                                                                |
| [EconomyTrack](economytrack/README.md) | Track your economy's total balance and member count over time. Visualize the total market cap of your Red economy. Note: Having bank set to local on a public bot will cause much heavier I/O load than using a global bank.                                                                                |
| [Cartographer](cartographer/README.md) | Discord Server Backup/Restore Create backups of your Discord server and restore them easily.                                                                                |
| [Crafter](crafter/README.md)           | Get crafting information for items in Ark View detailed breakdowns of all ingredients needed to craft items from the game Ark Survival Evolved.                                                                                |
| [BankDecay](bankdecay/README.md)       | Inactivity-based economy credit decay with customizable settings Introduces an economy decay system to Discord servers, automatically reducing inactive users' currency to stimulate engagement. Admins can customize and oversee decay settings, fostering active participation within the community.                                                                                |
| [EmojiTracker](emojitracker/README.md) | Simple emoji tracker with leaderboards. Track reactions in your server and get leaderboards for emojis that are most used, or users that have reacted the most.                                                                                |
| [Events](events/README.md)             | Host events in your Discord. Create and manage events easily with a variety of entry types and requirements. Event submissions will be posted and counted with a winner or winners announced automatically when the event is complete.                                                                                |
| [Fluent](fluent/README.md)             | Easy to use translation tools. Translate messages or set a channel and both languages, if a message is in language 1 it gets converted to language 2 and vice versa using googles free api.                                                                                |
| [Gmail](gmail/README.md)               | Send emails from your Gmail account. Easily send emails with customizable signatures and allowed roles.                                                                                |
| [GuildLock](guildlock/README.md)       | Manage what kind of servers the bot can join. Manage what kind of servers the bot can join. A stripped down rewrite of Phen's Baron cog.                                                                                |
| [GuildLog](guildlog/README.md)         | Set a channel to log guilds the bot leaves/joins. Configure a join/leave message of your choice and whether you want to use embeds or not, the bot will log when it joins or leaves a guild. All guilds can use this cog to see what servers the bot is joining(Guild Name/Bot Name/Total Servers)                                                                                |
| [Hunting](hunting/README.md)           | Hunt birds in your server! Originally created by Paddo for red v2, then ported to v3 by aikaterna#1393, and now maintained by me. This cog allows you to 'hunt' birds in your server via reactions or text.                                                                                |
| [IdeaBoard](ideaboard/README.md)       | Make suggestions and share ideas anonymously. Allow users to create suggestions for your server. With optional anonymity and customization.                                                                                |
| [LevelUp](levelup/README.md)           | Streamlined Discord Leveling System. An intuitive full-featured leveling system with prestige features, customizable backgrounds, toggleable embed/image profiles, and extensive voice tracking options.                                                                                |                                                                                |
| [Meow](meow/README.md)                 | Meow Replaces the word "now" with "meow" in someone's latest message, if word doesnt exist in the most recent 2 messages, it sends a random cat unicode emoji. Yall have a good day meow.                                                                                |
| [NoBot](nobot/README.md)               | Filter certain messages from other bots. (ONLY checks messages from other bots), Add a bot to be filtered and a key phrase to check for. When that bot sends a message containing that phrase the message will be auto-deleted.                                                                                |
| [NoNuke](nonuke/README.md)             | A very straightforward Anti-Nuke cog. Set a cooldown and overload count(X events in X seconds), if any user with perms exceeds them, you can set an action to be taken and logged. Events include Kicks/Bans, Channel Creation/Edit/Deletion, Role Creation/Edit/Deletion. Events are not counted separately so any action taken in any order applies to the cooldown bucket.                                                    |
| [Pixl](pixl/README.md)                 | Guess pictures with friends and earn points. Start a game to have a mostly blank image pop up. Every few seconds a few blocks will show up and the goal is to guess what it is before the image is completed or time runs out. You are also competing with everyone else in the channel the game is running in!                                                                                |
| [Profiler](profiler/README.md)         | Cog profiling tools for bot owners and developers! Cog profiling tools for bot owners and developers. Attach profilers to cogs and get runtime metrics for their commands and methods.                                                                                |
| [Pupper](pupper/README.md)             | Pet the doggo! Originally created by aikaterna#1393, now maintained by me. This cog has pet that comes around on an on_message listener and waits for someone to pet it (react with a standard wave emoji), and rewards with credits. Many attributes are configurable.                                                                                |
| [SETools](setools/README.md)           | Space Engineers Server Manager! Manage your Space Engineers server remotely with various useful commands.                                                                                |
| [Tickets](tickets/README.md)           | Customizable Multi-Panel support ticket system. 'Tickets' is an easy to use, fully customizable multi-panel ticketing system with a variety of options and features including modals, logging, transcripts, multi-button panels and more!                                                                                |
| [UpgradeChat](upgradechat/README.md)   | API Integration for the Upgrade.Chat API. Allows you to add your api key and products to the bot and set a dollar to credit conversion ratio. When a user makes a purchase, they can claim it in your Discord to receive economy credits.                                                                                |
| [VrtUtils](vrtutils/README.md)         | Random utility commands. Small collection of commands used for my personal bot.                                                                                |
| [XTools](xtools/README.md)             | Access your xbox profile and much more. Various tools for Xbox using Microsoft's XSAPI. View your Xbox profile, friends, screenshots and game clips using simple commands and interactive menus. (You will need to register a Microsoft Azure application to use this cog. Type "[p]apiset help" after install for more info)                                                                                |

[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/vertyco)

# Installation

Run the following commands with your Red instance, replacing `[p]` with your prefix:

If you haven't loaded the Downloader cog already, go ahead and do that first with: `[p]load downloader`. Then,

```ini
[p]repo add vrt-cogs https://github.com/vertyco/vrt-cogs
[p]cog install vrt-cogs <list of cogs>
[p]load <list of cogs>
```

# Credits

- Obi-Wan3 for holding my hand through a lot of my issues and questions I've had in the very beginning while learning
  python, and providing me with tips for finding things by myself.
- All the Red staff members for being so damn helpful and having valuable insight.
- The Red discord coding channel for having lots of helpful cog creators!

# Contact

For help with my cogs, feel free to reach out in my support channel [#support_vrt-cogs](https://discord.com/channels/240154543684321280/1123338506396254240/1123341301212000429) in
the [Red Cog Support](https://discord.gg/GET4DVk) server.

# Contributing

If you have any suggestions, or found any bugs, please ping me in Discord (Vertyco#0117)
or [open an issue](https://github.com/vertyco/vrt-cogs/issues) on my repo!

If you would like to contribute, please talk to me on Discord first about your ideas before opening a PR.

# Translate my cogs

If you are interested in contributing to translations for my cogs, [Click Here](https://crowdin.com/project/vrt-cogs)!

# Feature Requests

I am open to ideas or suggestions for new cogs and features!

---

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=vertyco/vrt-cogs&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=vertyco/vrt-cogs&type=Date" />

  </picture>

![Forks](https://img.shields.io/github/forks/Vertyco/Vrt-Cogs?style=for-the-badge&color=9cf)
![Stars](https://img.shields.io/github/stars/Vertyco/Vrt-Cogs?style=for-the-badge&color=yellow)
[![Visitors](https://api.visitorbadge.io/api/daily?path=https%3A%2F%2Fgithub.com%2Fvertyco%2Fvrt-cogs&label=Visitors%20Today&countColor=%230015b3)](https://visitorbadge.io/status?path=https%3A%2F%2Fgithub.com%2Fvertyco%2Fvrt-cogs)
[![Visitors](https://api.visitorbadge.io/api/visitors?path=https%3A%2F%2Fgithub.com%2Fvertyco%2Fvrt-cogs&label=Total&countColor=%239085b3)](https://visitorbadge.io/status?path=https%3A%2F%2Fgithub.com%2Fvertyco%2Fvrt-cogs)

![](.github/ASSETS/Bottom_down.svg)


[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![CI](https://github.com/torchmd/torchmd-net/actions/workflows/CI.yml/badge.svg)](https://github.com/torchmd/torchmd-net/actions/workflows/CI.yml)
[![Documentation Status](https://readthedocs.org/projects/torchmd-net/badge/?version=latest)](https://torchmd-net.readthedocs.io/en/latest/?badge=latest)

# TorchMD-NET

TorchMD-NET provides state-of-the-art neural networks potentials (NNPs) and a mechanism to train them. It offers efficient and fast implementations if several NNPs and it is integrated in GPU-accelerated molecular dynamics code like [ACEMD](https://www.acellera.com/products/molecular-dynamics-software-gpu-acemd/), [OpenMM](https://www.openmm.org) and [TorchMD](https://github.com/torchmd/torchmd). TorchMD-NET exposes its NNPs as [PyTorch](https://pytorch.org) modules.

## Documentation

Documentation is available at https://torchmd-net.readthedocs.io

## Available architectures

- [Equivariant Transformer (ET)](https://arxiv.org/abs/2202.02541)
- [Transformer (T)](https://arxiv.org/abs/2202.02541)
- [Graph Neural Network (GN)](https://arxiv.org/abs/2212.07492)
- [TensorNet](https://arxiv.org/abs/2306.06482)

## Installation
TorchMD-Net is available in [conda-forge](https://conda-forge.org/) and can be installed with:
```shell
mamba install torchmd-net
```
We recommend using [Mamba](https://github.com/conda-forge/miniforge/#mambaforge) instead of conda.

### Install from source

TorchMD-Net is installed using pip, but you will need to install some dependencies before. Check [this documentation page](https://torchmd-net.readthedocs.io/en/latest/installation.html#install-from-source).

## Usage
Specifying training arguments can either be done via a configuration yaml file or through command line arguments directly. Several examples of architectural and training specifications for some models and datasets can be found in [examples/](https://github.com/torchmd/torchmd-net/tree/main/examples). Note that if a parameter is present both in the yaml file and the command line, the command line version takes precedence.
GPUs can be selected by setting the `CUDA_VISIBLE_DEVICES` environment variable. Otherwise, the argument `--ngpus` can be used to select the number of GPUs to train on (-1, the default, uses all available GPUs or the ones specified in `CUDA_VISIBLE_DEVICES`). Keep in mind that the [GPU ID reported by nvidia-smi might not be the same as the one `CUDA_VISIBLE_DEVICES` uses](https://stackoverflow.com/questions/26123252/inconsistency-of-ids-between-nvidia-smi-l-and-cudevicegetname).
For example, to train the Equivariant Transformer on the QM9 dataset with the architectural and training hyperparameters described in the paper, one can run:
```shell
mkdir output
CUDA_VISIBLE_DEVICES=0 torchmd-train --conf torchmd-net/examples/ET-QM9.yaml --log-dir output/
```
Run `torchmd-train --help` to see all available options and their descriptions.

## Pretrained models
See [here](https://github.com/torchmd/torchmd-net/tree/main/examples#loading-checkpoints) for instructions on how to load pretrained models.

## Creating a new dataset
If you want to train on custom data, first have a look at `torchmdnet.datasets.Custom`, which provides functionalities for
loading a NumPy dataset consisting of atom types and coordinates, as well as energies, forces or both as the labels.
Alternatively, you can implement a custom class according to the torch-geometric way of implementing a dataset. That is,
derive the `Dataset` or `InMemoryDataset` class and implement the necessary functions (more info [here](https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html#creating-your-own-datasets)). The dataset must return torch-geometric `Data`
objects, containing at least the keys `z` (atom types) and `pos` (atomic coordinates), as well as `y` (label), `neg_dy` (negative derivative of the label w.r.t atom coordinates) or both.

### Custom prior models
In addition to implementing a custom dataset class, it is also possible to add a custom prior model to the model. This can be
done by implementing a new prior model class in `torchmdnet.priors` and adding the argument `--prior-model <PriorModelName>`.
As an example, have a look at `torchmdnet.priors.Atomref`.

## Multi-Node Training

In order to train models on multiple nodes some environment variables have to be set, which provide all necessary information to PyTorch Lightning. In the following we provide an example bash script to start training on two machines with two GPUs each. The script has to be started once on each node. Once `torchmd-train` is started on all nodes, a network connection between the nodes will be established using NCCL.

In addition to the environment variables the argument `--num-nodes` has to be specified with the number of nodes involved during training.

```shell
export NODE_RANK=0
export MASTER_ADDR=hostname1
export MASTER_PORT=12910

mkdir -p output
CUDA_VISIBLE_DEVICES=0,1 torchmd-train --conf torchmd-net/examples/ET-QM9.yaml.yaml --num-nodes 2 --log-dir output/
```

- `NODE_RANK` : Integer indicating the node index. Must be `0` for the main node and incremented by one for each additional node.
- `MASTER_ADDR` : Hostname or IP address of the main node. The same for all involved nodes.
- `MASTER_PORT` : A free network port for communication between nodes. PyTorch Lightning suggests port `12910` as a default.

### Known Limitations
- Due to the way PyTorch Lightning calculates the number of required DDP processes, all nodes must use the same number of GPUs. Otherwise training will not start or crash.
- We observe a 50x decrease in performance when mixing nodes with different GPU architectures (tested with RTX 2080 Ti and RTX 3090).
- Some CUDA systems might hang during a multi-GPU parallel training. Try `export NCCL_P2P_DISABLE=1`, which disables direct peer to peer GPU communication.

## Cite
If you use TorchMD-NET in your research, please cite the following papers:

#### Main reference

```
@misc{pelaez2024torchmdnet,
title={TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulations},
author={Raul P. Pelaez and Guillem Simeon and Raimondas Galvelis and Antonio Mirarchi and Peter Eastman and Stefan Doerr and Philipp Thölke and Thomas E. Markland and Gianni De Fabritiis},
year={2024},
eprint={2402.17660},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```

#### TensorNet

```
@inproceedings{simeon2023tensornet,
title={TensorNet: Cartesian Tensor Representations for Efficient Learning of Molecular Potentials},
author={Guillem Simeon and Gianni De Fabritiis},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
url={https://openreview.net/forum?id=BEHlPdBZ2e}
}
```

#### Equivariant Transformer
```
@inproceedings{
tholke2021equivariant,
title={Equivariant Transformers for Neural Network based Molecular Potentials},
author={Philipp Th{\"o}lke and Gianni De Fabritiis},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=zNHzqZ9wrRB}
}
```

#### Graph Network

```
@article{Majewski2023,
  title = {Machine learning coarse-grained potentials of protein thermodynamics},
  volume = {14},
  ISSN = {2041-1723},
  url = {http://dx.doi.org/10.1038/s41467-023-41343-1},
  DOI = {10.1038/s41467-023-41343-1},
  number = {1},
  journal = {Nature Communications},
  publisher = {Springer Science and Business Media LLC},
  author = {Majewski,  Maciej and Pérez,  Adrià and Th\"{o}lke,  Philipp and Doerr,  Stefan and Charron,  Nicholas E. and Giorgino,  Toni and Husic,  Brooke E. and Clementi,  Cecilia and Noé,  Frank and De Fabritiis,  Gianni},
  year = {2023},
  month = sep
}
```

## Developer guide

### Implementing a new architecture

To implement a new architecture, you need to follow these steps:
 **1.** Create a new class in `torchmdnet.models` that inherits from `torch.nn.Model`. Follow TorchMD_ET as a template. This is a minimum implementation of a model:
```python
class MyModule(nn.Module):
  def __init__(self, parameter1, parameter2):
	super(MyModule, self).__init__()
	# Define your model here
	self.layer1 = nn.Linear(10, 10)
	...
	# Initialize your model parameters here
	self.reset_parameters()

    def reset_parameters(self):
      # Initialize your model parameters here
	  nn.init.xavier_uniform_(self.layer1.weight)
	...

  def forward(self,
        z: Tensor, # Atomic numbers, shape (n_atoms, 1)
        pos: Tensor, # Atomic positions, shape (n_atoms, 3)
        batch: Tensor, # Batch vector, shape (n_atoms, 1). All atoms in the same molecule have the same value and are contiguous.
        q: Optional[Tensor] = None, # Atomic charges, shape (n_atoms, 1)
        s: Optional[Tensor] = None, # Atomic spins, shape (n_atoms, 1)
    ) -> Tuple[Tensor, Tensor, Tensor, Tensor, Tensor]:
	# Define your forward pass here
	scalar_features = ...
	vector_features = ...
	# Return the scalar and vector features, as well as the atomic numbers, positions and batch vector
	return scalar_features, vector_features, z, pos, batch
```
 **2.** Add the model to the `__all__` list in `torchmdnet.models.__init__.py`. This will make the tests pick your model up.
 **3.** Tell models.model.create_model how to initialize your module by adding a new entry, for instance:
 ```python
     elif args["model"] == "mymodule":
        from torchmdnet.models.torchmd_mymodule import MyModule
        is_equivariant = False # Set to True if your model is equivariant
        representation_model = MyModule(
            parameter1=args["parameter1"],
            parameter2=args["parameter2"],
            **shared_args, # Arguments typically shared by all models
        )
 ```

 **4.** Add any new parameters required to initialize your module to scripts.train.get_args. For instance:
 ```python
   parser.add_argument('--parameter1', type=int, default=32, help='Parameter1 required by MyModule')
   ...
 ```
 **5.** Add an example configuration file to `torchmd-net/examples` that uses your model.
 **6.** Make tests use your configuration file by adding a case to tests.utils.load_example_args. For instance:
 ```python
 if model_name == "mymodule":
        config_file = join(dirname(dirname(__file__)), "examples", "MyModule-QM9.yaml")
 ```

At this point, if your module is missing some feature the tests will let you know, and you can add it. If you add a new feature to the package, please add a test for it.

### Code style

We use [black](https://https://black.readthedocs.io/en/stable/). Please run `black` on your modified files before committing.

### Testing

To run the tests, install the package and run `pytest` in the root directory of the repository. Tests are a good source of knowledge on how to use the different components of the package.


**Before doing anything, please read
[Leapp framework documentation](https://leapp.readthedocs.io/).**

---

## Troubleshooting

### Where can I report an issue or RFE related to the framework or other actors?

- GitHub issues are preferred:
  - Leapp framework: [https://github.com/oamg/leapp/issues/new/choose](https://github.com/oamg/leapp/issues/new/choose)
  - Leapp actors: [https://github.com/oamg/leapp-repository/issues/new/choose](https://github.com/oamg/leapp-repository/issues/new/choose)

- When filing an issue, include:
  - Steps to reproduce the issue
  - *All files in /var/log/leapp*
  - */var/lib/leapp/leapp.db*
  - *journalctl*
  - If you want, you can optionally send anything else would you like to provide (e.g. storage info)

**For your convenience you can pack all logs with this command:**

`# tar -czf leapp-logs.tgz /var/log/leapp /var/lib/leapp/leapp.db`

Then you may attach only the `leapp-logs.tgz` file.

### Where can I seek help?
We’ll gladly answer your questions and lead you to through any troubles with the
actor development.

You can reach us at IRC: `#leapp` on Libera.Chat.


[![logo](https://raw.githubusercontent.com/aws-ia/taskcat/main/assets/docs/images/tcat.png)](https://taskcat.io/)
[![Build Status](https://travis-ci.com/aws-ia/taskcat.svg?branch=main)](https://travis-ci.com/aws-ia/taskcat) [![PyPI version](https://badge.fury.io/py/taskcat.svg)](https://badge.fury.io/py/taskcat) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

## What is TaskCat?
**TaskCat** is a tool that tests AWS CloudFormation templates. It deploys your AWS
CloudFormation template in multiple AWS Regions and generates a report with a pass/fail
grade for each region. You can specify the regions and number of Availability Zones you
want to include in the test, and pass in parameter values from your AWS CloudFormation
template. TaskCat is implemented as a Python class that you import, instantiate, and run.

TaskCat was developed by the aws-ia team to test AWS CloudFormation templates
that automatically deploy workloads on AWS. We’re pleased to make the tool available to
all developers who want to validate their custom AWS CloudFormation templates across
AWS Regions

__See [TaskCat documentation](https://aws-ia.github.io/taskcat/).__

## Support
[![Feature Request](https://img.shields.io/badge/Open%20Issues-Feature%20Request-green.svg)](https://github.com/aws-ia/taskcat/issues/new/choose)
[![Report Bugs](https://img.shields.io/badge/Open%20Issue-Report%20Bug-red.svg)](https://github.com/aws-ia/taskcat/issues/new/choose)

## GitHub

[![GitHub stars](https://img.shields.io/github/stars/aws-ia/taskcat.svg?style=social&label=Stars)](https://github.com/aws-ia/taskcat)
[![GitHub issues](https://img.shields.io/github/issues/aws-ia/taskcat.svg)](https://github.com/aws-ia/taskcat/issues)
[![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/aws-ia/taskcat.svg)](https://github.com/aws-ia/taskcat/issues?q=is%3Aissue+is%3Aclosed)
[![GitHub pull requests](https://img.shields.io/github/issues-pr/aws-ia/taskcat.svg)](https://github.com/aws-ia/taskcat/pulls)
[![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed-raw/aws-ia/taskcat.svg)](https://github.com/aws-ia/taskcat/pulls?q=is%3Apr+is%3Aclosed)

## PyPi

[![PyPI - Downloads](https://img.shields.io/pypi/dw/taskcat.svg)](https://pypi.org/project/taskcat/#history)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/taskcat.svg)](https://pypi.org/project/taskcat/#history)


## [NVTabular](https://github.com/NVIDIA/NVTabular)

[![PyPI](https://img.shields.io/pypi/v/NVTabular?color=orange&label=version)](https://pypi.python.org/pypi/NVTabular/)
[![LICENSE](https://img.shields.io/github/license/NVIDIA-Merlin/NVTabular)](https://github.com/NVIDIA-Merlin/NVTabular/blob/stable/LICENSE)
[![Documentation](https://img.shields.io/badge/documentation-blue.svg)](https://nvidia-merlin.github.io/NVTabular/stable/Introduction.html)

NVTabular is a feature engineering and preprocessing library for tabular data that is designed to easily manipulate terabyte scale datasets and train deep learning (DL) based recommender systems. It provides high-level abstraction to simplify code and accelerates computation on the GPU using the [RAPIDS Dask-cuDF](https://github.com/rapidsai/cudf/tree/main/python/dask_cudf) library.

NVTabular is a component of [NVIDIA Merlin](https://developer.nvidia.com/nvidia-merlin), an open source framework for building and deploying recommender systems and works with the other Merlin components including [Merlin Models](https://github.com/NVIDIA-Merlin/models), [HugeCTR](https://github.com/NVIDIA/HugeCTR) and [Merlin Systems](https://github.com/NVIDIA-Merlin/systems) to provide end-to-end acceleration of recommender systems on the GPU. Extending beyond model training, with NVIDIA’s [Triton Inference Server](https://github.com/NVIDIA/tensorrt-inference-server), the feature engineering and preprocessing steps performed on the data during training can be automatically applied to incoming data during inference.

<!-- <img src='https://developer.nvidia.com/blog/wp-content/uploads/2020/07/recommender-system-training-pipeline-1.png'/> -->

### Benefits

When training DL recommender systems, data scientists and machine learning (ML) engineers have been faced with the following challenges:

- **Huge Datasets**: Commercial recommenders are trained on huge datasets that may be several terabytes in scale.
- **Complex Data Feature Engineering and Preprocessing Pipelines**: Datasets need to be preprocessed and transformed so that they can be used with DL models and frameworks. In addition, feature engineering creates an extensive set of new features from existing ones, requiring multiple iterations to arrive at an optimal solution.
- **Input Bottleneck**: Data loading, if not well optimized, can be the slowest part of the training process, leading to under-utilization of high-throughput computing devices such as GPUs.
- **Extensive Repeated Experimentation**: The entire data engineering, training, and evaluation process can be repetitious and time consuming, requiring significant computational resources.

NVTabular alleviates these challenges and helps data scientists and ML engineers:

- process datasets that exceed GPU and CPU memory without having to worry about scale.
- focus on what to do with the data and not how to do it by using abstraction at the operation level.
- prepare datasets quickly and easily for experimentation so that more models can be trained.
- deploy models into production by providing faster dataset transformation

Learn more in the NVTabular [core features documentation](https://nvidia-merlin.github.io/NVTabular/stable/core_features.html).

### Performance

When running NVTabular on the Criteo 1TB Click Logs Dataset using a single V100 32GB GPU, feature engineering and preprocessing was able to be completed in 13 minutes. Furthermore, when running NVTabular on a DGX-1 cluster with eight V100 GPUs, feature engineering and preprocessing was able to be completed within three minutes. Combined with [HugeCTR](http://www.github.com/NVIDIA/HugeCTR/), the dataset can be processed and a full model can be trained in only six minutes.

The performance of the Criteo DRLM workflow also demonstrates the effectiveness of the NVTabular library. The original ETL script provided in Numpy took over five days to complete. Combined with CPU training, the total iteration time is over one week. By optimizing the ETL code in Spark and running on a DGX-1 equivalent cluster, the time to complete feature engineering and preprocessing was reduced to three hours. Meanwhile, training was completed in one hour.

### Installation

NVTabular requires Python version 3.7+. Additionally, GPU support requires:

- CUDA version 11.0+
- NVIDIA Pascal GPU or later (Compute Capability >=6.0)
- NVIDIA driver 450.80.02+
- Linux or WSL

#### Installing NVTabular Using Conda

NVTabular can be installed with Anaconda from the `nvidia` channel by running the following command:

```
conda install -c nvidia -c rapidsai -c numba -c conda-forge nvtabular python=3.7 cudatoolkit=11.2
```

#### Installing NVTabular Using Pip

NVTabular can be installed with `pip` by running the following command:

```
pip install nvtabular
```

> Installing NVTabular with Pip causes NVTabular to run on the CPU only and might require installing additional dependencies manually.
> When you run NVTabular in one of our Docker containers, the dependencies are already installed.

#### Installing NVTabular with Docker

NVTabular Docker containers are available in the [NVIDIA Merlin container
repository](https://catalog.ngc.nvidia.com/?filters=&orderBy=scoreDESC&query=merlin).
The following table summarizes the key information about the containers:

| Container Name    | Container Location                                                                   | Functionality                              |
| ----------------- | ------------------------------------------------------------------------------------ | ------------------------------------------ |
| merlin-hugectr    | https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr    | NVTabular, HugeCTR, and Triton Inference   |
| merlin-tensorflow | https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow | NVTabular, Tensorflow and Triton Inference |
| merlin-pytorch    | https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-pytorch    | NVTabular, PyTorch, and Triton Inference   |

To use these Docker containers, you'll first need to install the [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker) to provide GPU support for Docker. You can use the NGC links referenced in the table above to obtain more information about how to launch and run these containers. To obtain more information about the software and model versions that NVTabular supports per container, see [Support Matrix](https://github.com/NVIDIA/NVTabular/blob/stable/docs/source/resources/support_matrix.rst).

### Notebook Examples and Tutorials

We provide a [collection of examples](https://github.com/NVIDIA-Merlin/NVTabular/tree/stable/examples) to demonstrate feature engineering with NVTabular as Jupyter notebooks:

- Introduction to NVTabular's High-Level API
- Advanced workflows with NVTabular
- NVTabular on CPU
- Scaling NVTabular to multi-GPU systems

In addition, NVTabular is used in many of our examples in other Merlin libraries:

- [End-To-End Examples with Merlin](https://github.com/NVIDIA-Merlin/Merlin/tree/stable/examples)
- [Training Examples with Merlin Models](https://github.com/NVIDIA-Merlin/models/tree/stable/examples)
- [Training Examples with Transformer4Rec](https://github.com/NVIDIA-Merlin/Transformers4Rec/tree/stable/examples)

### Feedback and Support

If you'd like to contribute to the library directly, see the [Contributing.md](https://github.com/NVIDIA/NVTabular/blob/stable/CONTRIBUTING.md). We're particularly interested in contributions or feature requests for our feature engineering and preprocessing operations. To further advance our Merlin Roadmap, we encourage you to share all the details regarding your recommender system pipeline in this [survey](https://developer.nvidia.com/merlin-devzone-survey).

If you're interested in learning more about how NVTabular works, see
[our NVTabular documentation](https://nvidia-merlin.github.io/NVTabular/stable/Introduction.html). We also have [API documentation](https://nvidia-merlin.github.io/NVTabular/stable/api/index.html) that outlines the specifics of the available calls within the library.


s3fs
====

|Build Status| |Doc Status|

S3FS builds on aiobotocore_ to provide a convenient Python filesystem interface for S3.

View the documentation_ for s3fs.

.. _documentation: http://s3fs.readthedocs.io/en/latest/
.. _aiobotocore: https://aiobotocore.readthedocs.io/en/latest/

.. |Build Status| image:: https://github.com/fsspec/s3fs/workflows/CI/badge.svg
    :target: https://github.com/fsspec/s3fs/actions
    :alt: Build Status
.. |Doc Status| image:: https://readthedocs.org/projects/s3fs/badge/?version=latest
    :target: https://s3fs.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status


# filesystem_spec

[![PyPI version](https://badge.fury.io/py/fsspec.svg)](https://pypi.python.org/pypi/fsspec/)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/fsspec/badges/version.svg)](https://anaconda.org/conda-forge/fsspec)
![Build](https://github.com/fsspec/filesystem_spec/workflows/CI/badge.svg)
[![Docs](https://readthedocs.org/projects/filesystem-spec/badge/?version=latest)](https://filesystem-spec.readthedocs.io/en/latest/?badge=latest)
[![fsspec Downloads Last Month](https://assets.piptrends.com/get-last-month-downloads-badge/fsspec.svg 'fsspec Downloads Last Month by pip Trends')](https://piptrends.com/package/fsspec)

A specification for pythonic filesystems.

## Install

```bash
pip install fsspec
```

would install the base fsspec. Various optionally supported features might require specification of custom
extra require, e.g. `pip install fsspec[ssh]` will install dependencies for `ssh` backends support.
Use `pip install fsspec[full]` for installation of all known extra dependencies.

Up-to-date package also provided through conda-forge distribution:

```bash
conda install -c conda-forge fsspec
```

## Purpose

To produce a template or specification for a file-system interface, that specific implementations should follow,
so that applications making use of them can rely on a common behaviour and not have to worry about the specific
internal implementation decisions with any given backend. Many such implementations are included in this package,
or in sister projects such as `s3fs` and `gcsfs`.

In addition, if this is well-designed, then additional functionality, such as a key-value store or FUSE
mounting of the file-system implementation may be available for all implementations "for free".

## Documentation

Please refer to [RTD](https://filesystem-spec.readthedocs.io/en/latest/?badge=latest)

## Develop

fsspec uses GitHub Actions for CI. Environment files can be found
in the "ci/" directory. Note that the main environment is called "py38",
but it is expected that the version of python installed be adjustable at
CI runtime. For local use, pick a version suitable for you.

```bash
# For a new environment (mamba / conda).
mamba create -n fsspec -c conda-forge  python=3.9 -y
conda activate fsspec

# Standard dev install with docs and tests.
pip install -e ".[dev,doc,test]"

# Full tests except for downstream
pip install s3fs
pip uninstall s3fs
pip install -e .[dev,doc,test_full]
pip install s3fs --no-deps
pytest -v

# Downstream tests.
sh install_s3fs.sh
# Windows powershell.
install_s3fs.sh
```

### Testing

Tests can be run in the dev environment, if activated, via ``pytest fsspec``.

The full fsspec suite requires a system-level docker, docker-compose, and fuse
installation. If only making changes to one backend implementation, it is
not generally necessary to run all tests locally.

It is expected that contributors ensure that any change to fsspec does not
cause issues or regressions for either other fsspec-related packages such
as gcsfs and s3fs, nor for downstream users of fsspec. The "downstream" CI
run and corresponding environment file run a set of tests from the dask
test suite, and very minimal tests against pandas and zarr from the
test_downstream.py module in this repo.

### Code Formatting

fsspec uses [Black](https://black.readthedocs.io/en/stable) to ensure
a consistent code format throughout the project.
Run ``black fsspec`` from the root of the filesystem_spec repository to
auto-format your code. Additionally, many editors have plugins that will apply
``black`` as you edit files. ``black`` is included in the ``tox`` environments.

Optionally, you may wish to setup [pre-commit hooks](https://pre-commit.com) to
automatically run ``black`` when you make a git commit.
Run ``pre-commit install --install-hooks`` from the root of the
filesystem_spec repository to setup pre-commit hooks. ``black`` will now be run
before you commit, reformatting any changed files. You can format without
committing via ``pre-commit run`` or skip these checks with ``git commit
--no-verify``.


# T5X

*Go to [T5X ReadTheDocs Documentation Page](https://t5x.readthedocs.io/).*

T5X is a modular, composable, research-friendly framework for high-performance,
configurable, self-service training, evaluation, and inference of sequence
models (starting with language) at many scales.

It is essentially a new and improved implementation of the
[T5 codebase](https://github.com/google-research/text-to-text-transfer-transformer)
(based on [Mesh TensorFlow](https://github.com/tensorflow/mesh)) in [JAX](https://github.com/google/jax) and [Flax](https://github.com/google/flax). To learn
more, see the [T5X Paper](https://arxiv.org/abs/2203.17189).

Below is a quick start guide for training models with TPUs on Google Cloud. For
additional tutorials and background, see the [complete documentation](docs/index.md).

## Quickstart (Recommended)

T5X can be run with [XManager](https://github.com/deepmind/xmanager) on
[Vertex AI](https://cloud.google.com/vertex-ai). Vertex AI is a platform for
training that creates TPU instances and runs code on the TPUs. Vertex AI will
also shut down the TPUs when the jobs terminate. This is signifcantly easier
than managing GCE VMs and TPU VM instances.

1. Follow the pre-requisites and directions to install [XManager](https://github.com/deepmind/xmanager).

2. Request TPU quota as required. GCP projects come with 8 cores by default,
which is enough to run one training experiment on a single TPU host. If you want
to run multi-host training or run multiple trials in parallel, you will need
more quota. Navigate to [Quotas](https://console.cloud.google.com/quotas).

  The quota you want is:

  * Service: `Vertex AI API`
  * Dimensions (location): `us-central1`
  * If you want to run single-host experiments:
    * `Custom model training TPU V2 cores per region`
    * `Custom model training TPU V3 cores per region`
  * If you want to run multi-host experiments:
    * `Custom model training TPU V2 pod cores per region`
    * `Custom model training TPU V3 pod cores per region`

  TIP: You won't be able to run single-host experiments with multi-host quota.
  (i.e. you can't run `tpu_v2=8` using `TPU V2 pod`)

3. Launch the xmanager script located at `t5x/scripts/xm_launch.py`.

As a running example, we use the WMT14 En-De translation which is described in
more detail in the Examples section below.

```sh
export GOOGLE_CLOUD_BUCKET_NAME=...
export TFDS_DATA_DIR=gs://$GOOGLE_CLOUD_BUCKET_NAME/t5x/data
export MODEL_DIR=gs://$GOOGLE_CLOUD_BUCKET_NAME/t5x/$(date +%Y%m%d)

# Pre-download dataset in multi-host experiments.
tfds build wmt_t2t_translate --data_dir=$TFDS_DATA_DIR

git clone https://github.com/google-research/t5x
cd ./t5x/

python3 ./t5x/scripts/xm_launch.py \
  --gin_file=t5x/examples/t5/t5_1_1/examples/base_wmt_from_scratch.gin \
  --model_dir=$MODEL_DIR \
  --tfds_data_dir=$TFDS_DATA_DIR
```

Check `gs://$GOOGLE_CLOUD_BUCKET_NAME/t5x/` for the output artifacts, which can
be read by TensorBoard.

## GPU Usage
Note: NVIDIA has released an updated version of this repository with H100 FP8 support and broad GPU performance improvements. Please visit the [NVIDIA Rosetta](https://github.com/NVIDIA/JAX-Toolbox/tree/main/rosetta/rosetta/projects/t5x) repository for more details and usage instructions.

T5X can be run easily on GPUs either in single-node configurations or multi-node configurations with a SLURM+pyxis cluster. Further instructions at [t5x/contrib/gpu](https://github.com/google-research/t5x/blob/main/t5x/contrib/gpu/README.md). The `t5x/contrib/gpu/scripts_gpu` folder contains example scripts for pretraining T5X on [The Pile](https://pile.eleuther.ai/) and for finetuning on SQuAD and MNLI. These scripts and associated `gin` configurations also contain additional GPU optimizations for better throughput. More examples and instructions can be found in the [NVIDIA Rosetta](https://github.com/NVIDIA/JAX-Toolbox/tree/main/rosetta/rosetta/projects/t5x) repository maintained by NVIDIA with H100 FP8 support and broad GPU performance improvements.

## Installation

Note that all the commands in this document should be run in the commandline of
the TPU VM instance unless otherwise stated.

1.  Follow the
    [instructions](https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm#install_the_google_cloud_sdk)
    to set up a Google Cloud Platform (GCP) account and enable the Cloud TPU
    API.

    **Note:** T5X also works with GPU, please follow instructions in [t5x/contrib/gpu](https://github.com/google-research/t5x/blob/main/t5x/contrib/gpu/README.md) if you'd like to use GPU version.

2.  Create a
    [Cloud TPU VM instance](https://cloud.google.com/blog/products/compute/introducing-cloud-tpu-vms)
    following
    [this instruction](https://cloud.google.com/tpu/docs/jax-quickstart-tpu-vm#create-vm).
    We recommend that you develop your workflow in a single v3-8 TPU (i.e.,
    `--accelerator-type=v3-8`) and scale up to pod slices once the pipeline is
    ready. In this README, we focus on using a single v3-8 TPU. See
    [here](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm) to
    learn more about TPU architectures.

3.  With Cloud TPU VMs, you ssh directly into the host machine of the TPU VM.
    You can install packages, run your code run, etc. in the host machine. Once
    the TPU instance is created, ssh into it with

    ```sh
    gcloud alpha compute tpus tpu-vm ssh ${TPU_NAME} --zone=${ZONE}
    ```

    where `TPU_NAME` and `ZONE` are the name and the zone used in step 2.

4.  Install T5X and the dependencies.

    ```sh
    git clone --branch=main https://github.com/google-research/t5x
    cd t5x

    python3 -m pip install -e '.[tpu]' -f \
      https://storage.googleapis.com/jax-releases/libtpu_releases.html

    ```

5.  Create Google Cloud Storage (GCS) bucket to store the dataset and model
    checkpoints. To create a GCS bucket, see these
    [instructions](https://cloud.google.com/storage/docs/creating-buckets).

6.  (optional) If you prefer working with Jupyter/Colab style environment
    you can setup a custom Colab runtime by following steps from
    [t5x/notebooks](https://github.com/google-research/t5x/blob/main/t5x/notebooks/README.md).

## Example: English to German translation

As a running example, we use the WMT14 En-De translation. The raw dataset is
available in TensorFlow Datasets as
["wmt_t2t_translate"](https://www.tensorflow.org/datasets/catalog/wmt_t2t_translate).

T5 casts the translation task such as the following

```py
{'en': 'That is good.', 'de': 'Das ist gut.'}
```

to the form called "text-to-text":

```py
{'inputs': 'translate English to German: That is good.', 'targets': 'Das ist gut.'}
```

This formulation allows many different classes of language tasks to be expressed
in a uniform manner and a single encoder-decoder architecture can handle them
without any task-specific parameters. For more detail, refer to the [T5 paper
(Raffel et al. 2019)][t5_paper].

For a scalable data pipeline and an evaluation framework, we use
[`SeqIO`](https://github.com/google/seqio), which was factored out of the [T5
library][t5_github]. A `seqio.Task` packages together the raw dataset, vocabulary,
preprocessing such as tokenization and evaluation metrics such as
[BLEU](https://aclanthology.org/P02-1040.pdf) and provides a
[`tf.data`](https://www.tensorflow.org/guide/data) instance.

[The T5 library][t5_github] provides a number of `seqio.Task`s that were used in the
[T5 paper][t5_paper]. In this example, we use [wmt_t2t_ende_v003](https://github.com/google-research/text-to-text-transfer-transformer/blob/d81c0bab2a41b4d5dfbe4971de32f7d67df65f31/t5/data/tasks.py#L212).

Before training or fine-tuning you need to download ["wmt_t2t_translate"]
(https://www.tensorflow.org/datasets/catalog/wmt_t2t_translate) dataset first.

```sh
# Data dir to save the processed dataset in "gs://data_dir" format.
TFDS_DATA_DIR="..."

# Make sure that dataset package is up-to-date.
python3 -m pip install --upgrade tfds-nightly

# Pre-download dataset.
tfds build wmt_t2t_translate ${TFDS_DATA_DIR}
```

### Training

To run a training job, we use the `t5x/train.py` script.

```sh
# Model dir to save logs, ckpts, etc. in "gs://model_dir" format.
MODEL_DIR="..."
T5X_DIR="..."  # directory where the T5X repo is cloned.
TFDS_DATA_DIR="..."

python3 ${T5X_DIR}/t5x/train.py \
  --gin_file="t5x/examples/t5/t5_1_1/examples/base_wmt_from_scratch.gin" \
  --gin.MODEL_DIR=\"${MODEL_DIR}\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

The configuration for this training run is defined in the Gin file
[base_wmt_from_scratch.gin](t5x/examples/t5/t5_1_1/examples/base_wmt_from_scratch.gin).
[Gin-config](https://github.com/google/gin-config) is a library to handle
configurations based on dependency injection. Among many benefits, Gin allows
users to pass custom components such as a custom model to the T5X library
without having to modify the core library. The [custom
components](#custom-components) section shows how this is done.

While the core library is independent of Gin, it is central to the examples we
provide. Therefore, we provide a short [introduction][gin-primer] to Gin in the
context of T5X.  All the configurations are written to a file "config.gin" in
`MODEL_DIR`. This makes debugging as well as reproducing the experiment much
easier.

In addition to the `config.json`, `model-info.txt` file summarizes the model
parameters (shape, names of the axes, partitioning info) as well as the
optimizer states.

#### TensorBoard

To monitor the training in [TensorBoard](https://www.tensorflow.org/tensorboard), it is much easier (due to
authentification issues) to launch the TensorBoard on your own machine and _not_ in
the TPU VM. So in the commandline where you ssh'ed into the TPU VM, launch the
TensorBoard with the `logdir` pointing to the `MODEL_DIR`.

```sh
# NB: run this on your machine not TPU VM!
MODEL_DIR="..."  # Copy from the TPU VM.
tensorboard --logdir=${MODEL_DIR}
```

Or you can launch the TensorBoard inside a Colab. In a Colab cell, run

```python
from google.colab import auth
auth.authenticate_user()
```

to authorize the Colab to access the GCS bucket and launch the TensorBoard.

```python
%load_ext tensorboard
model_dir = "..."  # Copy from the TPU VM.
%tensorboard --logdir=model_dir
```

### Fine-tuning

We can leverage the benefits of self-supervised pre-training by initializing
from one of our pre-trained models. Here we use the T5.1.1 Base checkpoint.

```sh
# Model dir to save logs, ckpts, etc. in "gs://model_dir" format.
MODEL_DIR="..."

# Data dir to save the processed dataset in "gs://data_dir" format.
TFDS_DATA_DIR="..."
T5X_DIR="..."  # directory where the T5X repo is cloned.

python3 ${T5X_DIR}/t5x/train.py \
  --gin_file="t5x/examples/t5/t5_1_1/examples/base_wmt_finetune.gin" \
  --gin.MODEL_DIR=\"${MODEL_DIR}\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

**Note:** when supplying a string, dict, list, tuple value, or a bash variable
via a flag, you must put it in quotes. In the case of strings, it requires
escaped quotes (`\"<string>\"`). For example:
`--gin.utils.DatasetConfig.split=\"validation\"` or
`--gin.MODEL_DIR=\"${MODEL_DIR}\"`.

Gin makes it easy to change a number of configurations. For example, you can
change the `partitioning.PjitPartitioner.num_partitions` (overriding
the value in
[base_wmt_from_scratch.gin](t5x/examples/t5/t5_1_1/examples/base_wmt_from_scratch.gin))
to chanage the parallelism strategy and pass it as a commandline arg.

```sh
--gin.partitioning.PjitPartitioner.num_partitions=8
```

### Evaluation

To run the offline (i.e. without training) evaluation, you can use `t5x/eval.py`
script.

```sh
EVAL_OUTPUT_DIR="..."  # directory to write eval output
T5X_DIR="..."  # directory where the t5x is cloned, e.g., ${HOME}"/t5x".
TFDS_DATA_DIR="..."
CHECKPOINT_PATH="..."

python3 ${T5X_DIR}/t5x/eval.py \
  --gin_file="t5x/examples/t5/t5_1_1/examples/base_wmt_eval.gin" \
  --gin.CHECKPOINT_PATH=\"${CHECKPOINT_PATH}\" \
  --gin.EVAL_OUTPUT_DIR=\"${EVAL_OUTPUT_DIR}\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

### Inference

To run inference, you can use `t5x/infer.py` script. Here we use the same
`seqio.Task`, but for inference we do not use the targets features other than
logging them alongside the prediction in a JSON file.

```sh
INFER_OUTPUT_DIR="..."  # directory to write infer output
T5X_DIR="..."  # directory where the t5x is cloned, e.g., ${HOME}"/t5x".
TFDS_DATA_DIR="..."
CHECKPOINT_PATH="..."

python3 ${T5X_DIR}/t5x/infer.py \
  --gin_file="t5x/examples/t5/t5_1_1/examples/base_wmt_infer.gin" \
  --gin.CHECKPOINT_PATH=\"${CHECKPOINT_PATH}\" \
  --gin.INFER_OUTPUT_DIR=\"${INFER_OUTPUT_DIR}\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

### Exporting as TensorFlow Saved Model

Pretrained model can be exported as TensorFlow Saved Model, and deployed
to Vertex AI Prediction service using [Optimized TensorFlow Runtime]
(https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime).
Please note that exported model won't work on OSS based
[TensorFlow Model Server](https://github.com/tensorflow/serving).

```sh
T5X_DIR="..."  # directory where the t5x is cloned, e.g., ${HOME}"/t5x".
CHECKPOINT_PATH="..."

BATCH_SIZE=None
BEAM_SIZE=1

# Use 'bfloat16' if you plan to run exported model on NVIDIA A100 or newer GPUs,
# for other GPUs use 'float32'.
ACTIVATION_DTYPE=bfloat16

# Version numbers must be numeric. We generate one based on datetime.
VERSION=$(date +%Y%m%d%H%M%S)

NAME=t5x_base_${ACTIVATION_DTYPE}  # Model name.

# Path to export model to. Note that export script is going to add _cpu suffix
# after model name.
OUTPUT=${CHECKPOINT_PATH}/saved_model.${NAME}/${VERSION}

declare -a ARGS=(
--gin_file=t5x/examples/t5/t5_1_1/base.gin
--gin_file=t5x/t5x/configs/runs/export.gin
--gin.TASK_FEATURE_LENGTHS="{'inputs': 256, 'targets': 256}"
--gin.CHECKPOINT_PATH=\"${CHECKPOINT_PATH}\"
--gin.MODEL_NAME=\"/ml/${USER}/t5x_base\"
--gin.MODEL_OUTPUT_DIR=\"${OUTPUT}\"
--gin.BEAM_SIZE=${BEAM_SIZE}
--gin.BATCH_SIZE=${BATCH_SIZE}
--gin.export_lib.save.partitioner=None
--gin.export_lib.save.warmup_examples="['hello world']"
--gin.export_lib.ExportableModule.use_batch_function=False
--gin.export_lib.ExportableModule.use_gpu=False
--gin.export_lib.ExportableModule.jit_compile=False
--gin.ACTIVATION_DTYPE=\"${ACTIVATION_DTYPE}\"
--gin.network.T5Config.dtype=\"${ACTIVATION_DTYPE}\"
--gin.utils.RestoreCheckpointConfig.dtype=\"${ACTIVATION_DTYPE}\"
--gin.DROPOUT_RATE=0.0
)

(python3 ${T5X_DIR}/t5x/export.py "${ARGS[@]}")
```

For detailed arguments definition refer to [export.gin]
(t5x/configs/runs/export.gin).

You can run XL and smaller models on NVIDIA A100 40GB, and XXL models on
NVIDIA A100 80GB.

## Custom components

[The translation example](#example-english-to-german-translation) uses the
encoder-decoder model that T5X provides as well as the dataset from the T5
library. This section shows how you can use your own dataset and a model and
pass via Gin.

### Example: custom dataset in a user directory

For this example, we have the following directory structure with
`${HOME}/dir1/user_dir` representing a user directory with custom components.

```
${HOME}
└── dir1
    └── user_dir
        ├── t5_1_1_base_de_en.gin
        └── tasks.py
```

As an example, let's define a new dataset. Here we use the same Translation
dataset but we define the translation task in the opposite direction, i.e.,
German to English intead of English to German. We define this task in `tasks.py`

```py
# ${HOME}/dir1/user_dir/tasks.py

import functools
import seqio
import tensorflow_datasets as tfds
from t5.evaluation import metrics
from t5.data import preprocessors

vocabulary = seqio.SentencePieceVocabulary(
    'gs://t5-data/vocabs/cc_all.32000/sentencepiece.model', extra_ids=100)
output_features = {
    'inputs': seqio.Feature(vocabulary=vocabulary),
    'targets': seqio.Feature(vocabulary=vocabulary)
}

seqio.TaskRegistry.add(
    'wmt_t2t_de_en_v003',
    source=seqio.TfdsDataSource(tfds_name='wmt_t2t_translate/de-en:1.0.0'),
    preprocessors=[
        functools.partial(
            preprocessors.translate,
            source_language='de', target_language='en'),
        seqio.preprocessors.tokenize,
        seqio.CacheDatasetPlaceholder(),
        seqio.preprocessors.append_eos_after_trim,
    ],
    metric_fns=[metrics.bleu],
    output_features=output_features)
```

In the Gin file, most of the settings are equivalent to those used in the
[En->De example](#example-english-to-german-translation). So we include the Gin
file from that example. To use "wmt_t2t_de_en_v003" task we just defined, we
need to import the task module "tasks.py". Note that we use a relative path
defined with respect to the user directory. This will be specified as a
flag.

```py
# ${HOME}/dir1/user_dir/t5_1_1_base_de_en.gin
from __gin__ import dynamic_registration
import tasks  # This imports the task defined in dir1/user_dir/tasks.py.

include "t5x-tmp/t5x/examples/t5/t5_1_1/examples/base_wmt_from_scratch.gin"
MIXTURE_OR_TASK_NAME = "wmt_t2t_de_en_v003"
```

Finally, we launch training passing the user directory as a flag
`gin_search_paths` such that the Gin file and python modules can be specified
with relative paths.

```sh
PROJECT_DIR=${HOME}"/dir1/user_dir"
T5X_DIR="..."  # directory where the t5x is cloned.
TFDS_DATA_DIR="..."
MODEL_DIR="..."
export PYTHONPATH=${PROJECT_DIR}

python3 ${T5X_DIR}/t5x/train.py \
  --gin_search_paths=${PROJECT_DIR} \
  --gin_file="t5_1_1_base_de_en.gin" \
  --gin.MODEL_DIR=\"${MODEL_DIR}\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

## Checkpoints

### Native Checkpoints

We have released the checkpoints of many of the original T5 models and their
variants a native T5X format for maximal efficiency.
See the [complete list](https://github.com/google-research/t5x/blob/main/docs/models.md) including the
matching Gin configuration files.

These are converted from the public [Mesh TensorFlow
checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511)
.

### Compatibility with the Mesh TensorFlow checkpoints
The Mesh TensorFlow checkpoints trained using the [T5 library][t5_github] can be
directly loaded into T5X. For example, we can rerun the fine-tuning example
initializing from the MTF checkpoint by changing the `INIT_CHECKPOINT` Gin
macro.

```sh
# Model dir to save logs, ckpts, etc. in "gs://model_dir" format.
MODEL_DIR="..."

# Data dir to save the processed dataset in "gs://data_dir" format.
TFDS_DATA_DIR="..."
T5X_DIR="..."  # directory where the T5X repo is cloned.

python3 ${T5X_DIR}/t5x/train.py \
  --gin_file="t5x/examples/t5/t5_1_1/examples/base_wmt19_ende_train.gin" \
  --gin.MODEL_DIR=\"${MODEL_DIR}\" \
  --gin.MIXTURE_OR_TASK_NAME=\"wmt_t2t_ende_v003\" \
  --gin.INIT_CHECKPOINT=\"gs://t5-data/pretrained_models/t5.1.1.base/model.ckpt-1000000\" \
  --tfds_data_dir=${TFDS_DATA_DIR}
```

Note that restoring directly from the Mesh TensorFlow checkpoints can be
inefficient if heavy model parallelism is used for large models. This is
because each host loads the entire copy of the model first and then keep only
the relevant slices dictated by the model parallelism specification. If you have
Mesh TensorFlow checkpoints that you run often, we recommend converting the
checkpoints to T5X native format using the
[convert_tf_checkpoint script](t5x/scripts/convert_tf_checkpoint.py).

## Citing T5X
Please use the following bibtex entry to cite T5X.

```
@article{roberts2022t5x,
  url = {https://arxiv.org/abs/2203.17189},
  author = {Roberts, Adam and Chung, Hyung Won and Levskaya, Anselm and Mishra, Gaurav and Bradbury, James and Andor, Daniel and Narang, Sharan and Lester, Brian and Gaffney, Colin and Mohiuddin, Afroz and Hawthorne, Curtis and Lewkowycz, Aitor and Salcianu, Alex and van Zee, Marc and Austin, Jacob and Goodman, Sebastian and Soares, Livio Baldini and Hu, Haitang and Tsvyashchenko, Sasha and Chowdhery, Aakanksha and Bastings, Jasmijn and Bulian, Jannis and Garcia, Xavier and Ni, Jianmo and Chen, Andrew and Kenealy, Kathleen and Clark, Jonathan H. and Lee, Stephan and Garrette, Dan and Lee-Thorp, James and Raffel, Colin and Shazeer, Noam and Ritter, Marvin and Bosma, Maarten and Passos, Alexandre and Maitin-Shepard, Jeremy and Fiedel, Noah and Omernick, Mark and Saeta, Brennan and Sepassi, Ryan and Spiridonov, Alexander and Newlan, Joshua and Gesmundo, Andrea},
  title = {Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$},
  journal={arXiv preprint arXiv:2203.17189},
  year = {2022},
}
```

## Note
This is not an officially supported Google product

[t5_paper]: https://arxiv.org/abs/1910.10683
[t5_github]: https://github.com/google-research/text-to-text-transfer-transformer
[gin-primer]: docs/usage/gin.md


# PlatformIO Documentation

[https://docs.platformio.org/](https://docs.platformio.org/)


![GHA build main](https://github.com/coreruleset/coreruleset/workflows/Regression%20Tests/badge.svg?branch=main)

![GHA build v4.0/dev](https://github.com/coreruleset/coreruleset/workflows/Regression%20Tests/badge.svg?branch=v4.0%2Fdev)
![GHA build v3.3/dev](https://github.com/coreruleset/coreruleset/workflows/Regression%20Tests/badge.svg?branch=v3.3%2Fdev)
[![OWASP Flagship](https://img.shields.io/badge/owasp-flagship%20project-38a047.svg)](https://owasp.org/projects/)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1390/badge)](https://bestpractices.coreinfrastructure.org/projects/1390)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

# OWASP CRS

The OWASP CRS is a set of generic attack detection rules for use with ModSecurity or compatible web application firewalls. The CRS aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts.

## CRS Resources

Please see the [OWASP CRS page](https://coreruleset.org/) to get introduced to the CRS and view resources on installation, configuration, and working with the CRS.

## Contributing to the CRS

We strive to make the OWASP ModSecurity CRS accessible to a wide audience of beginner and experienced users. We are interested in hearing any bug reports, false-positive alert reports, evasions, usability issues, and suggestions for new detections.

[Create an issue on GitHub](https://github.com/coreruleset/coreruleset/issues) to report a false positive or false negative (evasion). Please include your installed version and the relevant portions of your ModSecurity audit log. We will try and address your issue and potentially ask for additional information to reproduce your problem. Please also note that stale issues will be flagged and closed after 120 days. You can search for stale issues with the following [search query](https://github.com/coreruleset/coreruleset/issues?q=label%3A%22Stale+issue%22).

[Sign up for our Google Group](https://groups.google.com/a/owasp.org/g/modsecurity-core-rule-set-project) to ask general usage questions and participate in discussions on the CRS. Also [here](https://lists.owasp.org/pipermail/owasp-modsecurity-core-rule-set/index) you can find the archives for the previous mailing list.

[Join the #coreruleset channel on OWASP Slack](https://owasp.slack.com/) to chat about the CRS. ([Click here](https://owasp.org/slack/invite) to get an invitation if you are not yet registered on the OWASP slack. It's open to non-members too.)

## License

Copyright (c) 2006-2020 Trustwave and contributors. All rights reserved.

Copyright (c) 2021-2024 CRS project. All rights reserved.

The OWASP CRS is distributed under Apache Software License (ASL) version 2. Please see the enclosed LICENSE file for full details.


<img src="https://waldur.com/assets/img/logo.svg" alt="Waldur Logo" width="400"/>

# Waldur MasterMind

Waldur is a cloud management, orchestration and accounting system. Used in government clouds, universities, HPC centres and mid-to-large enterprises.

Common uses include:

* HPC centres
* Private and public cloud
* Service marketplace
* Federated infrastructure

[![Docs](https://img.shields.io/badge/Docs-docs.waldur.com-green)](https://docs.waldur.com)

![Waldur Screenshot](https://docs.waldur.com/about/img/org-dashboard.png)

## Features

* User self-service
* Service Desk with Atlassian Service Desk
* Support of multiple authentication and authorization mechanisms
* Billing and fine grained accounting

Automated provisioning:

* OpenStack
* SLURM
* Kubernetes clusters using Rancher

## Quick install guide

* [Deploy using docker-compose](https://docs.waldur.com/admin-guide/deployment/docker-compose/)
* [Deploy on Kubernetes using Helm](https://docs.waldur.com/admin-guide/deployment/helm/)
* [Development environment setup](https://docs.waldur.com/developer-guide/install-from-src/)

## Documentation

Administration and usage guides can be found at [documentation site](http://docs.waldur.com).

## License

Waldur MasterMind is released under MIT license.

## Contribution

Waldur is an open-source project co-developed by OpenNode and University of Tartu. [Contribution](https://docs.waldur.com/about/contributing/) rules are maintained at documentation website.

## Commercial support

For professional support and integration services please contact <info@opennodecloud.com>.


*****************************************************************
PyUNIxMD: Python-based UNIversal eXcited state Molecular Dynamics
*****************************************************************

.. image:: image/logo.svg
      :width: 600pt
      :align: center

PyUNIxMD is an object-oriented Python program for molecular dynamics simulations involving multiple electronic states.
It is mainly for studying the nonadiabatic dynamics of excited molecules.

Citation
========

When you publish a work using any part of PyUNIxMD code, please cite the following publication:

* PyUNIxMD: A Python-based excited state molecular dynamics package. *J. Comp. Chem.* **2021**, DOI: `10.1002/jcc.26711 <https://doi.org/10.1002/jcc.26711>`_

* Coupled- and independent-trajectory approaches based on the exact factorization using the PyUNIxMD package. *Top. Cur. Chem.* **2022**, DOI: `10.1007/s41061-021-00361-7 <https://doi.org/10.1007/s41061-021-00361-7>`_

Requirements
============
* Python 3.6 or later
* Numpy
* Cython https://cython.org
* BLAS/LAPACK libraries or Math Kernel Library

You can easily install the latest Numpy and Cython via Python's pip command.

::

  $ pip install --upgrade numpy Cython

Build
=====
You can build PyUNIxMD by the following command.

::

  $ python3 setup.py build_ext -b ./src/build

If you want to run polariton dynamics, first build PyUNIxMD as above
and then set 'do_qed = True' in setup.py and compile using the following command:

::

  $ python3 setup.py build_qed_ext -b ./src/build_qed

Test
====
Without the aid of external QM programs, You can test the PyUNIxMD package with model systems.
The corresponding examples are:

* $PYUNIXMDHOME/examples/qm/Eh-Shin_Metiu

* $PYUNIXMDHOME/examples/qm/SHXF-SAC

* $PYUNIXMDHOME/examples/qm/SHXF-Shin_Metiu

$PYUNIXMDHOME is the top-level directory where this file belongs.

In each directory, you can find the running script named run.py.

Before running test jobs, add the path of the PyUNIxMD package to your Python path.

::

  $ export PYTHONPATH=$PYUNIXMDHOME/src:$PYTHONPATH

Then execute run.py as follows.

::

  $ python3 run.py >& log

Utility scripts
===============
PyUNIxMD provides other Python scripts to analyze results of dynamics calculations.
To use the scripts, you need to add the path of the scripts.

::

  $ export PYTHONPATH=$PYUNIXMDHOME/util:$PYTHONPATH

Documentation
=============
If you have Sphinx, you can locally build the manual of PyUNIxMD by the following command.

::

  $ cd docs
  $ make html


Anubis LMS

  Distributed LMS for automating Computing Science Courses From NYU

[![Anubis Main CD/CI](https://github.com/AnubisLMS/Anubis/actions/workflows/main-ci.yml/badge.svg)](https://github.com/AnubisLMS/Anubis/actions/workflows/main-ci.yml)

# Table of contents

* [Design and Internals](#design-and-internals)
* [Blog](#blog)
* [What is this project?](#what-is-this-project)
  * [Autograding](#autograding)
  * [Anubis Cloud IDEs](#anubis-cloud-ides)
  * [Insights](#insights)
* [How to contribute](#how-to-contribute)
* [Further reading](#further-reading)
* [License](#license)

## Design and Internals

Want to know how Anubis works internally? We have a 20+ page design doc that thoroughly explains how Anubis works
under [our docs](./docs). Don't worry, there are a lot of pictures :)

## Blog

Be sure to check out our [blog](https://anubis.osiris.services/blog)!

## What is this project?

Anubis is a distributed LMS (learning management system) created by [John Cunniff](https://github.com/wabscale) that
is specifically designed for automating CS courses. Anubis has been used and battle tested at [NYU Tandon](https://engineering.nyu.edu/)
for several semesters. The main purposes of this system is to autograde homework submissions, and provide our own Cloud IDE solution to
simplify the student experience.

> _Anubis is built by CS students for CS students_

### Autograding

Under Anubis each student gets their own github repo for each assignment. When they push to their repos, Anubis sees the
push and runs tests on their code. The results are then available to students on the Anubis website _before the deadline_
Under this model students can push as many times as they would like before the assignment deadline.

![alt docs/img/autograde-results.png](docs/design-tex/figures/autograde-results.png)

> _If you would like to read more on how autograding and assignment management works from a TA/Professors
> perspective, check out our [detailed design doc](docs/README.md)_

### Anubis Cloud IDEs

Anubis also provides student one click Cloud IDEs that are accessible in the browser. The Cloud IDEs are optimized for
each courses needs. Students can get a fully insulated, pre-configured, and stable IDE and linux environment in seconds.
No more clunky cloud VM solutions, or spending the first several weeks of the class getting Vagrant to work on hundreds
of student laptops. Students do not need to install, or configure anything for Anubis. Everything is accessible in
the browser.

![alt docs/img/theia-fullscreen.png](docs/design-tex/figures/theia-fullscreen.png)

> _If you would like a more detailed description of how the IDEs work internally, check out the
> [sections on the IDEs in our design doc](docs/README.md)_

### Insights

Simply by placing timestamps on thing that are already tracked like submissions and the test results for submissions,
we can start to both ask and answer some interesting questions about how well students are understanding certain topics.

> _For a more detailed description (plus visuals) of how admin insights work in Anubis, check out
> the [section on usage statistics in our design doc](/docs/README.md)_

## How to contribute

- Checkout the [project board](https://github.com/AnubisLMS/Anubis/projects) to see what you can
contribute
- Checkout our [Development Guide](./docs/development_guide.md) to get up and running debugging Anubis
- Checkout the [Contribution Guide](./.github/CONTRIBUTING.md) to see how to submit PRs and Issues

## Further reading

The best place to get familiar with the internals of Anubis is to read our
[very comprehensive design doc](docs/README.md). Try finding the places in the code that are referenced in the code doc.
You will get familiar with the layout of the project, and the internals of the system.

## License

Anubis is available under the [MIT license](https://opensource.org/licenses/MIT). Anubis also includes external
libraries that are available under a variety of licenses. See [LICENSE](./LICENSE) for the full license text.


[![Build Status](https://github.com/pyeventsourcing/eventsourcing/actions/workflows/runtests.yaml/badge.svg?branch=main)](https://github.com/pyeventsourcing/eventsourcing/tree/main)
[![Coverage Status](https://coveralls.io/repos/github/pyeventsourcing/eventsourcing/badge.svg?branch=main)](https://coveralls.io/github/pyeventsourcing/eventsourcing?branch=main)
[![Documentation Status](https://readthedocs.org/projects/eventsourcing/badge/?version=stable)](https://eventsourcing.readthedocs.io/en/stable/)
[![Latest Release](https://badge.fury.io/py/eventsourcing.svg)](https://pypi.org/project/eventsourcing/)
[![Downloads](https://static.pepy.tech/personalized-badge/eventsourcing?period=total&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads)](https://pypistats.org/packages/eventsourcing)
[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# Event Sourcing in Python

A library for event sourcing in Python.

*"totally amazing and a pleasure to use"*

*"very clean and intuitive"*

*"a huge help and time saver"*

Please [read the docs](https://eventsourcing.readthedocs.io/). See also [extension projects](https://github.com/pyeventsourcing).

## Installation

Use pip to install the [stable distribution](https://pypi.org/project/eventsourcing/)
from the Python Package Index.

    $ pip install eventsourcing

Please note, it is recommended to install Python
packages into a Python virtual environment.

## Synopsis

Define aggregates with the `Aggregate` class and the `@event` decorator.

```python
from eventsourcing.domain import Aggregate, event

class Dog(Aggregate):
    @event('Registered')
    def __init__(self, name):
        self.name = name
        self.tricks = []

    @event('TrickAdded')
    def add_trick(self, trick):
        self.tricks.append(trick)
```

Define application objects with the `Application` class.

```python
from eventsourcing.application import Application

class DogSchool(Application):
    def register_dog(self, name):
        dog = Dog(name)
        self.save(dog)
        return dog.id

    def add_trick(self, dog_id, trick):
        dog = self.repository.get(dog_id)
        dog.add_trick(trick)
        self.save(dog)

    def get_dog(self, dog_id):
        dog = self.repository.get(dog_id)
        return {'name': dog.name, 'tricks': tuple(dog.tricks)}
```

Write a test.

```python
def test_dog_school():
    # Construct application object.
    school = DogSchool()

    # Evolve application state.
    dog_id = school.register_dog('Fido')
    school.add_trick(dog_id, 'roll over')
    school.add_trick(dog_id, 'play dead')

    # Query application state.
    dog = school.get_dog(dog_id)
    assert dog['name'] == 'Fido'
    assert dog['tricks'] == ('roll over', 'play dead')

    # Select notifications.
    notifications = school.notification_log.select(start=1, limit=10)
    assert len(notifications) == 3
```

Run the test with the default persistence module. Events are stored
in memory using Python objects.

```python
test_dog_school()
```

Configure the application to run with an SQLite database. Other persistence modules are available.

```python
import os

os.environ["PERSISTENCE_MODULE"] = 'eventsourcing.sqlite'
os.environ["SQLITE_DBNAME"] = 'dog-school.db'
```

Run the test with SQLite.

```python
test_dog_school()
```

See the [documentation](https://eventsourcing.readthedocs.io/) for more information.

## Features

**Aggregates and applications** — base classes for event-sourced aggregates
and applications. Suggests how to structure an event-sourced application. All
classes are fully type-hinted to guide developers in using the library.

**Flexible event store** — flexible persistence of aggregate events. Combines
an event mapper and an event recorder in ways that can be easily extended.
Mapper uses a transcoder that can be easily extended to support custom
model object types. Recorders supporting different databases can be easily
substituted and configured with environment variables.

**Application-level encryption and compression** — encrypts and decrypts events inside the
application. This means data will be encrypted in transit across a network ("on the wire")
and at disk level including backups ("at rest"), which is a legal requirement in some
jurisdictions when dealing with personally identifiable information (PII) for example
the EU's GDPR. Compression reduces the size of stored aggregate events and snapshots, usually
by around 25% to 50% of the original size. Compression reduces the size of data
in the database and decreases transit time across a network.

**Snapshotting** — reduces access-time for aggregates that have many events.

**Versioning** - allows changes to be introduced after an application
has been deployed. Both aggregate events and aggregate snapshots can be versioned.

**Optimistic concurrency control** — ensures a distributed or horizontally scaled
application doesn't become inconsistent due to concurrent method execution. Leverages
optimistic concurrency controls in adapted database management systems.

**Notifications and projections** — reliable propagation of application
events with pull-based notifications allows the application state to be
projected accurately into replicas, indexes, view models, and other applications.
Supports materialized views and CQRS.

**Event-driven systems** — reliable event processing. Event-driven systems
can be defined independently of particular persistence infrastructure and mode of
running.

**Detailed documentation** — documentation provides general overview, introduction
of concepts, explanation of usage, and detailed descriptions of library classes.
All code is annotated with type hints.

**Worked examples** — includes examples showing how to develop aggregates, applications
and systems.

## Extensions

The GitHub organisation
[Event Sourcing in Python](https://github.com/pyeventsourcing)
hosts extension projects for the Python eventsourcing library.
There are projects that adapt popular ORMs such as
[Django](https://github.com/pyeventsourcing/eventsourcing-django#readme)
and [SQLAlchemy](https://github.com/pyeventsourcing/eventsourcing-sqlalchemy#readme).
There are projects that adapt specialist event stores such as
[Axon Server](https://github.com/pyeventsourcing/eventsourcing-axonserver#readme) and
[EventStoreDB](https://github.com/pyeventsourcing/eventsourcing-eventstoredb#readme).
There are projects that support popular NoSQL databases such as
[DynamoDB](https://github.com/pyeventsourcing/eventsourcing-dynamodb#readme).
There are also projects that provide examples of using the
library with web frameworks such as
[FastAPI](https://github.com/pyeventsourcing/example-fastapi#readme)
and [Flask](https://github.com/pyeventsourcing/example-flask#readme),
and for serving applications and running systems with efficient
inter-process communication technologies like [gRPC](https://github.com/pyeventsourcing/eventsourcing-grpc#readme).
And there are examples of event-sourced applications and systems
of event-sourced applications, such as the
[Paxos system](https://github.com/pyeventsourcing/example-paxos#readme),
which is used as the basis for a
[replicated state machine](https://github.com/pyeventsourcing/example-paxos/tree/master/replicatedstatemachine),
which is used as the basis for a
[distributed key-value store](https://github.com/pyeventsourcing/example-paxos/tree/master/keyvaluestore).

## Project

This project is [hosted on GitHub](https://github.com/pyeventsourcing/eventsourcing).

Please register questions, requests and
[issues on GitHub](https://github.com/pyeventsourcing/eventsourcing/issues),
or post in the project's Slack channel.

There is a [Slack channel](https://join.slack.com/t/eventsourcinginpython/shared_invite/zt-3hogb36o-LCvKd4Rz8JMALoLSl_pQ8g)
for this project, which you are [welcome to join](https://join.slack.com/t/eventsourcinginpython/shared_invite/zt-3hogb36o-LCvKd4Rz8JMALoLSl_pQ8g).

Please refer to the [documentation](https://eventsourcing.readthedocs.io/) for installation and usage guides.


![image](./docs/source/img/logos/mala_horizontal.png)

# MALA

[![CPU](https://github.com/mala-project/mala/actions/workflows/cpu-tests.yml/badge.svg)](https://github.com/mala-project/mala/actions/workflows/cpu-tests.yml)
[![image](https://github.com/mala-project/mala/actions/workflows/gh-pages.yml/badge.svg)](https://mala-project.github.io/mala/)
[![image](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5557255.svg)](https://doi.org/10.5281/zenodo.5557255)

MALA (Materials Learning Algorithms) is a data-driven framework to generate surrogate models of density functional theory calculations based on machine learning. Its purpose is to enable multiscale modeling by bypassing computationally expensive steps in state-of-the-art density functional simulations.

MALA is designed as a modular and open-source python package. It enables users to perform the entire modeling toolchain using only a few lines of code. MALA is jointly developed by the Sandia National Laboratories (SNL) and the Center for Advanced Systems Understanding (CASUS). See [Contributing](docs/source/CONTRIBUTE.md) for contributing code to the repository.

This repository is structured as follows:
```
├── examples : contains useful examples to get you started with the package
├── install : contains scripts for setting up this package on your machine
├── mala : the source code itself
├── test : test scripts used during development, will hold tests for CI in the future
└── docs : Sphinx documentation folder
```

## Installation

> **WARNING**: Even if you install MALA via PyPI, please consult the full installation instructions afterwards. External modules (like the QuantumESPRESSO bindings) are not distributed via PyPI!

Please refer to [Installation of MALA](docs/source/install/installing_mala.rst).

## Running

You can familiarize yourself with the usage of this package by running
the examples in the `example/` folder.

## Contributors

MALA is jointly maintained by

- [Sandia National Laboratories](https://www.sandia.gov/) (SNL), USA.
    - Scientific supervisor: Sivasankaran Rajamanickam, code maintenance:
Jon Vogel
- [Center for Advanced Systems Understanding](https://www.casus.science/) (CASUS), Germany.
    - Scientific supervisor: Attila Cangi, code maintenance: Lenz Fiedler

A full list of contributors can be found [here](docs/source/CONTRIBUTE.md).

## Citing MALA

If you publish work which uses or mentions MALA, please cite the following paper:

J. A. Ellis, L. Fiedler, G. A. Popoola, N. A. Modine, J. A. Stephens, A. P. Thompson,
A. Cangi, S. Rajamanickam (2021). Accelerating Finite-temperature
Kohn-Sham Density Functional Theory with Deep Neural Networks.
[Phys. Rev. B 104, 035120 (2021)](https://doi.org/10.1103/PhysRevB.104.035120)

alongside this repository.