Supercharge your Model Training

Deep Learning Framework for Training at Scale

[Website]
- [Getting Started]
- [Docs]
- [We're Hiring!]

# **👋 Welcome**

Composer is an open-source deep learning training library by [MosaicML](https://www.mosaicml.com/). Built on top of PyTorch, the Composer library makes it easier to implement distributed training workflows on large-scale clusters.

We built Composer to be **optimized for scalability and usability**, integrating best practices for efficient, multi-node training. By abstracting away low-level complexities like parallelism techniques, distributed data loading, and memory optimization, you can focus on training modern ML models and running experiments without slowing down.

We recommend using Composer to speedup your experimentation workflow if you’re training neural networks of any size, including:

- Large Language Models (LLMs)
- Diffusion models
- Embedding models (e.g. BERT)
- Transformer-based models
- Convolutional Neural Networks (CNNs)

Composer is heavily used by the MosaicML research team to train state-of-the-art models like MPT, and we open-sourced this library to enable the ML community to do the same. This framework is used by organizations in both the tech industry and the academic sphere and is continually updated with new features, bug fixes, and stability improvements for production workloads.

# **🔑 Key Features**
![Composer is to give you better workflows with the ability to maximize scale and customizability.](docs/source/_static/images/key_features.png)

We designed Composer from the ground up for modern deep learning workloads. Gone are the days of AlexNet and ResNet, when state-of-the-art models could be trained on a couple of desktop GPUs. Today, developing the latest and greatest deep learning models often requires cluster-scale hardware — but with Composer’s help, you’ll hardly notice the difference.

The heart of Composer is our Trainer abstraction: a highly optimized PyTorch training loop designed to allow both you and your model to iterate faster. Our trainer has simple ways for you to configure your parallelization scheme, data loaders, metrics, loggers, and more.

## Scalability

Whether you’re training on 1 GPU or 512 GPUs, 50MB or 10TB of data - Composer is built to keep your workflow simple.

- [**FSDP**](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#fullyshardeddataparallel-fsdp): For large models that are too large to fit on GPUs, Composer has integrated PyTorch [FullyShardedDataParallelism](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#fullyshardeddataparallel-fsdp) into our trainer and made it simple to efficiently parallelize custom models. We’ve found FSDP is competitive performance-wise with much more complex parallelism strategies. Alternatively, Composer also supports standard PyTorch distributed data parallelism (DDP) and Deepspeed execution.
- [**Elastic sharded checkpointing**](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#saving-and-loading-sharded-checkpoints-with-fsdp): Save on eight GPUs, resume on sixteen. Composer supports elastic sharded checkpointing, so you never have to worry if your sharded saved state is compatible with your new hardware setup.
- **Data streaming:** Working with large datasets? Download datasets from cloud blob storage on the fly by integrating with MosaicML [StreamingDataset](https://github.com/mosaicml/streaming) during model training.

## Customizability

Other high-level deep learning trainers provide simplicity at the cost of rigidity. When you want to add your own features, their abstractions get in your way. Composer, on the other hand, provides simple ways for you to customize our Trainer to your needs.

![Composer’s training loop has a series of events that occur at each stage in the training process.](docs/source/_static/images/traning_loop.png)

***Fig. 1:** Composer’s training loop has a series of events that occur at each stage in the training process. Callbacks are functions that users write to run at specific events. For example, our [Learning Rate Monitor Callback](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.LRMonitor.html#composer.callbacks.LRMonitor) logs the learning rate at every BATCH_END event.*

- [**Callbacks**](https://docs.mosaicml.com/projects/composer/en/stable/trainer/callbacks.html): Composer’s callback system allows you to insert custom logic at any point in the training loop. We’ve written callbacks to monitor memory usage, log and visualize images, and estimate your model’s remaining training time, to name a few. This feature is popular among researchers who want to implement and experiment with custom training techniques.
- [**Speedup algorithms**](https://docs.mosaicml.com/projects/composer/en/stable/examples/custom_speedup_methods.html): We draw from the latest research to create a collection of algorithmic speedups. Stack these speedups into MosaicML recipes to boost your training speeds. Our team has open-sourced the optimal combinations of speedups for different types of models.
    - **8x speedup: Stable Diffusion**
        - $200k original SD2 cost —> $50k ([Blog](https://www.mosaicml.com/blog/diffusion))
    - **7x speedup: ResNet-50 on ImageNet**
        - 3h33m —> 25m on 8xA100 ([Blog](https://www.mosaicml.com/blog/mosaic-resnet))
    - **8.8x speedup: BERT-Base Pretraining**
        - 10h —> 1.13h on 8xA100 ([Blog](https://www.mosaicml.com/blog/mosaicbert))
    - **5.4x speedup: DeepLab v3 on ADE20K**
        - 3h30m —> 39m on 8xA100 ([Blog](https://www.mosaicml.com/blog/behind-the-scenes))

## Better workflows

Composer is built to automate away low-level pain points and headaches so you can focus on the important (and fun) parts of deep learning and iterate faster.

- [**Auto-resumption**](https://docs.mosaicml.com/projects/composer/en/stable/notes/resumption.html): Failed training run? Have no fear — just re-run your code, and Composer will automatically resume from your latest saved checkpoint.
- [**CUDA OOM Prevention**](https://docs.mosaicml.com/projects/composer/en/stable/examples/auto_microbatching.html): Say goodbye to out-of-memory errors. Set your microbatch size to “auto”, and Composer will automatically select the biggest one that fits on your GPUs.
- **[Time Abstractions](https://docs.mosaicml.com/projects/composer/en/latest/trainer/time.html):** Ever messed up your conversion between update steps, epochs, samples, and tokens? Specify your training duration with custom units (epochs, batches, samples, and tokens) in your training loop with our `Time` class.

## Integrations

Integrate with the tools you know and love for experiment tracking and data streaming.

- **Cloud integrations**: Our Checkpointing and logging features have first-class support for remote storage and loading from Cloud bucket (OCI, GCP, AWS S3).
- **********Experiment tracking:********** Weights and Biases, MLFlow, CometML, and neptune.ai — the choice is yours, easily log your data to your favorite platform.

# **🚀 Getting Started**

## **📍**Prerequisites

Composer is designed for users who are comfortable with Python and have basic familiarity with deep learning fundamentals and PyTorch.

**********************************************Software requirements:**********************************************  A recent version of PyTorch.

**********************************************Hardware requirements:**********************************************  System with CUDA-compatible GPUs (AMD + RoCM coming soon!). Composer can run on CPUs, but for full benefits, we recommend using it on hardware accelerators.

## **💾 Installation**

Composer can be installed with `pip`:

```bash
pip install mosaicml
```

To simplify the environment setup for Composer, we also provide a set of [pre-built Docker images](https://docs.mosaicml.com/projects/composer/en/stable/getting_started/installation.html#docker). We *highly recommend* you use our Docker images.

## **🏁 Quick Start**

Here is a code snippet demonstrating our Trainer on the MNIST dataset.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

from composer import Trainer
from composer.models import ComposerClassifier
from composer.algorithms import LabelSmoothing, CutMix, ChannelsLast

class Model(nn.Module):
    """Toy convolutional neural network architecture in pytorch for MNIST."""

    def __init__(self, num_classes: int = 10):
        super().__init__()

        self.num_classes = num_classes

        self.conv1 = nn.Conv2d(1, 16, (3, 3), padding=0)
        self.conv2 = nn.Conv2d(16, 32, (3, 3), padding=0)
        self.bn = nn.BatchNorm2d(32)
        self.fc1 = nn.Linear(32 * 16, 32)
        self.fc2 = nn.Linear(32, num_classes)

    def forward(self, x):
        out = self.conv1(x)
        out = F.relu(out)
        out = self.conv2(out)
        out = self.bn(out)
        out = F.relu(out)
        out = F.adaptive_avg_pool2d(out, (4, 4))
        out = torch.flatten(out, 1, -1)
        out = self.fc1(out)
        out = F.relu(out)
        return self.fc2(out)

transform = transforms.Compose([transforms.ToTensor()])
dataset = datasets.MNIST("data", train=True, download=True, transform=transform)
train_dataloader = DataLoader(dataset, batch_size=128)

trainer = Trainer(
    model=ComposerClassifier(module=Model(), num_classes=10),
    train_dataloader=train_dataloader,
    max_duration="2ep",
    algorithms=[
        LabelSmoothing(smoothing=0.1),
        CutMix(alpha=1.0),
        ChannelsLast(),
    ],
)
trainer.fit()
```

Next, check out our [Getting Started Colab](https://colab.research.google.com/github/mosaicml/composer/blob/9f594876f957c912758e540598ac9f47a468c39d/examples/getting_started.ipynb) for a walk-through of Composer’s main features. In this tutorial, we will cover the basics of the Composer Trainer:

- Dataloader
- Trainer
- Optimizer and Scheduler
- Logging
- Training a baseline model
- Speeding up training

## **📚 Learn more**

Once you’ve completed the Quick Start, you can go through the below tutorials or our [documentation](https://docs.mosaicml.com/projects/composer/en/stable/) to further familiarize yourself with Composer.

If you have any questions, please feel free to reach out to us on our [Community Slack](https://dub.sh/mcomm)!

Here are some resources actively maintained by the Composer community to help you get started:

      Resource
      Details

    Training BERTs with Composer and 🤗
    A Colab Notebook showing how to train BERT models with Composer and 🤗!

    Pretraining and Finetuning an LLM Tutorial
    A tutorial from MosaicML’s LLM Foundry, using MosaicML Composer, StreamingDataset, and MCLI on training and evaluating LLMs.

    Migrating from PyTorch Lightning
    A tutorial is to illustrating a path from working in PyTorch Lightning to working in Composer.

    Finetuning and Pretraining HuggingFace Models
    Want to use Hugging Face models with Composer? No problem. Here, we’ll walk through using Composer to fine-tune a pretrained Hugging Face BERT model.

    Building Speedup Methods
    A Colab Notebook showing how to build new training modifications on top of Composer

# 🛠 For Best Results, Use within the Databricks & MosaicML Ecosystem

Composer can be used on its own, but for the smoothest experience we recommend using it in combination with other components of the MosaicML ecosystem:

![We recommend that you train models with Composer, MosaicML StreamingDatasets, and Mosaic AI training.](docs/source/_static/images/ecosystem.png)

- [**Mosaic AI training**](https://www.databricks.com/product/machine-learning/mosaic-ai-training) (MCLI)- Our proprietary Command Line Interface (CLI) and Python SDK for orchestrating, scaling, and monitoring the GPU nodes and container images executing training and deployment. Used by our customers for training their own Generative AI models.
    - **To get started, [reach out here](https://www.databricks.com/company/contact) and check out our [Training](https://www.databricks.com/product/machine-learning/mosaic-ai-training) product pages**
- [**MosaicML LLM Foundry**](https://github.com/mosaicml/llm-foundry) - This open source repository contains code for training, finetuning, evaluating, and preparing LLMs for inference with [Composer](https://github.com/mosaicml/composer). Designed to be easy to use, efficient and flexible, this codebase is designed to enable rapid experimentation with the latest techniques.
- [**MosaicML StreamingDataset**](https://github.com/mosaicml/streaming) - Open-source library for fast, accurate streaming from cloud storage.
- [**MosaicML Diffusion**](https://github.com/mosaicml/diffusion) - Open-source code to train your own Stable Diffusion model on your own data.  Learn more via our blogs: ([Results](https://www.mosaicml.com/blog/stable-diffusion-2) , [Speedup Details](https://www.mosaicml.com/blog/diffusion))

# **🏆 Project Showcase**

Here are some projects and experiments that used Composer. Got something to add? Share in our [Community Slack](https://dub.sh/mcomm)!

- [**MPT Foundation Series:**](https://www.mosaicml.com/mpt) Commercially usable open source LLMs, optimized for fast training and inference and trained with Composer.
    - [MPT-7B Blog](https://www.mosaicml.com/blog/mpt-7b)
    - [MPT-7B-8k Blog](https://www.mosaicml.com/blog/long-context-mpt-7b-8k)
    - [MPT-30B Blog](https://www.mosaicml.com/blog/mpt-30b)
- [**Mosaic Diffusion Models**](https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k): see how we trained a stable diffusion model from scratch for <$50k
- [**replit-code-v1-3b**](https://huggingface.co/replit/replit-code-v1-3b): A 2.7B Causal Language Model focused on **Code Completion,** trained by Replit on Mosaic AI training in 10 days.
- **BabyLLM:** the first LLM to support both Arabic and English. This 7B model was trained by MetaDialog on the world’s largest Arabic/English dataset to improve customer support workflows ([Blog](https://blogs.nvidia.com/blog/2023/08/31/generative-ai-startups-africa-middle-east/))
- [**BioMedLM**](https://www.mosaicml.com/blog/introducing-pubmed-gpt): a domain-specific LLM for Bio Medicine built by MosaicML and [Stanford CRFM](https://crfm.stanford.edu/)

# 💫 Contributors

Composer is part of the broader Machine Learning community, and we welcome any contributions, pull requests, or issues!

To start contributing, see our [Contributing](https://github.com/mosaicml/composer/blob/dev/CONTRIBUTING.md) page.

P.S.: [We're hiring](https://www.databricks.com/company/careers/open-positions?department=Mosaic%20AI&location=all)!

# ❓FAQ

- **What is the best tech stack you recommend when training large models?**
    - We recommend that users combine components of the MosaicML ecosystem for the smoothest experience:
        - Composer
        - [StreamingDataset](https://github.com/mosaicml/streaming)
        - [MCLI](https://www.databricks.com/product/machine-learning/mosaic-ai-training) (Databricks Mosaic AI Training)
- **How can I get community support for using Composer?**
    - You can join our [Community Slack](https://dub.sh/mcomm)!
- **How does Composer compare to other trainers like NeMo Megatron and PyTorch Lightning?**
    - We built Composer to be optimized for both simplicity and efficiency. Community users have shared that they enjoy Composer for its capabilities and ease of use compared to alternative libraries.
- **How do I use Composer to train graph neural networks (GNNs), or Generative Adversarial Networks (GANs), or models for reinforcement learning (RL)?**
    - We recommend you use alternative libraries for if you want to train these types of models - a lot of assumptions we made when designing Composer are suboptimal for GNNs, RL, and GANs

# ✍ Citation
```
@misc{mosaicml2022composer,
    author = {The Mosaic ML Team},
    title = {composer},
    year = {2021},
    howpublished = {\url{https://github.com/mosaicml/composer/}},
}
```


# Azure Trusted Research Environment

**Azure TRE documentation site**: <https://microsoft.github.io/AzureTRE/>

## Background
<img align="right" src="./docs/assets/azure-tre-logo.svg" width="33%" />

Across the health industry, be it a pharmaceutical company interrogating clinical trial results, or a public health provider analyzing electronic health records, there is the need to enable researchers, analysts, and developers to work with sensitive data sets.

Trusted Research Environments (TREs) enable organisations to provide research teams secure access to these data sets alongside appropriate tooling to ensure researchers can remain efficient and productive despite the security controls in place.

Further information on TREs in general can be found in many places, one good resource is [HDR UK's website](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/).

The Azure Trusted Research Environment project is an accelerator to assist Microsoft customers and partners who want to build out Trusted Research environments on Azure. This project enables authorized users to deploy and configure secure workspaces and researcher tooling without a dependency on IT teams.

This project is typically implemented alongside a data platform that provides research ready datasets to TRE workspaces.

TREs are not “one size fits all”, hence although the Azure TRE has a number of out of the box features, the project has been built be extensible, and hence tooling and data platform agnostic.

Core features include:
- Self-service workspace management for TRE administrators
- Self-service provisioning of research tooling for research teams
- Package and repository mirroring - PyPi, R-CRAN, Apt and more.
- Extensible architecture - build your own service templates as required
- Microsoft Entra ID integration
- Airlock - import and export
- Cost reporting
- Ready to workspace templates including:
  - Restricted with data exfiltration control
  - Unrestricted for open data
- Ready to go workspace service templates including:
  - Virtual Desktops: Windows, Linux
  - AzureML (Jupyter, R Studio, VS Code)
  - ML Flow
  - Gitea

## Project Status and Support

***This project's code base is still under development and breaking changes will happen. Whilst the maintainers will do our best to minimise disruption to existing deployments, this may not always be possible. Stable releases will be published when the project is more mature.***

The aim is to bring together learnings from past customer engagements where TREs have been built into a single reference solution. This is a solution accelerator aiming to be a great starting point for a customized TRE solution. You're encouraged to download and customize the solution to meet your requirements

This project does not have a dedicated team of maintainers but relies on you and the community to maintain and enhance the solution. Microsoft will on project-to-project basis continue to extend the solution in collaboration with customers and partners. No guarantees can be offered as to response times on issues, feature requests, or to the long term road map for the project.

It is important before deployment of the solution that the [Support Policy](SUPPORT.md) is read and understood.

## Contributing

This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit [https://cla.opensource.microsoft.com](https://cla.opensource.microsoft.com).

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

Note: maintainers should refer to the [maintainers guide](maintainers.md)

## Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.

## Repository structure

```text
├── .github
│   ├── ISSUE_TEMPLATE     - Templates for GitHub issues
│   ├── linters            - Linter definitions for workflows
│   └── workflows          - GitHub Actions workflows (CI/CD)
│
├── devops
│   ├── scripts            - DevOps scripts
│   └── terraform          - Terraform specific DevOps files/scripts for bootstrapping
│
├── docs                   - Documentation
│
├── e2e_tests              - pytest-based end-to-end tests
│
├── api_app                - API source code and docs
│
├── resource_processor     - VMSS Porter Runner
│
├── scripts                - Utility scripts
│
└── templates
    ├── core/terraform     - Terraform definitions of Azure TRE core resources
    ├── shared_services    - Terraform definitions of shared services
    ├── workspace_services - Workspace services
    └── workspaces         - Workspace templates
```


# openshift-python-wrapper
Pypi: [openshift-python-wrapper](https://pypi.org/project/openshift-python-wrapper)
A python wrapper for [kubernetes-python-client](https://github.com/kubernetes-client/python) with support for RedHat Container Virtualization. ([Openshift Virtualization](https://www.openshift.com/learn/topics/virtualization))
Docs: [openshift-python-wrapper docs](https://openshift-python-wrapper.readthedocs.io/en/latest/)

## Installation
From source:
```bash
git clone https://github.com/RedHatQE/openshift-python-wrapper.git
cd openshift-python-wrapper
python setup.py install --user
```
From pypi:
```bash
pip install openshift-python-wrapper --user
```

## Release new version
### requirements:
* Export GitHub token
```bash
export GITHUB_TOKEN=<your_github_token>
```
* [release-it](https://github.com/release-it/release-it)
```bash
sudo npm install --global release-it
npm install --save-dev @release-it/bumper
```
### usage:
* Create a release, run from the relevant branch.
To create a 4.11 release, run:
```bash
git checkout v4.11
git pull
release-it # Follow the instructions
```

## docs
Hosted on readthedocs.io [openshift-python-wrapper](https://openshift-python-wrapper.readthedocs.io/en/latest/)

## PR dependency
For PR dependency we use [dpulls](https://www.dpulls.com/)
To make PR depends on other PR add `depends on #<PR NUMBER>` in the PR description.

## Logging configuration
To change log level export OPENSHIFT_PYTHON_WRAPPER_LOG_LEVEL:

```bash
export OPENSHIFT_PYTHON_WRAPPER_LOG_LEVEL=<LOG_LEVEL> # can be: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"
```

## Code check
We use pre-commit for code check.
```bash
pre-commit install
```

Some code examples locate at `examples` directory

## Contribute to the project
To contribute new additions or changes to the project, please refer to the [contribution guide](CONTRIBUTING.md) first.


# Home Assistant Operating System

Home Assistant Operating System (formerly HassOS) is a Linux based operating system optimized to host [Home Assistant](https://www.home-assistant.io) and its [Add-ons](https://www.home-assistant.io/addons/).

Home Assistant Operating System uses Docker as its container engine. By default it deploys the Home Assistant Supervisor as a container. Home Assistant Supervisor in turn uses the Docker container engine to control Home Assistant Core and Add-Ons in separate containers. Home Assistant Operating System is **not** based on a regular Linux distribution like Ubuntu. It is built using [Buildroot](https://buildroot.org/) and it is optimized to run Home Assistant. It targets single board compute (SBC) devices like the Raspberry Pi or ODROID but also supports x86-64 systems with UEFI.

## Features

- Lightweight and memory-efficient
- Minimized I/O
- Over The Air (OTA) updates
- Offline updates
- Modular using Docker container engine

## Supported hardware

- Nabu Casa
- Raspberry Pi
- Hardkernel ODROID
- Asus Tinker Board
- Generic x86-64 (e.g. Intel NUC)
- Virtual appliances

See the full list and specific models [here](./Documentation/boards/README.md)

## Getting Started

If you just want to use Home Assistant the official [getting started guide](https://www.home-assistant.io/getting-started/) and [installation instructions](https://www.home-assistant.io/hassio/installation/) take you through how to download Home Assistant Operating System and get it running on your machine.

If you're interested in finding out more about Home Assistant Operating System and how it works read on...

## Development

If you don't have experience with embedded systems, Buildroot or the build process for Linux distributions it is recommended to read up on these topics first (e.g. [Bootlin](https://bootlin.com/docs/) has excellent resources).

The Home Assistant Operating System documentation can be found on the [Home Assistant Developer Docs website](https://developers.home-assistant.io/docs/operating-system).

### Components

- **Bootloader:**
  - [Barebox](https://barebox.org/) for devices that support UEFI
  - [U-Boot](https://www.denx.de/wiki/U-Boot) for devices that don't support UEFI
- **Operating System:**
  - [Buildroot](https://buildroot.org/) LTS Linux
- **File Systems:**
  - [SquashFS](https://www.kernel.org/doc/Documentation/filesystems/squashfs.txt) for read-only file systems (using LZ4 compression)
  - [ZRAM](https://www.kernel.org/doc/Documentation/blockdev/zram.txt) for `/tmp`, `/var` and swap (using LZ4 compression)
- **Container Platform:**
  - [Docker Engine](https://docs.docker.com/engine/) for running Home Assistant components in containers
- **Updates:**
  - [RAUC](https://rauc.io/) for Over The Air (OTA) and USB updates
- **Security:**
  - [AppArmor](https://apparmor.net/) Linux kernel security module

### Development builds

The Development build GitHub Action Workflow is a manually triggered workflow
which creates Home Assistant OS development builds. The development builds are
available at [https://os-artifacts.home-assistant.io/index.html](https://os-artifacts.home-assistant.io/index.html).


Energy Management for Home Assistant

If you like this work please consider buying a coffee ;-)

EHMASS is a Python module designed to optimize your home energy interfacing with Home Assistant.

## Introduction

EMHASS (Energy Management for Home Assistant) is an optimization tool designed for residential households. The package uses a Linear Programming approach to optimize energy usage while considering factors such as electricity prices, power generation from solar panels, and energy storage from batteries. EMHASS provides a high degree of configurability, making it easy to integrate with Home Assistant and other smart home systems. Whether you have solar panels, energy storage, or just a controllable load, EMHASS can provide an optimized daily schedule for your devices, allowing you to save money and minimize your environmental impact.

The complete documentation for this package is [available here](https://emhass.readthedocs.io/en/latest/).

## What is Energy Management for Home Assistant (EMHASS)?

EMHASS and Home Assistant provide a comprehensive energy management solution that can optimize energy usage and reduce costs for households. By integrating these two systems, households can take advantage of advanced energy management features that provide significant cost savings, increased energy efficiency, and greater sustainability.

EMHASS is a powerful energy management tool that generates an optimization plan based on variables such as solar power production, energy usage, and energy costs. The plan provides valuable insights into how energy can be better managed and utilized in the household. Even if households do not have all the necessary equipment, such as solar panels or batteries, EMHASS can still provide a minimal use case solution to optimize energy usage for controllable/deferrable loads.

Home Assistant provides a platform for the automation of household devices based on the optimization plan generated by EMHASS. This includes devices such as batteries, pool pumps, hot water heaters, and electric vehicle (EV) chargers. By automating EV charging and other devices, households can take advantage of off-peak energy rates and optimize their EV charging schedule based on the optimization plan generated by EMHASS.

One of the main benefits of integrating EMHASS and Home Assistant is the ability to customize and tailor the energy management solution to the specific needs and preferences of each household. With EMHASS, households can define their energy management objectives and constraints, such as maximizing self-consumption or minimizing energy costs, and the system will generate an optimization plan accordingly. Home Assistant provides a platform for the automation of devices based on the optimization plan, allowing households to create a fully customized and optimized energy management solution.

Overall, the integration of EMHASS and Home Assistant offers a comprehensive energy management solution that provides significant cost savings, increased energy efficiency, and greater sustainability for households. By leveraging advanced energy management features and automation capabilities, households can achieve their energy management objectives while enjoying the benefits of a more efficient and sustainable energy usage, including optimized EV charging schedules.

The package flow can be graphically represented as follows:

![](https://raw.githubusercontent.com/davidusb-geek/emhass/master/docs/images/ems_schema.png)

## Configuration and Installation

The package is meant to be highly configurable with an object oriented modular approach and a main configuration file defined by the user.
EMHASS was designed to be integrated with Home Assistant, hence it's name.
Installation instructions and example Home Assistant automation configurations are given below.

You must follow these steps to make EMHASS work properly:

1) Define all the parameters in the configuration file according to your installation. See the description for each parameter in the **configuration** section.

2) You most notably will need to define the main data entering EMHASS. This will be the `sensor.power_photovoltaics` for the name of the your hass variable containing the PV produced power and the variable `sensor.power_load_no_var_loads` for the load power of your household excluding the power of the deferrable loads that you want to optimize.

3) Launch the actual optimization and check the results. This can be done manually using the buttons in the web ui or with a `curl` command like this: `curl -i -H 'Content-Type:application/json' -X POST -d '{}' http://localhost:5000/action/dayahead-optim`.

4) If you’re satisfied with the optimization results then you can set the optimization and data publish task commands in an automation. You can read more about this on the **usage** section below.

5) The final step is to link the deferrable loads variables to real switchs on your installation. An example code for this using automations and the shell command integration is presented below in the **usage** section.

A more detailed workflow is given below:

![](https://raw.githubusercontent.com/davidusb-geek/emhass/master/docs/images/workflow.png)

### Method 1) The EMHASS add-on for Home Assistant OS and supervised users

For Home Assistant OS and HA Supervised users, I've developed an add-on that will help you use EMHASS. The add-on is more user friendly as the configuration can be modified directly in the add-on options pane and as with the standalone docker it exposes a web ui that can be used to inspect the optimization results and manually trigger a new optimization.

You can find the add-on with the installation instructions here: [https://github.com/davidusb-geek/emhass-add-on](https://github.com/davidusb-geek/emhass-add-on)

The add-on usage instructions can be found on the documentation pane of the add-on once installed or directly here: [EMHASS Add-on documentation](https://github.com/davidusb-geek/emhass-add-on/blob/main/emhass/DOCS.md)

These architectures are supported: `amd64`, `armv7`, `armhf` and `aarch64`.

### Method 2) Using Docker in standalone mode

You can also install EMHASS using docker. This can be in the same machine as Home Assistant (if using the supervised install method) or in a different distant machine. To install first pull the latest image from docker hub:
```bash
docker pull davidusb/emhass-docker-standalone
```

You can also build your image locally. For this clone this repository, setup your `config_emhass.yaml` file and use the provided make file with this command:
```bash
make -f deploy_docker.mk clean_deploy
```
Then load the image in the .tar file:
```bash
docker load -i <TarFileName>.tar
```
Finally check your image tag with `docker images` and launch the docker itself:
```bash
docker run -it --restart always -p 5000:5000 -e LOCAL_COSTFUN="profit" -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml --name DockerEMHASS <REPOSITORY:TAG>
```
  - If you wish to keep a local, persistent copy of the EMHASS generated data, create a local folder on your device, then mount said folder inside the container.
    ```bash
    mkdir -p $(pwd)/data #linux: create data folder on local device

    docker run -it --restart always -p 5000:5000 -e LOCAL_COSTFUN="profit" -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/data:/app/data  -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml --name DockerEMHASS <REPOSITORY:TAG>
    ```

If you wish to set the web_server's diagrams to a timezone other than UTC, set `TZ` environment variable on:
```bash
docker run -it --restart always -p 5000:5000  -e TZ="Europe/Paris"  -e LOCAL_COSTFUN="profit" -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml --name DockerEMHASS <REPOSITORY:TAG>
```
### Method 3) Legacy method using a Python virtual environment

With this method it is recommended to install on a virtual environment.
For this you will need `virtualenv`, install it using:
```bash
sudo apt install python3-virtualenv
```
Then create and activate the virtual environment:
```bash
virtualenv -p /usr/bin/python3 emhassenv
cd emhassenv
source bin/activate
```
Install using the distribution files:
```bash
python3 -m pip install emhass
```
Clone this repository to obtain the example configuration files.
We will suppose that this repository is cloned to:
```
/home/user/emhass
```
This will be the root path containing the yaml configuration files (`config_emhass.yaml` and `secrets_emhass.yaml`) and the different needed folders (a `data` folder to store the optimizations results and a `scripts` folder containing the bash scripts described further below).

To upgrade the installation in the future just use:
```bash
python3 -m pip install --upgrade emhass
```

## Usage

### Method 1) Add-on and docker standalone

If using the add-on or the standalone docker installation, it exposes a simple webserver on port 5000. You can access it directly using your brower, ex: http://localhost:5000.

With this web server you can perform RESTful POST commands on multiple ENDPOINTS with prefix `action/*`:

- A POST call to `action/perfect-optim` to perform a perfect optimization task on the historical data.
- A POST call to `action/dayahead-optim` to perform a day-ahead optimization task of your home energy.
- A POST call to `action/naive-mpc-optim` to perform a naive Model Predictive Controller optimization task. If using this option you will need to define the correct `runtimeparams` (see further below).
- A POST call to `action/publish-data` to publish the optimization results data for the current timestamp.
- A POST call to `action/forecast-model-fit` to train a machine learning forecaster model with the passed data (see the [dedicated section](https://emhass.readthedocs.io/en/latest/mlforecaster.html) for more help).
- A POST call to `action/forecast-model-predict` to obtain a forecast from a pre-trained machine learning forecaster model (see the [dedicated section](https://emhass.readthedocs.io/en/latest/mlforecaster.html) for more help).
- A POST call to `action/forecast-model-tune` to optimize the machine learning forecaster models hyperparameters using bayesian optimization (see the [dedicated section](https://emhass.readthedocs.io/en/latest/mlforecaster.html) for more help).

A `curl` command can then be used to launch an optimization task like this: `curl -i -H 'Content-Type:application/json' -X POST -d '{}' http://localhost:5000/action/dayahead-optim`.

### Method 2) Legacy method using a Python virtual environment

To run a command simply use the `emhass` CLI command followed by the needed arguments.
The available arguments are:
- `--action`: That is used to set the desired action, options are: `perfect-optim`, `dayahead-optim`, `naive-mpc-optim`, `publish-data`, `forecast-model-fit`, `forecast-model-predict` and `forecast-model-tune`.
- `--config`: Define path to the config.yaml file (including the yaml file itself)
- `--costfun`: Define the type of cost function, this is optional and the options are: `profit` (default), `cost`, `self-consumption`
- `--log2file`: Define if we should log to a file or not, this is optional and the options are: `True` or `False` (default)
- `--params`: Configuration as JSON.
- `--runtimeparams`: Data passed at runtime. This can be used to pass your own forecast data to EMHASS.
- `--debug`: Use `True` for testing purposes.
- `--version`: Show the current version of EMHASS.

For example, the following line command can be used to perform a day-ahead optimization task:
```bash
emhass --action 'dayahead-optim' --config '/home/user/emhass/config_emhass.yaml' --costfun 'profit'
```
Before running any valuable command you need to modify the `config_emhass.yaml` and `secrets_emhass.yaml` files. These files should contain the information adapted to your own system. To do this take a look at the special section for this in the [documentation](https://emhass.readthedocs.io/en/latest/config.html).

## Home Assistant integration

To integrate with home assistant we will need to define some shell commands in the `configuration.yaml` file and some basic automations in the `automations.yaml` file.
In the next few paragraphs we are going to consider the `dayahead-optim` optimization strategy, which is also the first that was implemented, and we will also cover how to publish the results.
Then additional optimization strategies were developed, that can be used in combination with/replace the `dayahead-optim` strategy, such as MPC, or to expland the funcitonalities such as the Machine Learning method to predict your hosehold consumption. Each of them has some specificities and features and will be considered in dedicated sections.

### Dayahead Optimization - Method 1) Add-on and docker standalone

In `configuration.yaml`:
```yaml
shell_command:
  dayahead_optim: "curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim"
  publish_data: "curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"
```
### Dayahead Optimization - Method 2) Legacy method using a Python virtual environment

In `configuration.yaml`:
```yaml
shell_command:
  dayahead_optim: /home/user/emhass/scripts/dayahead_optim.sh
  publish_data: /home/user/emhass/scripts/publish_data.sh
```
Create the file `dayahead_optim.sh` with the following content:
```bash
#!/bin/bash
. /home/user/emhassenv/bin/activate
emhass --action 'dayahead-optim' --config '/home/user/emhass/config_emhass.yaml'
```
And the file `publish_data.sh` with the following content:
```bash
#!/bin/bash
. /home/user/emhassenv/bin/activate
emhass --action 'publish-data' --config '/home/user/emhass/config_emhass.yaml'
```
Then specify user rights and make the files executables:
```bash
sudo chmod -R 755 /home/user/emhass/scripts/dayahead_optim.sh
sudo chmod -R 755 /home/user/emhass/scripts/publish_data.sh
sudo chmod +x /home/user/emhass/scripts/dayahead_optim.sh
sudo chmod +x /home/user/emhass/scripts/publish_data.sh
```
### Common for any installation method

#### Options 1, Home Assistant automate publish

In `automations.yaml`:
```yaml
- alias: EMHASS day-ahead optimization
  trigger:
    platform: time
    at: '05:30:00'
  action:
  - service: shell_command.dayahead_optim
- alias: EMHASS publish data
  trigger:
  - minutes: /5
    platform: time_pattern
  action:
  - service: shell_command.publish_data
```
In these automation's the day-ahead optimization is performed once a day, everyday at 5:30am, and the data *(output of automation)* is published every 5 minutes.

#### Option 2, EMHASS automate publish

In `automations.yaml`:
```yaml
- alias: EMHASS day-ahead optimization
  trigger:
    platform: time
    at: '05:30:00'
  action:
  - service: shell_command.dayahead_optim
  - service: shell_command.publish_data
```
in configuration page/`config_emhass.yaml`
```json
"method_ts_round": "first"
"continual_publish": true
```
In this automation the day-ahead optimization is performed once a day, everyday at 5:30am.
If the `freq` parameter is set to `30` *(default)* in the configuration, the results of the day-ahead optimization will generate 48 values *(for each entity)*, a value for each 30 minutes in a day *(i.e. 24 hrs x 2)*.

Setting the parameter `continual_publish` to `true` in the configuration page, will allow EMHASS to store the optimization results as entities/sensors into seperate json files. `continual_publish` will periodically (every `freq` amount of minutes) run a publish, and publish the optimization results of each generated entities/sensors to Home Assistant. The current state of the sensor/entity being updated every time publish runs, selecting one of the 48 stored values, by comparing the stored values timestamps, the current timestamp and [`"method_ts_round": "first"`](#the-publish-data-specificities) to select the optimal stored value for the current state.

option 1 and 2 are very similar, however option 2 (`continual_publish`) will require a cpu thread to constantly be run inside of EMHASS, lowering efficiency. The reason why you may pick one over the other is explained in more detail bellow in [continual_publish](#continual_publish-emhass-automation).

Lastly, we can link a EMHASS published entities/sensor's current state to a Home Assistant entity on/off switch, controlling a desired controllable load.
For example, imagine that I want to control my water heater. I can use a published `deferrable` EMHASS entity to control my water heaters desired behavior. In this case, we could use an automation like below, to control the desired water heater on and off:

on:
```yaml
automation:
- alias: Water Heater Optimized ON
  trigger:
  - minutes: /5
    platform: time_pattern
  condition:
  - condition: numeric_state
    entity_id: sensor.p_deferrable0
    above: 0.1
  action:
    - service: homeassistant.turn_on
      entity_id: switch.water_heater_switch
```
off:
```yaml
automation:
- alias: Water Heater Optimized OFF
  trigger:
  - minutes: /5
    platform: time_pattern
  condition:
  - condition: numeric_state
    entity_id: sensor.p_deferrable0
    below: 0.1
  action:
    - service: homeassistant.turn_off
      entity_id: switch.water_heater_switch
```
The result of these automation's will turn on and off the Home Assistant entity `switch.water_heater_switch` using the current state from the EMHASS entity `sensor.p_deferrable0`. `sensor.p_deferrable0`  being the entity generated from the EMHASS day-ahead optimization and published by examples above. The `sensor.p_deferrable0` entity current state being updated every 30 minutes (or `freq` minutes) via a automated publish option 1 or 2. *(selecting one of the 48 stored data values)*

## The publish-data specificities

`publish-data` (which is either run manually, or automatically via `continual_publish` or Home Assistant automation), will push the optimization results to Home Assistant for each deferrable load defined in the configuration. For example if you have defined two deferrable loads, then the command will publish `sensor.p_deferrable0` and `sensor.p_deferrable1` to Home Assistant. When the `dayahead-optim` is launched, after the optimization, either entity json files or a csv file will be saved on disk. The `publish-data` command will load the latest csv/json files to look for the closest timestamp that match the current time using the `datetime.now()` method in Python. This means that if EMHASS is configured for 30min time step optimizations, the csv/json will be saved with timestamps 00:00, 00:30, 01:00, 01:30, ... and so on. If the current time is 00:05, and parameter `method_ts_round` is set to `nearest` in the configuration, then the closest timestamp of the optimization results that will be published is 00:00. If the current time is 00:25, then the closest timestamp of the optimization results that will be published is 00:30.

The `publish-data` command will also publish PV and load forecast data on sensors `p_pv_forecast` and `p_load_forecast`. If using a battery, then the battery optimized power and the SOC will be published on sensors `p_batt_forecast` and `soc_batt_forecast`. On these sensors the future values are passed as nested attributes.

If you run publish manually *(or via a Home Assistant Automation)*, it is possible to provide custom sensor names for all the data exported by the `publish-data` command. For this, when using the `publish-data` endpoint we can just add some runtime parameters as dictionaries like this:
```yaml
shell_command:
  publish_data: "curl -i -H \"Content-Type:application/json\" -X POST -d '{\"custom_load_forecast_id\": {\"entity_id\": \"sensor.p_load_forecast\", \"unit_of_measurement\": \"W\", \"friendly_name\": \"Load Power Forecast\"}}' http://localhost:5000/action/publish-data"
```

These keys are available to modify: `custom_pv_forecast_id`, `custom_load_forecast_id`, `custom_batt_forecast_id`, `custom_batt_soc_forecast_id`,  `custom_grid_forecast_id`, `custom_cost_fun_id`, `custom_deferrable_forecast_id`, `custom_unit_load_cost_id` and `custom_unit_prod_price_id`.

If you provide the `custom_deferrable_forecast_id` then the passed data should be a list of dictionaries, like this:
```yaml
shell_command:
  publish_data: "curl -i -H \"Content-Type:application/json\" -X POST -d '{\"custom_deferrable_forecast_id\": [{\"entity_id\": \"sensor.p_deferrable0\",\"unit_of_measurement\": \"W\", \"friendly_name\": \"Deferrable Load 0\"},{\"entity_id\": \"sensor.p_deferrable1\",\"unit_of_measurement\": \"W\", \"friendly_name\": \"Deferrable Load 1\"}]}' http://localhost:5000/action/publish-data"
```
And you should be careful that the list of dictionaries has the correct length, which is the number of defined deferrable loads.

### Computed variables and published data

Below you can find a list of the variables resulting from EMHASS computation, shown in the charts and published to Home Assistant through the ```publish_data``` command:

| EMHASS variable | Definition | Home Assistant published sensor |
| --------------- | ---------- | --------------------------------|
| P_PV | Forecasted power generation from your solar panels (Watts). This helps you predict how much solar energy you will produce during the forecast period. | sensor.p_pv_forecast |
| P_Load | Forecasted household power consumption (Watts). This gives you an idea of how much energy your appliances are expected to use. | sensor.p_load_forecast |
| P_deferrableX
[X = 0, 1, 2, ...] | Forecasted power consumption of deferrable loads (Watts). Deferable loads are appliances that can be managed by EMHASS. EMHASS helps you optimise energy usage by prioritising solar self-consumption and minimizing reliance on the grid or by taking advantage or supply and feed-in tariff volatility. You can have multiple deferable loads and you use this sensor in HA to control these loads via smart switch or other IoT means at your disposal. | sensor.p_deferrableX |
| P_grid_pos | Forecasted power imported from the grid (Watts). This indicates the amount of energy you are expected to draw from the grid when your solar production is insufficient to meet your needs or it is advantagous to consume from the grid. | - |
| P_grid_neg | Forecasted power exported to the grid (Watts). This indicates the amount of excess solar energy you are expected to send back to the grid during the forecast period. | - |
| P_batt | Forecasted (dis)charge power load (Watts) for the battery (if installed). If negative it indicates the battery is charging, if positive that the battery is discharging. | sensor.p_batt_forecast |
| P_grid | Forecasted net power flow between your home and the grid (Watts). This is calculated as P_grid_pos - P_grid_neg. A positive value indicates net export, while a negative value indicates net import. | sensor.p_grid_forecast |
| SOC_opt | Forecasted battery optimized Status Of Charge (SOC) percentage level | sensor.soc_batt_forecast |
| unit_load_cost | Forecasted cost per unit of energy you pay to the grid (typically "Currency"/kWh). This helps you understand the expected energy cost during the forecast period. | sensor.unit_load_cost |
| unit_prod_price | Forecasted price you receive for selling excess solar energy back to the grid (typically "Currency"/kWh). This helps you understand the potential income from your solar production. | sensor.unit_prod_price |
| cost_profit | Forecasted profit or loss from your energy usage for the forecast period. This is calculated as unit_load_cost * P_Load - unit_prod_price * P_grid_pos. A positive value indicates a profit, while a negative value indicates a loss. | sensor.total_cost_profit_value |
| cost_fun_cost | Forecasted cost associated with deferring loads to maximize solar self-consumption. This helps you evaluate the trade-off between managing the load and not managing and potential cost savings. | sensor.total_cost_fun_value |
| optim_status | This contains the status of the latest execution and is the same you can see in the Log following an optimization job. Its values can be Optimal or Infeasible. | sensor.optim_status |

## Passing your own data

In EMHASS we have basically 4 forecasts to deal with:

- PV power production forecast (internally based on the weather forecast and the characteristics of your PV plant). This is given in Watts.

- Load power forecast: how much power your house will demand on the next 24h. This is given in Watts.

- Load cost forecast: the price of the energy from the grid on the next 24h. This is given in EUR/kWh.

- PV production selling price forecast: at what price are you selling your excess PV production on the next 24h. This is given in EUR/kWh.

The sensor containing the load data should be specified in parameter `var_load` in the configuration file. As we want to optimize the household energies, when need to forecast the load power consumption. The default method for this is a naive approach using 1-day persistence. The load data variable should not contain the data from the deferrable loads themselves. For example, lets say that you set your deferrable load to be the washing machine. The variable that you should enter in EMHASS will be: `var_load: 'sensor.power_load_no_var_loads'` and `sensor.power_load_no_var_loads = sensor.power_load - sensor.power_washing_machine`. This is supposing that the overall load of your house is contained in variable: `sensor.power_load`. The sensor `sensor.power_load_no_var_loads` can be easily created with a new template sensor in Home Assistant.

If you are implementing a MPC controller, then you should also need to provide some data at the optimization runtime using the key `runtimeparams`.

The valid values to pass for both forecast data and MPC related data are explained below.

### Alternative publish methods
Due to the flexibility of EMHASS, multiple different approaches to publishing the optimization results have been created. Select a option that best meets your use case:

#### publish last optimization *(manual)*
By default, running an optimization in EMHASS will output the results into the csv file: `data_path/opt_res_latest.csv`  *(overriding the existing data on that file)*. We run the publish command to publish the last optimization saved in the `opt_res_latest.csv`:
```bash
# RUN dayahead
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/dayahead-optim
# Then publish teh results of dayahead
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/publish-data
```
*Note, the published entities from the publish-data action will not automatically update the entities current state (current state being used to check when to turn on and off appliances via Home Assistant automatons). To update the EMHASS entities state, another publish would have to be re-run later when the current time matches the next values timestamp (E.g every 30 minutes). See examples bellow for methods to automate the publish-action.*

#### continual_publish *(EMHASS Automation)*
As discussed in [Common for any installation method - option 2](#option-2-emhass-automate-publish), setting `continual_publish` to `true` in the configuration saves the output of the optimization into the `data_path/entities` folder *(a .json file for each sensor/entity)*. A constant loop (in `freq` minutes) will run, observe the .json files in that folder, and publish the saved files periodically (updating the current state of the entity by comparing date.now with the saved data value timestamps).

For users that wish to run multiple different optimizations, you can set the runtime parameter: `publish_prefix` to something like: `"mpc_"` or `"dh_"`. This will generate unique entity_id names per optimization and save these unique entities as separate files in the folder. All the entity files will then be updated when the next loop iteration runs. If a different `freq` integer was passed as a runtime parameter in an optimization, the `continual_publish` loop will be based on the lowest `freq` saved. An example:

```bash
# RUN dayahead, with freq=30 (default), prefix=dh_
curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"dh_"}' http://localhost:5000/action/dayahead-optim
# RUN MPC, with freq=5, prefix=mpc_
curl -i -H 'Content-Type:application/json' -X POST -d '{"freq":5,"publish_prefix":"mpc_"}' http://localhost:5000/action/naive-mpc-optim
```
This will tell continual_publish to loop every 5 minutes based on the freq passed in MPC. All entities from the output of dayahead "dh_" and MPC "mpc_" will be published every 5 minutes.

*It is recommended to use the 2 other options bellow once you have a more advance understanding of EMHASS and/or Home Assistant.*

#### Mixture of continual_publish and manual *(Home Assistant Automation for Publish)*

You can choose to save one optimization for continual_publish and bypass another optimization by setting `"continual_publish":false` runtime parameter:
```bash
# RUN dayahead, with freq=30 (default), prefix=dh_, included into continual_publish
curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"dh_"}' http://localhost:5000/action/dayahead-optim

# RUN MPC, with freq=5, prefix=mpc_, Manually publish, excluded from continual_publish loop
curl -i -H 'Content-Type:application/json' -X POST -d '{"continual_publish":false,"freq":5,"publish_prefix":"mpc_"}' http://localhost:5000/action/naive-mpc-optim
# Publish MPC output
curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/publish-data
```
This example saves the dayahead optimization into `data_path/entities` as .json files, being included in the `continutal_publish` loop (publishing every 30 minutes). The MPC optimization will not be saved in `data_path/entities`, and therefore only into `data_path/opt_res_latest.csv`. Requiring a publish-data action to be run manually (or via a Home Assistant) Automation for the MPC results.

#### Manual *(Home Assistant Automation for Publish)*

For users who wish to have full control of exactly when they will like to run a publish and have the ability to save multiple different optimizations. The `entity_save` runtime parameter has been created to save the optimization output entities to .json files whilst  `continual_publish` is set to `false` in the configuration. Allowing the user to reference the saved .json files manually via a publish:

in configuration page/`config_emhass.yaml` :
```json
"continual_publish": false
```
POST action :
```bash
# RUN dayahead, with freq=30 (default), prefix=dh_, save entity
curl -i -H 'Content-Type:application/json' -X POST -d '{"entity_save": true, "publish_prefix":"dh_"}' http://localhost:5000/action/dayahead-optim
# RUN MPC, with freq=5, prefix=mpc_, save entity
curl -i -H 'Content-Type:application/json' -X POST -d '{"entity_save": true", "freq":5,"publish_prefix":"mpc_"}' http://localhost:5000/action/naive-mpc-optim
```
You can then reference these .json saved entities via their `publish_prefix`. Include the same `publish_prefix` in the `publish_data` action:
```bash
#Publish the MPC optimization ran above
curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"mpc_"}'  http://localhost:5000/action/publish-data
```
This will publish all entities from the MPC (_mpc) optimization above.

Alternatively, you can choose to publish all the saved files .json files with `publish_prefix` = all:
```bash
#Publish all saved entities
curl -i -H 'Content-Type:application/json' -X POST -d '{"publish_prefix":"all"}'  http://localhost:5000/action/publish-data
```
This action will publish the dayahead (_dh) and MPC (_mpc) optimization results from the optimizations above.

### Forecast data

It is possible to provide EMHASS with your own forecast data. For this just add the data as list of values to a data dictionary during the call to `emhass` using the `runtimeparams` option.

For example if using the add-on or the standalone docker installation you can pass this data as list of values to the data dictionary during the `curl` POST:
```bash
curl -i -H 'Content-Type:application/json' -X POST -d '{"pv_power_forecast":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 70, 141.22, 246.18, 513.5, 753.27, 1049.89, 1797.93, 1697.3, 3078.93, 1164.33, 1046.68, 1559.1, 2091.26, 1556.76, 1166.73, 1516.63, 1391.13, 1720.13, 820.75, 804.41, 251.63, 79.25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}' http://localhost:5000/action/dayahead-optim
```
Or if using the legacy method using a Python virtual environment:
```bash
emhass --action 'dayahead-optim' --config '/home/user/emhass/config_emhass.yaml' --runtimeparams '{"pv_power_forecast":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 70, 141.22, 246.18, 513.5, 753.27, 1049.89, 1797.93, 1697.3, 3078.93, 1164.33, 1046.68, 1559.1, 2091.26, 1556.76, 1166.73, 1516.63, 1391.13, 1720.13, 820.75, 804.41, 251.63, 79.25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}'
```

The possible dictionary keys to pass data are:

- `pv_power_forecast` for the PV power production forecast.

- `load_power_forecast` for the Load power forecast.

- `load_cost_forecast` for the Load cost forecast.

- `prod_price_forecast` for the PV production selling price forecast.

### Passing other data

It is possible to also pass other data during runtime in order to automate the energy management. For example, it could be useful to dynamically update the total number of hours for each deferrable load (`def_total_hours`) using for instance a correlation with the outdoor temperature (useful for water heater for example).

Here is the list of the other additional dictionary keys that can be passed at runtime:

- `num_def_loads` for the number of deferrable loads to consider.

- `P_deferrable_nom` for the nominal power for each deferrable load in Watts.

- `def_total_hours` for the total number of hours that each deferrable load should operate.

- `def_start_timestep` for the timestep as from which each deferrable load is allowed to operate (if you don't want the deferrable load to use the whole optimization timewindow).

- `def_end_timestep` for the timestep before which each deferrable load should operate (if you don't want the deferrable load to use the whole optimization timewindow).

- `treat_def_as_semi_cont` to define if we should treat each deferrable load as a semi-continuous variable.

- `set_def_constant` to define if we should set each deferrable load as a constant fixed value variable with just one startup for each optimization task.

- `solcast_api_key` for the SolCast API key if you want to use this service for PV power production forecast.

- `solcast_rooftop_id` for the ID of your rooftop for the SolCast service implementation.

- `solar_forecast_kwp` for the PV peak installed power in kW used for the solar.forecast API call.

- `SOCtarget` for the desired target value of initial and final SOC.

- `publish_prefix` use this key to pass a common prefix to all published data. This will add a prefix to the sensor name but also to the forecasts attributes keys within the sensor.

## A naive Model Predictive Controller

A MPC controller was introduced in v0.3.0. This is an informal/naive representation of a MPC controller. This can be used in combination with/as a replacement of the Dayahead Optimization.

A MPC controller performs the following actions:

- Set the prediction horizon and receding horizon parameters.
- Perform an optimization on the prediction horizon.
- Apply the first element of the obtained optimized control variables.
- Repeat at a relatively high frequency, ex: 5 min.

This is the receding horizon principle.

When applying this controller, the following `runtimeparams` should be defined:

- `prediction_horizon` for the MPC prediction horizon. Fix this at at least 5 times the optimization time step.

- `soc_init` for the initial value of the battery SOC for the current iteration of the MPC.

- `soc_final` for the final value of the battery SOC for the current iteration of the MPC.

- `def_total_hours` for the list of deferrable loads functioning hours. These values can decrease as the day advances to take into account receding horizon daily energy objectives for each deferrable load.

- `def_start_timestep` for the timestep as from which each deferrable load is allowed to operate (if you don't want the deferrable load to use the whole optimization timewindow). If you specify a value of 0 (or negative), the deferrable load will be optimized as from the beginning of the complete prediction horizon window.

- `def_end_timestep` for the timestep before which each deferrable load should operate (if you don't want the deferrable load to use the whole optimization timewindow). If you specify a value of 0 (or negative), the deferrable load optimization window will extend up to the end of the prediction horizon window.

A correct call for a MPC optimization should look like:

```bash
curl -i -H 'Content-Type:application/json' -X POST -d '{"pv_power_forecast":[0, 70, 141.22, 246.18, 513.5, 753.27, 1049.89, 1797.93, 1697.3, 3078.93], "prediction_horizon":10, "soc_init":0.5,"soc_final":0.6}' http://192.168.3.159:5000/action/naive-mpc-optim
```
*Example with :`def_total_hours`, `def_start_timestep`, `def_end_timestep`.*
```bash
curl -i -H 'Content-Type:application/json' -X POST -d '{"pv_power_forecast":[0, 70, 141.22, 246.18, 513.5, 753.27, 1049.89, 1797.93, 1697.3, 3078.93], "prediction_horizon":10, "soc_init":0.5,"soc_final":0.6,"def_total_hours":[1,3],"def_start_timestep":[0,3],"def_end_timestep":[0,6]}' http://localhost:5000/action/naive-mpc-optim
```

## A machine learning forecaster

Starting in v0.4.0 a new machine learning forecaster class was introduced.
This is intended to provide a new and alternative method to forecast your household consumption and use it when such forecast is needed to optimize your energy through the available strategies.
Check the dedicated section in the documentation here: [https://emhass.readthedocs.io/en/latest/mlforecaster.html](https://emhass.readthedocs.io/en/latest/mlforecaster.html)

## Development

Pull request are very much accepted on this project. For development you can find some instructions here [Development](https://emhass.readthedocs.io/en/latest/develop.html).

## Troubleshooting

Some problems may arise from solver related issues in the Pulp package. It was found that for arm64 architectures (ie. Raspberry Pi4, 64 bits) the default solver is not avaliable. A workaround is to use another solver. The `glpk` solver is an option.

This can be controlled in the configuration file with parameters `lp_solver` and `lp_solver_path`. The options for `lp_solver` are: 'PULP_CBC_CMD', 'GLPK_CMD' and 'COIN_CMD'. If using 'COIN_CMD' as the solver you will need to provide the correct path to this solver in parameter `lp_solver_path`, ex: '/usr/bin/cbc'.

## License

MIT License

Copyright (c) 2021-2023 David HERNANDEZ

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


.. image:: https://img.shields.io/pypi/v/distutils.svg
   :target: https://pypi.org/project/distutils

.. image:: https://img.shields.io/pypi/pyversions/distutils.svg

.. image:: https://github.com/pypa/distutils/actions/workflows/main.yml/badge.svg
   :target: https://github.com/pypa/distutils/actions?query=workflow%3A%22tests%22
   :alt: tests

.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json
    :target: https://github.com/astral-sh/ruff
    :alt: Ruff

.. .. image:: https://readthedocs.org/projects/PROJECT_RTD/badge/?version=latest
..    :target: https://PROJECT_RTD.readthedocs.io/en/latest/?badge=latest

.. image:: https://img.shields.io/badge/skeleton-2024-informational
   :target: https://blog.jaraco.com/skeleton

Python Module Distribution Utilities extracted from the Python Standard Library

This package is unsupported except as integrated into and exposed by Setuptools.

Integration
-----------

Simply merge the changes directly into setuptools' repo.


# MemberMatters

MemberMatters is an open source membership, billing and access control solution for makerspaces and community groups.

### Main Features

* Supports all major browsers with mobile support and dark mode.
* Kiosk mode (electron) with site sign/sign out feature to restrict access to tools.
* Member registration system (signup, login, profile etc).
* Recurring membership payments via credit card with Stripe including:
  * Integration with Canvas or Moodle for a mandatory online induction course during sign up.
  * Support for manual account activation/deactivations for members without a Stripe subscription.
  * Self service billing (ie members can cancel their membership or update credit card themselves).
  * Automatic swipe card deactivation on overdue payments.
  * Automatic swipe card activation for returning members.
* Single Sign On (SSO) provider with support for both OIDC and the discourse protocol.
* Issue reporting form with Trello integration.
* Access Control system built in including:
  * Access permissions checking screen for members.
  * Default access permissions for new members.
  * Audit logs (including interlock time used and posts to Discord) for every successful/failed swipe.
  * Integration with our Access Control firmware which supports:
    * Communication over websockets for flexibility in placement (your MM server can be offsite!).
    * Offline cache that is updated automatically in realtime as permissions change.
    * Remote unlocking, syncing and rebooting of devices.
    * Controlling access to most types of doors and tools.
    * Memberbucks purchases via RFID and integration with some vending machines (MDB support coming soon).
* Integration with Sendgrid/Postmark for emails and Twilio for SMS notifications including:
  * SMS and email alerts for swipe card deactivation/failed swipes etc.
  * Manual SMS notifications to members for urgent issues (they can take a while to check emails)
* Admin Interface for managing all aspects of the system (and Django admin fills in the gaps).
* Show members Webcam snapshots, latest swipes, who's on site, and other member tools.
* Full "Memberbucks" Support for on site and in-portal payments.
* And many more! Feel free to open an issue or discussion if you see anything missing or want to suggest new features.

## Getting Started

See the [getting started](/docs/GETTING_STARTED.md) instructions for how to run the MemberMatters software for your organisation. Once you've finished installing MemberMatters, your should read the [post installation steps](/docs/POST_INSTALL_STEPS.md) for important instructions on setting up and configuring your instance.

MemberMatters is production quality software and has been used by several makerspaces for over 5 years. However, as an open source project, we focus our limited resources on features and bug fixes so our documentation can always be improved. Please open an issue if you have any suggestions to make it better.

### Compatibility

Officially, the only supported way to run MemberMatters is with the official Docker Hub image on a Ubuntu Server LTS host. Being Docker, you should be able to run it on various other platforms, but minimal support will be available. There is both an AMD64 and ARM64 build on Dockerhub for each release.

## Updates and Releases

The official changelog is available on the website [here](/CHANGELOG.md). You will also find each release [here](https://github.com/membermatters/MemberMatters/releases) on GitHub.

## Door, Interlock & Vending Machine Hardware

As part of our access control system (including doors, tool interlocks & vending machine payments) @jabelone has developed a "mainboard" PCB and firmware that is compatible with MemberMatters. The second iteration of this PCB has proven quite reliable over the last ~2yrs, despite a handful of hardware bugs. This PCB will be published eventually as the "standard" access control hardware for use with MemberMatters once (if) this design reaches production ready quality. It will not be published before this to prevent issues/disappointment/support requests.

However, our Access Control Firmware should run on most ESP32 devices. It is also compatible with many off the shelf dev boards and peripherals including UART and Wiegand RFID readers, 16x2 i2c LCD screens, etc. To use this firmware, connect a compatible RFID reader, some sort of output (contactor for a tool interlock, electronic door strike, vend relay etc.) and configure it to connect to your MemberMatters instance. Once connected, you can authorise it and use it within MemberMatters.

You can read more about how to use the firmware in it's repo at [MemberMatters Access Control Firmware](https://github.com/membermatters/mainboard-firmware).

## Screenshots

### Mobile

### Desktop

### Desktop (Dark Mode)

See the [screenshots](screenshots) folder for more screenshots.

# Developers Information

## Getting Started
[![Build Docker Image (Prod)](https://github.com/membermatters/MemberMatters/actions/workflows/build_docker.yml/badge.svg?branch=main)](https://github.com/membermatters/MemberMatters/actions/workflows/build_docker.yml)

[![Build Docker Image (Dev Branch)](https://github.com/membermatters/MemberMatters/actions/workflows/build_docker.dev.yml/badge.svg)](https://github.com/membermatters/MemberMatters/actions/workflows/build_docker.dev.yml)

### Pre-Commit Hooks

We use husky and lint-staged to manage pre commit hooks. The first thing you should do is run `npm install` in this
directory. This installs and configures the pre commit hooks automatically. After doing this, you should see them run
when you try to commit a file (for example, with `git commit -m "update thing"`).

### Development tip (dev server proxy)

The Vue.js (Quasar) frontend and Django backend run different dev servers on different ports. Due to
this you will run into issues with sharing cookies etc across the different URIs. To get around
this, the webpack dev server has been setup to proxy all requests to `/api` to `localhost:8000` so
you'll need to make sure the django dev server is running there.

### Vue.js JavaScript frontend

Please see the [readme](src-frontend/README.md) file inside the `src-frontend` folder. This folder contains all of the source
code and other assets needed by the frontend.

### Django Python backend

Please see the [readme](memberportal/README.md) file inside the `memberportal` folder. This folder contains all of the source
code and other assets needed by the backend.

## Contributing guidelines

By contributing code, or any other resource to this repository, you agree to license it
under the open source MIT license. Please use good coding practices, comment
your code well and ensure compliance with any code formatting or linting that's in place. Also
avoid "weird tricks" and optimisations that don't read easily - this is a web app, not a high
performance algorithm. Your contributions *will* be rejected if you do not follow these guidelines so please be careful.

All PRs should be made from your own branch/fork into the `dev` branch. Every now and again we'll collect the changes in dev, do up a release, and push to `main`. The `main` branch should always contain the most recent release.

## Release Checklist

We use [semantic versioning](https://semver.org).

* Update package.json version.
* Update src-frontend/package.json version.
* Update CHANGELOG.md with all changes since the last version.
* Make a new commit with above changes and commit title in the format of e.g. `v3.4.5`.
* Once this is merged into the `main` branch, tag the merge commit with the same name and GHA will build/deploy the docker image.
* Create a [new release on GitHub](https://github.com/membermatters/MemberMatters/releases/new)
  * Choose the tag you just created.
  * For the title use the version number as above e.g. `v3.4.5`.
  * For the description use the new entries added to the changelog.
* Close out any issues fixed by this release (preferably after confirming that it fixes it).

# Organisations using MemberMatters

Feel free to add your organisation to this list (via a pull request) if you're actively using or trialling MemberMatters. Make sure to include a link to your website and the date you added it.

* [BMS (Brisbane Makerspace)](https://brisbanemaker.space) (October 2021) - Brisbane's friendliest community workshop based in Brisbane, QLD, Australia.
* [Make Monmouth](https://www.makemonmouth.co.uk/) (November 2023) - A community makerspace based in Monmouth, Wales, UK.
* [SparkCC](https://www.sparkcc.org) (September 2021) - A community of makers on the NSW Central Coast based in Palmdale, NSW, Australia.


Augraphy is a Python library that creates multiple copies of original documents though an augmentation pipeline that randomly distorts each copy -- degrading the clean version into dirty and realistic copies rendered through synthetic paper printing, faxing, scanning and copy machine processes.

Highly-configurable pipelines apply adjustments to the originals to create realistic old or noisy documents by acting as a factory, producing almost an infinite number of variations from their source.  This simulation of realistic paper-oriented process distortions can create large amounts of training data for AI/ML processes to learn how to remove those distortions.

Treatments applied by Augraphy fabricate realistic documents that appear to have been printed on dirty laser or inkjet printers, scanned by dirty office scanners, faxed by low-resolution fax machines and otherwise mistreated by real-world paper handling office equipment.

# What makes Augraphy Magical?
Virtually no readily available datasets exist with both a _clean_ and _noisy_ version of target documents.  Augraphy addresses that problem by manufacturing large volumes of high-quality noisy documents to train alongside their clean source originals.

Training neural networks typically requires augmenting limited sources of data in a variety of ways so that networks can learn to generalize their solutions.  Networks designed to work with scanned document images must be trained with images that have the type of distortions and noise typical of real-world scanned office documents.

However, if we only have real-world dirty documents, then we don’t have a good way to know for sure what the right answer is when training a neural network.  By going in the reverse direction, starting with the clean document we hope a trained network will produce, we can simulate training data with dirty documents for which we already have a perfect original.

With flawless rendering of distorted "originals", we can train a model to undo all that distortion and restore the document to its original form.  It’s pretty much magic!

# How It Works
Augraphy's augmentation pipeline starts with an image of a clean document.  The pipeline begins by extracting the text and graphics from the source into an "ink" layer.  (Ink is synonymous with toner within Augraphy.)  The augmentation pipeline then distorts and degrades the ink layer.

A paper factory provides either a white page or a randomly-selected paper texture base.  Like the ink layer, the paper can also be processed through a pipeline to further provide random realistic paper textures.

After both the ink and paper phases are completed, processing continues by applying the ink, with its desired effects, to the paper.  This merged document image is then augmented further with distortions such as adding folds or other physical deformations or distortions that rely on simultaneous interactions of paper and ink layers.

The end result is an image that mimics real documents.

## Example Before / After Images

# Example Usage
To use the default pipeline which contains all available augmentations and sensible defaults:

```python
from augraphy import *

pipeline = default_augraphy_pipeline()

image = cv2.imread("image.png")

augmented = pipeline(image)

```

# Documentation
For full documentation, including installation and tutorials, check the [doc directory](https://github.com/sparkfish/augraphy/tree/dev/doc).

# List of Augmentations
## Pixel Level Augmentations
Pixel level augmentations apply augmentation to the input image only, that including alpha layer of the image. Additional inputs such as mask, keypoints or bounding boxes will not be affected.

|    Augmentation    |      Image      |   Alpha Layer   |
|--------------------|----------------:|----------------:|
|BadPhotoCopy        |        ✓        |        -        |
|BindingsAndFasteners|        ✓        |        -        |
|BleedThrough        |        ✓        |        -        |
|Brightness          |        ✓        |        -        |
|BrightnessTexturize |        ✓        |        -        |
|ColorPaper          |        ✓        |        -        |
|ColorShift          |        ✓        |        -        |
|DelaunayTessellation|        ✓        |        -        |
|DirtyDrum           |        ✓        |        -        |
|DirtyRollers        |        ✓        |        -        |
|Dithering           |        ✓        |        -        |
|DotMatrix           |        ✓        |        -        |
|Faxify              |        ✓        |        -        |
|Gamma               |        ✓        |        -        |
|Hollow              |        ✓        |        -        |
|InkBleed            |        ✓        |        -        |
|InkColorSwap        |        ✓        |        -        |
|InkMottling         |        ✓        |        -        |
|Jpeg                |        ✓        |        -        |
|Letterpress         |        ✓        |        -        |
|LightingGradient    |        ✓        |        -        |
|LinesDegradation    |        ✓        |        -        |
|LowInkPeriodicLines |        ✓        |        -        |
|LowInkRandomLines   |        ✓        |        -        |
|LowLightNoise       |        ✓        |        -        |
|Markup              |        ✓        |        -        |
|NoiseTexturize      |        ✓        |        -        |
|NoisyLines          |        ✓        |        -        |
|PatternGenerator    |        ✓        |        -        |
|ReflectedLight      |        ✓        |        -        |
|Scribbles           |        ✓        |        -        |
|ShadowCast          |        ✓        |        -        |
|SubtleNoise         |        ✓        |        -        |
|VoronoiTessellation |        ✓        |        -        |
|WaterMark           |        ✓        |        -        |

## Spatial level Augmentations
Spatial level augmentations apply augmentation to all inputs such as image (including alpha layer), mask, keypoints and bounding boxes.

|    Augmentation    |      Image      |   Alpha Layer   |       Mask      |    Keypoints    | Bounding Boxes  |
|--------------------|----------------:|----------------:|----------------:|----------------:|----------------:|
|BookBinding         |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|Folding             |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|Geometric           |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|GlitchEffect        |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|InkShifter          |        ✓        |        ✓        |        ✓        |        ✕        |        ✕        |
|PageBorder          |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|SectionShift        |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |
|Squish              |        ✓        |        ✓        |        ✓        |        ✓        |        ✓*       |

Remarks:

[-] : augmentation doesn't affect this input.

[✓] : augmentation is supported on this input.

[✕] : augmentation is not supported on this input.

[✓*] : augmentation is supported on this input under certain criteria.

# Benchmark Results
The benchmark results are computed with Augraphy 8.2 and Tobacco3482 dataset (resume subset with a total of 120 images). It is evaluated with a 2 cores machine - Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz. The mask is using a binarized version of image. For keypoints, it is using 1000 random points in the image and for bounding boxes, 100 random bounding boxes with random size are used.

|    Augmentation    |Img/sec|Memory(MB)|Img/sec(mask)|Memory(MB)(mask)|Img/sec(keypoints)|Memory(MB)(keypoints)|Img/sec(bounding boxes)|Memory(MB)(bounding boxes)|
|--------------------|------:|---------:|------------:|---------------:|-----------------:|--------------------:|----------------------:|-------------------------:|
|BadPhotoCopy        |   0.17|    202.81|         0.12|          215.60|              0.13|               204.55|                   0.12|                    216.29|
|BindingsAndFasteners|  28.21|     21.08|        29.75|           21.02|             36.39|                21.02|                  29.09|                     21.23|
|BleedThrough        |   0.39|    684.69|         0.39|          684.69|              0.39|               684.69|                   0.39|                    684.69|
|BookBinding         |   0.09|    611.71|         0.08|          616.87|              0.09|               611.74|                   0.09|                    611.67|
|Brightness          |   4.92|    147.99|         4.90|          147.99|              4.95|               147.99|                   5.07|                    147.99|
|BrightnessTexturize |   1.83|    181.74|         1.83|          181.74|              1.84|               181.74|                   1.86|                    181.74|
|ColorPaper          |   4.83|    105.66|         4.99|          105.66|              4.96|               105.66|                   4.96|                    105.66|
|ColorShift          |   0.79|    126.94|         0.82|          126.94|              0.77|               126.94|                   0.76|                    126.94|
|DelaunayTessellation|   0.11|     60.41|         0.12|           60.29|              0.11|                60.36|                   0.10|                     60.37|
|DepthSimulatedBlur  |   0.01|     76.08|         0.01|           76.08|              0.01|                76.08|                   0.01|                     76.08|
|DirtyDrum           |   0.83|    482.51|         0.94|          481.56|              0.90|               481.68|                   0.92|                    481.52|
|DirtyRollers        |   1.47|    249.55|         1.77|          249.43|              1.80|               249.43|                   1.78|                    249.43|
|DirtyScreen         |   0.78|    435.36|         0.78|          435.36|              0.77|               435.36|                   0.77|                    435.36|
|Dithering           |   3.39|    126.82|         3.66|          126.80|              3.66|               126.81|                   3.79|                    126.80|
|DotMatrix           |   0.53|     80.75|         0.57|           80.52|              0.57|                80.52|                   0.57|                     80.52|
|DoubleExposure      |   1.64|     63.40|         1.67|           63.40|              1.62|                63.40|                   1.65|                     63.40|
|Faxify              |   1.37|    138.34|         1.43|          141.28|              1.41|               136.95|                   1.27|                    149.24|
|Folding             |   3.18|     57.50|         1.24|           60.40|              3.44|                57.60|                   3.55|                     57.20|
|Gamma               |  29.26|     25.36|        26.90|           25.36|             28.23|                25.36|                  32.03|                     25.36|
|Geometric           | 135.75|     12.68|       137.64|           12.68|            145.80|                12.68|                 127.10|                     12.68|
|GlitchEffect        |   1.14|    132.35|         1.10|          132.65|              1.03|               134.14|                   1.05|                    134.11|
|Hollow              |   0.17|    343.17|         0.17|          343.17|              0.17|               343.17|                   0.17|                    343.17|
|InkBleed            |   3.23|    177.51|         3.19|          177.51|              3.17|               177.51|                   3.16|                    177.51|
|InkColorSwap        |   3.47|     51.99|         3.58|           51.99|              3.51|                51.99|                   3.61|                     51.99|
|InkMottling         |   5.41|     55.99|         5.47|           55.99|              5.49|                55.99|                   5.39|                     55.99|
|InkShifter          |   0.17|    426.86|         0.15|          426.43|              0.17|               426.78|                   0.17|                    426.58|
|LCDScreenPattern    |   2.14|    494.09|         2.12|          493.62|              2.14|               494.74|                   2.13|                    496.46|
|Jpeg                |   5.55|     25.87|         5.60|           25.86|              5.52|                25.87|                   5.66|                     25.87|
|LensFlare           |   0.02|    405.97|         0.01|          405.82|              0.01|               405.82|                   0.01|                    405.82|
|Letterpress         |   0.35|    135.71|         0.33|          140.72|              0.34|               137.25|                   0.34|                    136.31|
|LightingGradient    |   0.37|    638.31|         0.38|          638.30|              0.39|               638.30|                   0.40|                    638.30|
|LinesDegradation    |   1.28|    174.76|         1.27|          174.93|              1.28|               174.92|                   1.31|                    174.59|
|LowInkPeriodicLines |   5.17|     12.75|         5.26|           12.75|              5.56|                12.75|                   5.10|                     12.75|
|LowInkRandomLines   |  91.52|     12.75|        86.12|           12.75|             87.58|                12.75|                  98.28|                     12.75|
|LowLightNoise       |   0.27|    481.95|         0.28|          481.95|              0.27|               481.95|                   0.27|                    481.95|
|Markup              |   2.33|    161.88|         2.41|          158.13|              2.58|               146.27|                   2.60|                    147.53|
|Moire               |   0.97|    575.74|         1.03|          575.57|              1.05|               575.57|                   1.05|                    575.57|
|NoiseTexturize      |   0.83|    249.36|         0.85|          249.36|              0.80|               249.36|                   0.82|                    249.36|
|NoisyLines          |   0.89|    446.65|         0.83|          448.43|              0.86|               447.88|                   0.85|                    448.52|
|PageBorder          |   0.49|    193.95|         0.49|          188.46|              0.48|               188.30|                   0.49|                    192.04|
|PatternGenerator    |   0.76|     51.53|         0.76|           51.50|              0.74|                51.50|                   0.74|                     51.50|
|ReflectedLight      |   0.06|    109.90|         0.06|          109.82|              0.06|               109.88|                   0.06|                    110.02|
|Scribbles           |   1.11|     94.73|         0.86|           96.90|              0.87|                96.96|                   0.86|                    100.55|
|SectionShift        | 117.15|     12.96|       101.71|           13.02|            107.88|                12.95|                 115.22|                     12.96|
|ShadowCast          |   0.75|     50.79|         0.68|           50.80|              0.68|                50.80|                   0.74|                     50.80|
|Squish              |   0.72|    450.44|         0.72|          450.79|              0.73|               450.83|                   0.76|                    451.00|
|Stains              |   1.11|    469.14|         1.14|          469.14|              1.16|               469.14|                   1.11|                    469.14|
|SubtleNoise         |   1.44|    215.55|         1.47|          215.55|              1.47|               215.55|                   1.48|                    215.55|
|VoronoiTessellation |   0.07|     58.07|         0.07|           57.74|              0.07|                57.89|                   0.07|                     58.13|
|WaterMark           |   2.09|    363.62|         1.86|          404.62|              1.78|               409.50|                   2.02|                    380.21|

# Alternative Augmentation Libraries
There are plenty of choices when it comes to [augmentation libraries](https://github.com/AgaMiko/data-augmentation-review).  However, only Augraphy is designed to address everyday office automation needs associated with paper-oriented process distortions that come from printing, faxing, scanning and copy machines.  Most other libraries focus on video and images pertinent to camera-oriented data sources and problem domains.  Augraphy is focused on supporting problems related to automation of document images such as OCR, form recognition, form data extraction, document classification, barcode decoding, denoising, document restoration, identity document data extraction, document cropping, etc.  Eventually, Augraphy will be able to support photo OCR problems with augmentations designed to emulate camera phone distortions.

# Contributing
Pull requests are _very_ welcome.  Please open an issue to propose and discuss feature requests and major changes.

# Citations
If you used Augraphy in your research, please cite the project.

BibTeX:
```
@inproceedings{augraphy_paper,
    author = {Groleau, Alexander and Chee, Kok Wei and Larson, Stefan and Maini, Samay and Boarman, Jonathan},
    title = {Augraphy: A Data Augmentation Library for Document Images},
    booktitle = {Proceedings of the 17th International Conference on Document Analysis and Recognition ({ICDAR})},
    year = {2023},
    url = {https://arxiv.org/pdf/2208.14558.pdf}
}

@software{augraphy_library,
    author = {The Augraphy Project},
    title = {Augraphy: an augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes},
    url = {https://github.com/sparkfish/augraphy},
    version = {8.2.6}
}
```

## Star History
Please add a "star" to the repo.  It's exciting to us when we see your interest, which keep us motivated to continue investing in the project!
<picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=Sparkfish/augraphy&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=Sparkfish/augraphy&type=Date" />

</picture>

# License
Copyright 2023 Sparkfish LLC

Augraphy is free and open-source software distributed under the terms of the [**MIT**](https://github.com/sparkfish/augraphy/blob/dev/LICENSE) license.


# Infrastructure for deployments

This repository contains deployment infrastructure and documentation for a federation of JupyterHubs that 2i2c manages for various communities.

See [the infrastructure documentation](https://infrastructure.2i2c.org) for more information.

## Building the documentation

The documentation is built with [the Sphinx documentation engine](https://sphinx-doc.org).

### Automatically with `nox`

The easiest way to build the documentation in this repository is to use [the `nox` automation tool](https://nox.thea.codes/), a tool for quickly building environments and running commands within them.
This ensures that your environment has all the dependencies needed to build the documentation.

To do so, follow these steps:

1. Install `nox`

   ```console
   $ pip install nox
   ```

2. Build the documentation:

   ```console
   $ nox -s docs
   ```

This should create a local environment in a `.nox` folder, build the documentation (as specified in the `noxfile.py` configuration), and the output will be in `docs/_build/dirhtml`.

To build live documentation that updates when you update local files, run the following command:

```console
$ nox -s docs -- live
```

### Manually with `conda`

If you wish to manually build the documentation, you can use `conda` to do so.

1. Create a `conda` environment to build the documentation.

   ```bash
   conda env create -f docs/environment.yml -n infrastructure-docs
   ```

2. Activate the new environment:

   ```bash
   conda activate infrastructure-docs
   ```

3. Build the documentation:

   ```bash
   make html
   ```

This will generate the HTML for the documentation in the `docs/_build/dirhtml` folder.
You may preview the documentation by opening any of the `.html` files inside.

### Build the documentation with a live server

You can optionally build the documentation with a **live server** to automatically preview the changes as you build the docs. To use this, run `make live` instead of `make html`.

### Check for broken links

You can check for broken links in our documentation with the [Sphinx linkcheck builder](https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-the-linkcheck-builder).
This will build the documentation and test every link to make sure that it resolves properly.
We use a GitHub Action to check this in our CI/CD, so this generally shouldn't be needed unless you want to manually test something.
To check our documentation for broken links, run the following command from the `docs/` folder:

```bash
make linkcheck
```

This will build the documentation, reporting broken links as it goes.
It will output a summary of all links in a file at `docs/_build/linkcheck/output.txt`.