![Lightly SSL self-supervised learning Logo](docs/logos/lightly_SSL_logo_crop.png)

![GitHub](https://img.shields.io/github/license/lightly-ai/lightly)
![Unit Tests](https://github.com/lightly-ai/lightly/workflows/Unit%20Tests/badge.svg)
[![PyPI](https://img.shields.io/pypi/v/lightly)](https://pypi.org/project/lightly/)
[![Downloads](https://static.pepy.tech/badge/lightly)](https://pepy.tech/project/lightly)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Discord](https://img.shields.io/discord/752876370337726585?logo=discord&logoColor=white&label=discord&color=7289da)](https://discord.gg/xvNJW94)

Lightly SSL is a computer vision framework for self-supervised learning.

- [Documentation](https://docs.lightly.ai/self-supervised-learning/)
- [Github](https://github.com/lightly-ai/lightly)
- [Discord](https://discord.gg/xvNJW94) (We have weekly paper sessions!)

For a commercial version with more features, including Docker support and pretraining
models for embedding, classification, detection, and segmentation tasks with
a single command, please contact sales@lightly.ai.

We've also built a whole platform on top, with additional features for active learning
and [data curation](https://docs.lightly.ai/docs/what-is-lightly). If you're interested in the
Lightly Worker Solution to easily process millions of samples and run [powerful algorithms](https://docs.lightly.ai/docs/selection)
on your data, check out [lightly.ai](https://www.lightly.ai). It's free to get started!

## Features

This self-supervised learning framework offers the following features:

- Modular framework, which exposes low-level building blocks such as loss functions and
  model heads.
- Easy to use and written in a PyTorch like style.
- Supports custom backbone models for self-supervised pre-training.
- Support for distributed training using PyTorch Lightning.

### Supported Models

You can [find sample code for all the supported models here.](https://docs.lightly.ai/self-supervised-learning/examples/models.html) We provide PyTorch, PyTorch Lightning,
and PyTorch Lightning distributed examples for all models to kickstart your project.

**Models**:

- AIM, 2024 [paper](https://arxiv.org/abs/2401.08541) [docs](https://docs.lightly.ai/self-supervised-learning/examples/aim.html)
- Barlow Twins, 2021 [paper](https://arxiv.org/abs/2103.03230) [docs](https://docs.lightly.ai/self-supervised-learning/examples/barlowtwins.html)
- BYOL, 2020 [paper](https://arxiv.org/abs/2006.07733) [docs](https://docs.lightly.ai/self-supervised-learning/examples/byol.html)
- DCL & DCLW, 2021 [paper](https://arxiv.org/abs/2110.06848) [docs](https://docs.lightly.ai/self-supervised-learning/examples/dcl.html)
- DINO, 2021 [paper](https://arxiv.org/abs/2104.14294) [docs](https://docs.lightly.ai/self-supervised-learning/examples/dino.html)
- MAE, 2021 [paper](https://arxiv.org/abs/2111.06377) [docs](https://docs.lightly.ai/self-supervised-learning/examples/mae.html)
- MSN, 2022 [paper](https://arxiv.org/abs/2204.07141) [docs](https://docs.lightly.ai/self-supervised-learning/examples/msn.html)
- MoCo, 2019 [paper](https://arxiv.org/abs/1911.05722) [docs](https://docs.lightly.ai/self-supervised-learning/examples/moco.html)
- NNCLR, 2021 [paper](https://arxiv.org/abs/2104.14548) [docs](https://docs.lightly.ai/self-supervised-learning/examples/nnclr.html)
- PMSN, 2022 [paper](https://arxiv.org/abs/2210.07277) [docs](https://docs.lightly.ai/self-supervised-learning/examples/pmsn.html)
- SimCLR, 2020 [paper](https://arxiv.org/abs/2002.05709) [docs](https://docs.lightly.ai/self-supervised-learning/examples/simclr.html)
- SimMIM, 2021 [paper](https://arxiv.org/abs/2111.09886) [docs](https://docs.lightly.ai/self-supervised-learning/examples/simmim.html)
- SimSiam, 2021 [paper](https://arxiv.org/abs/2011.10566) [docs](https://docs.lightly.ai/self-supervised-learning/examples/simsiam.html)
- SMoG, 2022 [paper](https://arxiv.org/abs/2207.06167) [docs](https://docs.lightly.ai/self-supervised-learning/examples/smog.html)
- SwaV, 2020 [paper](https://arxiv.org/abs/2006.09882) [docs](https://docs.lightly.ai/self-supervised-learning/examples/swav.html)
- TiCo, 2022 [paper](https://arxiv.org/abs/2206.10698) [docs](https://docs.lightly.ai/self-supervised-learning/examples/tico.html)
- VICReg, 2022 [paper](https://arxiv.org/abs/2105.04906) [docs](https://docs.lightly.ai/self-supervised-learning/examples/vicreg.html)
- VICRegL, 2022 [paper](https://arxiv.org/abs/2210.01571) [docs](https://docs.lightly.ai/self-supervised-learning/examples/vicregl.html)

## Tutorials

Want to jump to the tutorials and see Lightly in action?

- [Train MoCo on CIFAR-10](https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_moco_memory_bank.html)
- [Train SimCLR on Clothing Data](https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_simclr_clothing.html)
- [Train SimSiam on Satellite Images](https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_simsiam_esa.html)
- [Use Lightly with Custom Augmentations](https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_custom_augmentations.html)
- [Pre-train a Detectron2 Backbone with Lightly](https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_pretrain_detectron2.html)

Community and partner projects:

- [On-Device Deep Learning with Lightly on an ARM microcontroller](https://github.com/ARM-software/EndpointAI/tree/master/ProofOfConcepts/Vision/OpenMvMaskDefaults)

## Quick Start

Lightly requires **Python 3.7+**. We recommend installing Lightly in a **Linux** or **OSX** environment. Python 3.12 is not yet supported, as PyTorch itself lacks Python 3.12 compatibility.

### Dependencies

Due to the modular nature of the Lightly package some modules can be used with older versions of dependencies. However, to use all features as of today lightly requires the following dependencies:

- [PyTorch](https://pytorch.org/)>=1.11.0
- [Torchvision](https://pytorch.org/vision/stable/index.html)>=0.12.0
- [PyTorch Lightning](https://www.pytorchlightning.ai/index.html)>=1.7.1

Lightly is compatible with PyTorch and PyTorch Lightning v2.0+!

### Installation

You can install Lightly and its dependencies from PyPI with:

```
pip3 install lightly
```

We strongly recommend that you install Lightly in a dedicated virtualenv, to avoid
conflicting with your system packages.

### Lightly in Action

With Lightly, you can use the latest self-supervised learning methods in a modular
way using the full power of PyTorch. Experiment with different backbones,
models, and loss functions. The framework has been designed to be easy to use
from the ground up. [Find more examples in our docs](https://docs.lightly.ai/self-supervised-learning/examples/models.html).

```python
import torch
import torchvision

from lightly import loss
from lightly import transforms
from lightly.data import LightlyDataset
from lightly.models.modules import heads

# Create a PyTorch module for the SimCLR model.
class SimCLR(torch.nn.Module):
    def __init__(self, backbone):
        super().__init__()
        self.backbone = backbone
        self.projection_head = heads.SimCLRProjectionHead(
            input_dim=512,  # Resnet18 features have 512 dimensions.
            hidden_dim=512,
            output_dim=128,
        )

    def forward(self, x):
        features = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(features)
        return z

# Use a resnet backbone.
backbone = torchvision.models.resnet18()
# Ignore the classification head as we only want the features.
backbone.fc = torch.nn.Identity()

# Build the SimCLR model.
model = SimCLR(backbone)

# Prepare transform that creates multiple random views for every image.
transform = transforms.SimCLRTransform(input_size=32, cj_prob=0.5)

# Create a dataset from your image folder.
dataset = LightlyDataset(input_dir="./my/cute/cats/dataset/", transform=transform)

# Build a PyTorch dataloader.
dataloader = torch.utils.data.DataLoader(
    dataset,  # Pass the dataset to the dataloader.
    batch_size=128,  # A large batch size helps with the learning.
    shuffle=True,  # Shuffling is important!
)

# Lightly exposes building blocks such as loss functions.
criterion = loss.NTXentLoss(temperature=0.5)

# Get a PyTorch optimizer.
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=1e-6)

# Train the model.
for epoch in range(10):
    for (view0, view1), targets, filenames in dataloader:
        z0 = model(view0)
        z1 = model(view1)
        loss = criterion(z0, z1)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        print(f"loss: {loss.item():.5f}")
```

You can easily use another model like SimSiam by swapping the model and the
loss function.

```python
# PyTorch module for the SimSiam model.
class SimSiam(torch.nn.Module):
    def __init__(self, backbone):
        super().__init__()
        self.backbone = backbone
        self.projection_head = heads.SimSiamProjectionHead(512, 512, 128)
        self.prediction_head = heads.SimSiamPredictionHead(128, 64, 128)

    def forward(self, x):
        features = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(features)
        p = self.prediction_head(z)
        z = z.detach()
        return z, p

model = SimSiam(backbone)

# Use the SimSiam loss function.
criterion = loss.NegativeCosineSimilarity()
```

You can [find a more complete example for SimSiam here.](https://docs.lightly.ai/self-supervised-learning/examples/simsiam.html)

Use PyTorch Lightning to train the model:

```python
from pytorch_lightning import LightningModule, Trainer

class SimCLR(LightningModule):
    def __init__(self):
        super().__init__()
        resnet = torchvision.models.resnet18()
        resnet.fc = torch.nn.Identity()
        self.backbone = resnet
        self.projection_head = heads.SimCLRProjectionHead(512, 512, 128)
        self.criterion = loss.NTXentLoss()

    def forward(self, x):
        features = self.backbone(x).flatten(start_dim=1)
        z = self.projection_head(features)
        return z

    def training_step(self, batch, batch_index):
        (view0, view1), _, _ = batch
        z0 = self.forward(view0)
        z1 = self.forward(view1)
        loss = self.criterion(z0, z1)
        return loss

    def configure_optimizers(self):
        optim = torch.optim.SGD(self.parameters(), lr=0.06)
        return optim

model = SimCLR()
trainer = Trainer(max_epochs=10, devices=1, accelerator="gpu")
trainer.fit(model, dataloader)
```

See [our docs for a full PyTorch Lightning example.](https://docs.lightly.ai/self-supervised-learning/examples/simclr.html)

Or train the model on 4 GPUs:

```python

# Use distributed version of loss functions.
criterion = loss.NTXentLoss(gather_distributed=True)

trainer = Trainer(
    max_epochs=10,
    devices=4,
    accelerator="gpu",
    strategy="ddp",
    sync_batchnorm=True,
    use_distributed_sampler=True,  # or replace_sampler_ddp=True for PyTorch Lightning <2.0
)
trainer.fit(model, dataloader)
```

We provide multi-GPU training examples with distributed gather and synchronized BatchNorm.
[Have a look at our docs regarding distributed training.](https://docs.lightly.ai/self-supervised-learning/getting_started/distributed_training.html)

## Benchmarks

Implemented models and their performance on various datasets. Hyperparameters are not
tuned for maximum accuracy. For detailed results and more information about the benchmarks click
[here](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html).

### ImageNet1k

[ImageNet1k benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenet1k)

**Note**: Evaluation settings are based on these papers:

- Linear: [SimCLR](https://arxiv.org/abs/2002.05709)
- Finetune: [SimCLR](https://arxiv.org/abs/2002.05709)
- KNN: [InstDisc](https://arxiv.org/abs/1805.01978)

See the [benchmarking scripts](./benchmarks/imagenet/resnet50/) for details.

| Model           | Backbone | Batch Size | Epochs | Linear Top1 | Finetune Top1 | kNN Top1 | Tensorboard                                                                                | Checkpoint                                                                                |
| --------------- | -------- | ---------- | ------ | ----------- | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BarlowTwins     | Res50    | 256        | 100    | 62.9        | 72.6          | 45.6     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_barlowtwins_2023-08-18_00-11-03/pretrain/version_0/events.out.tfevents.1692310273.Machine2.569794.0) | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_barlowtwins_2023-08-18_00-11-03/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt) |
| BYOL            | Res50    | 256        | 100    | 62.5        | 74.5          | 46.0     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_byol_2024-02-14_16-10-09/pretrain/version_0/events.out.tfevents.1707923418.Machine2.3205.0)          | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_byol_2024-02-14_16-10-09/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)        |
| DINO            | Res50    | 128        | 100    | 68.2        | 72.5          | 49.9     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dino_2023-06-06_13-59-48/pretrain/version_0/events.out.tfevents.1686052799.Machine2.482599.0)        | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dino_2023-06-06_13-59-48/pretrain/version_0/checkpoints/epoch%3D99-step%3D1000900.ckpt)       |
| MAE             | ViT-B/16 | 256        | 100    | 46.0        | 81.3          | 11.2     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_vitb16_mae_2024-02-25_19-57-30/pretrain/version_0/events.out.tfevents.1708887459.Machine2.1092409.0)          | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_vitb16_mae_2024-02-25_19-57-30/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)           |
| MoCoV2          | Res50    | 256        | 100    | 61.5        | 74.3          | 41.8     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_mocov2_2024-02-18_10-29-14/pretrain/version_0/events.out.tfevents.1708248562.Machine2.439033.0)      | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_mocov2_2024-02-18_10-29-14/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)      |
| SimCLR\*        | Res50    | 256        | 100    | 63.2        | 73.9          | 44.8     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_simclr_2023-06-22_09-11-13/pretrain/version_0/events.out.tfevents.1687417883.Machine2.33270.0)       | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_simclr_2023-06-22_09-11-13/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)      |
| SimCLR\* + DCL  | Res50    | 256        | 100    | 65.1        | 73.5          | 49.6     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dcl_2023-07-04_16-51-40/pretrain/version_0/events.out.tfevents.1688482310.Machine2.247807.0)         | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dcl_2023-07-04_16-51-40/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)         |
| SimCLR\* + DCLW | Res50    | 256        | 100    | 64.5        | 73.2          | 48.5     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dclw_2023-07-07_14-57-13/pretrain/version_0/events.out.tfevents.1688734645.Machine2.3176.0)          | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_dclw_2023-07-07_14-57-13/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)        |
| SwAV            | Res50    | 256        | 100    | 67.2        | 75.4          | 49.5     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_swav_2023-05-25_08-29-14/pretrain/version_0/events.out.tfevents.1684996168.Machine2.1445108.0)       | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_swav_2023-05-25_08-29-14/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)        |
| TiCo            | Res50    | 256        | 100    | 49.7        | 72.7          | 26.6     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_tico_2024-01-07_18-40-57/pretrain/version_0/events.out.tfevents.1704649265.Machine2.1604956.0)       | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_tico_2024-01-07_18-40-57/pretrain/version_0/checkpoints/epoch%3D99-step%3D250200.ckpt)        |
| VICReg          | Res50    | 256        | 100    | 63.0        | 73.7          | 46.3     | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_vicreg_2023-09-11_10-53-08/pretrain/version_0/events.out.tfevents.1694422401.Machine2.556563.0)      | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_vicreg_2023-09-11_10-53-08/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt)      |

_\*We use square root learning rate scaling instead of linear scaling as it yields
better results for smaller batch sizes. See Appendix B.1 in the [SimCLR paper](https://arxiv.org/abs/2002.05709)._

### ImageNet100

[ImageNet100 benchmarks detailed results](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenet100)

### Imagenette

[Imagenette benchmarks detailed results](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenette)

### CIFAR-10

[CIFAR-10 benchmarks detailed results](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#cifar-10)

## Terminology

Below you can see a schematic overview of the different concepts in the package.
The terms in bold are explained in more detail in our [documentation](https://docs.lightly.ai/self-supervised-learning/).

### Next Steps

Head to the [documentation](https://docs.lightly.ai/self-supervised-learning/) and see the things you can achieve with Lightly!

## Development

To install dev dependencies (for example to contribute to the framework) you can use the following command:

```
pip3 install -e ".[dev]"
```

For more information about how to contribute have a look [here](CONTRIBUTING.md).

### Running Tests

Unit tests are within the [tests directory](tests/) and we recommend running them using
[pytest](https://docs.pytest.org/en/stable/). There are two test configurations
available. By default, only a subset will be run:

```
make test-fast
```

To run all tests (including the slow ones) you can use the following command:

```
make test
```

To test a specific file or directory use:

```
pytest <path to file or directory>
```

### Code Formatting

To format code with [black](https://black.readthedocs.io/en/stable/) and [isort](https://docs.pytest.org) run:

```
make format
```

## Further Reading

**Self-Supervised Learning**:

- Have a look at our [#papers channel on discord](https://discord.com/channels/752876370337726585/815153188487299083)
  for the newest self-supervised learning papers.
- [A Cookbook of Self-Supervised Learning, 2023](https://arxiv.org/abs/2304.12210)
- [Masked Autoencoders Are Scalable Vision Learners, 2021](https://arxiv.org/abs/2111.06377)
- [Emerging Properties in Self-Supervised Vision Transformers, 2021](https://arxiv.org/abs/2104.14294)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, 2021](https://arxiv.org/abs/2006.09882)
- [What Should Not Be Contrastive in Contrastive Learning, 2020](https://arxiv.org/abs/2008.05659)
- [A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [Momentum Contrast for Unsupervised Visual Representation Learning, 2020](https://arxiv.org/abs/1911.05722)

## FAQ

- Why should I care about self-supervised learning? Aren't pre-trained models from ImageNet much better for transfer learning?

  - Self-supervised learning has become increasingly popular among scientists over the last years because the learned representations perform extraordinarily well on downstream tasks. This means that they capture the important information in an image better than other types of pre-trained models. By training a self-supervised model on _your_ dataset, you can make sure that the representations have all the necessary information about your images.

- How can I contribute?

  - Create an issue if you encounter bugs or have ideas for features we should implement. You can also add your own code by forking this repository and creating a PR. More details about how to contribute with code is in our [contribution guide](CONTRIBUTING.md).

- Is this framework for free?

  - Yes, this framework is completely free to use and we provide the source code. We believe that we need to make training deep learning models more data efficient to achieve widespread adoption. One step to achieve this goal is by leveraging self-supervised learning. The company behind Lightly is committed to keep this framework open-source.

- If this framework is free, how is the company behind Lightly making money?
  - Training self-supervised models is only one part of our solution.
    [The company behind Lightly](https://lightly.ai/) focuses on processing and analyzing embeddings created by self-supervised models.
    By building, what we call a self-supervised active learning loop we help companies understand and work with their data more efficiently.
    As the [Lightly Solution](https://docs.lightly.ai) is a freemium product, you can try it out for free. However, we will charge for some features.
  - In any case this framework will always be free to use, even for commercial purposes.

## Lightly in Research

- [Reverse Engineering Self-Supervised Learning, 2023](https://arxiv.org/abs/2305.15614)
- [Learning Visual Representations via Language-Guided Sampling, 2023](https://arxiv.org/pdf/2302.12248.pdf)
- [Self-Supervised Learning Methods for Label-Efficient Dental Caries Classification, 2022](https://www.mdpi.com/2075-4418/12/5/1237)
- [DPCL: Constrative Representation Learning with Differential Privacy, 2022](https://assets.researchsquare.com/files/rs-1516950/v1_covered.pdf?c=1654486158)
- [Decoupled Contrastive Learning, 2021](https://arxiv.org/abs/2110.06848)
- [solo-learn: A Library of Self-supervised Methods for Visual Representation Learning, 2021](https://www.jmlr.org/papers/volume23/21-1155/21-1155.pdf)

## Company behind this Open Source Framework

[Lightly](https://www.lightly.ai) is a spin-off from ETH Zurich that helps companies
build efficient active learning pipelines to select the most relevant data for their models.

You can find out more about the company and it's services by following the links below:

- [Homepage](https://www.lightly.ai)
- [Web-App](https://app.lightly.ai)
- [Lightly Solution Documentation (Lightly Worker & API)](https://docs.lightly.ai/)

## BibTeX

If you want to cite the framework feel free to use this:

```bibtex
@article{susmelj2020lightly,
  title={Lightly},
  author={Igor Susmelj and Matthias Heller and Philipp Wirth and Jeremy Prescott and Malte Ebner et al.},
  journal={GitHub. Note: https://github.com/lightly-ai/lightly},
  year={2020}
}
```


MapProxy is an open source proxy for geospatial data. It caches, accelerates and transforms data from existing map services and serves any desktop or web GIS client.

MapProxy is a tile cache, but also offers many new and innovative features like full support for WMS clients.

MapProxy is released under the Apache Software License 2.0, runs on Unix/Linux and Windows and is easy to install and to configure.

Go to https://mapproxy.org/ for more information.

The documentation is available at: http://mapproxy.github.io/mapproxy/latest/


elastic-apm -- Elastic APM agent for Python
===========================================

.. image:: https://github.com/elastic/apm-agent-python/actions/workflows/test.yml/badge.svg?branch=main
    :target: https://github.com/elastic/apm-agent-python/actions/workflows/test.yml
    :alt: Build Status

.. image:: https://img.shields.io/pypi/v/elastic-apm.svg?style=flat
    :target: https://pypi.python.org/pypi/elastic-apm/
    :alt: Latest Version

.. image:: https://img.shields.io/pypi/pyversions/elastic-apm.svg?style=flat
    :target: https://pypi.python.org/pypi/elastic-apm/
    :alt: Supported Python versions

This is the official Python module for Elastic APM.

It provides full out-of-the-box support for many of the popular frameworks,
including Django, and Flask. Elastic APM is also easy to adapt for most
WSGI-compatible web applications via `custom integrations`_.

Your application doesn't live on the web? No problem! Elastic APM is easy to use in
any Python application.

Read the documentation_, including instructions on `running the tests locally`_.

If you're interested in contributing, `start here!`_

.. _documentation: https://www.elastic.co/guide/en/apm/agent/python/current/index.html
.. _`custom integrations`: https://www.elastic.co/blog/creating-custom-framework-integrations-with-the-elastic-apm-python-agent
.. _`running the tests locally`: https://www.elastic.co/guide/en/apm/agent/python/current/run-tests-locally.html
.. _`start here!`: https://github.com/elastic/apm-agent-python/blob/main/CONTRIBUTING.md

License
-------

BSD-3-Clause

Made with ♥ and ☕ by Elastic and our community.


# About
Microsoft 365 & Microsoft Graph library for Python

# Usage

1. [Installation](#Installation)
2. [Working with SharePoint API](#Working-with-SharePoint-API)
3. [Working with Outlook API](#Working-with-Outlook-API)
4. [Working with OneDrive and SharePoint API v2 APIs](#working-with-onedrive-and-sharepoint-v2-apis)
5. [Working with Teams API](#Working-with-Microsoft-Teams-API)
6. [Working with OneNote API](#Working-with-Microsoft-OneNote-API)
7. [Working with Planner API](#Working-with-Microsoft-Planner-API)

## Status
[![Downloads](https://pepy.tech/badge/office365-rest-python-client/month)](https://pepy.tech/project/office365-rest-python-client)
[![PyPI](https://img.shields.io/pypi/v/Office365-REST-Python-Client.svg)](https://pypi.python.org/pypi/Office365-REST-Python-Client)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/Office365-REST-Python-Client.svg)](https://pypi.python.org/pypi/Office365-REST-Python-Client/)
[![Build Status](https://travis-ci.com/vgrem/Office365-REST-Python-Client.svg?branch=master)](https://travis-ci.com/vgrem/Office365-REST-Python-Client)

# Installation

Use pip:

```
pip install Office365-REST-Python-Client
```

### Note
>
>Alternatively the _latest_ version could be directly installed via GitHub:
>```
>pip install git+https://github.com/vgrem/Office365-REST-Python-Client.git
>```

# Authentication
For the following examples, relevant credentials can be found in the Azure Portal.

Steps to access:
1. Login to the home page of the Azure Portal
2. Navigate to "Azure Active Directory" using the three bars in the top right corner of the portal
3. Select "App registrations" in the navigation panel on the left
4. Search for and select your relevant application
5. In the application's "Overview" page, the client id can be found under "Application (client) id"
6. In the application's "Certificates & Secrets" page, the client secret can be found under the "Value" of the "Client Secrets." If there is no client secret yet, create one here.

# Working with SharePoint API

   The `ClientContext` client provides the support for a legacy SharePoint REST and OneDrive for Business REST APIs,
   the list of supported versions:
   -   [SharePoint 2013 REST API](https://msdn.microsoft.com/en-us/library/office/jj860569.aspx) and above
   -   [SharePoint Online REST API](https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/get-to-know-the-sharepoint-rest-service)
   -   OneDrive for Business REST API

### Authentication

   The following auth flows are supported:

#### 1. Using a SharePoint App-Only principal (client credentials flow)

   This auth method is compatible with SharePoint on-premises and still relevant
   model in both SharePoint on-premises as SharePoint Online,
   the following methods are available:

   - `ClientContext.with_credentials(client_credentials)`
   - `ClientContext.with_client_credentials(client_id, client_secret)`

   Usage:
   ```
   client_credentials = ClientCredential('{client_id}','{client_secret}')
   ctx = ClientContext('{url}').with_credentials(client_credentials)
   ```

   Documentation:
   - [Granting access using SharePoint App-Only](https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs)
   - [wiki](https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-and-and-SharePoint-2013-2016-2019-on-premises--with-app-principal)

   Example: [connect_with_app_principal.py](examples/sharepoint/auth_app_only.py)

#### 2. Using username and password

   Usage:
   ```
   user_credentials = UserCredential('{username}','{password}')
   ctx = ClientContext('{url}').with_credentials(user_credentials)
   ```

   Example: [connect_with_user_credential.py](examples/sharepoint/auth_user_credential.py)

#### 3. Using an Azure AD application (certificate credentials flow)

  Documentation:
   - [Granting access via Azure AD App-Only](https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread)
   - [wiki](https://github.com/vgrem/Office365-REST-Python-Client/wiki/How-to-connect-to-SharePoint-Online-with-certificate-credentials)

  Example: [connect_with_client_certificate.py](examples/sharepoint/auth_client_certificate.py)

#### 4. Interactive

   to login interactively i.e. via a local browser

   Prerequisite:

   > In Azure Portal, configure the Redirect URI of your
   "Mobile and Desktop application" as ``http://localhost``.

  Example: [connect_interactive.py](examples/sharepoint/auth_interactive.py)

  Usage:
```python
from office365.sharepoint.client_context import ClientContext
ctx = ClientContext(site_url).with_interactive(tenant_name_or_id, client_id)
me = ctx.web.current_user.get().execute_query()
print(me.login_name)
```

### Examples

There are **two approaches** available to perform API queries:

1. `ClientContext class` - where you target SharePoint resources such as `Web`, `ListItem` and etc (recommended)

```python
from office365.runtime.auth.user_credential import UserCredential
from office365.sharepoint.client_context import ClientContext
site_url = "https://{your-tenant-prefix}.sharepoint.com"
ctx = ClientContext(site_url).with_credentials(UserCredential("{username}", "{password}"))
web = ctx.web
ctx.load(web)
ctx.execute_query()
print("Web title: {0}".format(web.properties['Title']))
```
or alternatively via method chaining (a.k.a Fluent Interface):

```python
from office365.runtime.auth.user_credential import UserCredential
from office365.sharepoint.client_context import ClientContext
site_url = "https://{your-tenant-prefix}.sharepoint.com"
ctx = ClientContext(site_url).with_credentials(UserCredential("{username}", "{password}"))
web = ctx.web.get().execute_query()
print("Web title: {0}".format(web.properties['Title']))
```

2. `RequestOptions class` - where you construct REST queries (and no model is involved)

   The example demonstrates how to read `Web` properties:

```python
import json
from office365.runtime.auth.user_credential import UserCredential
from office365.runtime.http.request_options import RequestOptions
from office365.sharepoint.client_context import ClientContext
site_url = "https://{your-tenant-prefix}.sharepoint.com"
ctx = ClientContext(site_url).with_credentials(UserCredential("{username}", "{password}"))
request = RequestOptions("{0}/_api/web/".format(site_url))
response = ctx.pending_request().execute_request_direct(request)
json = json.loads(response.content)
web_title = json['d']['Title']
print("Web title: {0}".format(web_title))
```

The list of examples:

- Working with files
  - [download a file](examples/sharepoint/files/download.py)
  - [upload a file](examples/sharepoint/files/upload.py)

- Working with lists and list items
  -  [create a list item](examples/sharepoint/lists/data_generator.py)
  -  [read a list item](examples/sharepoint/lists/read_paged.py)
  -  [update a list item](examples/sharepoint/listitems/update_batch.py)
  -  [delete a list item](examples/sharepoint/listitems/delete.py)

Refer [examples section](examples/sharepoint) for another scenarios

### Support for non-standard SharePoint Online Environments

  Support for non-standard SharePoint Environments is currently being implemented. Currently supported:
  - GCC High

  To enable authentication to GCC High endpoints, add the `environment='GCCH'` parameter when calling the
  `ClientContext class` with `.with_user_credentials`, `.with_client_credentials`, or `.with_credentials`

   Example:
   ```python
   client_credentials = ClientCredential('{client_id}','{client_secret}')
   ctx = ClientContext('{url}').with_credentials(client_credentials, environment='GCCH')
   ```

# Working with Outlook API

The list of supported APIs:
-   [Outlook Contacts REST API](https://msdn.microsoft.com/en-us/office/office365/api/contacts-rest-operations)
-   [Outlook Calendar REST API](https://msdn.microsoft.com/en-us/office/office365/api/calendar-rest-operations)
-   [Outlook Mail REST API](https://msdn.microsoft.com/en-us/office/office365/api/mail-rest-operations)

Since Outlook REST APIs are available in both Microsoft Graph and the Outlook API endpoint,
the following clients are available:

- `GraphClient` which targets Outlook API `v2.0` version (*preferable* nowadays, refer [transition to Microsoft Graph-based Outlook REST API](https://docs.microsoft.com/en-us/outlook/rest/compare-graph-outlook) for a details)
~~- `OutlookClient` which targets Outlook API `v1.0` version (not recommended for usage since `v1.0` version is being deprecated.)~~

### Authentication

[The Microsoft Authentication Library (MSAL) for Python](https://pypi.org/project/msal/) which comes as a dependency
is used as a default library to obtain tokens to call Microsoft Graph API.

Using [Microsoft Authentication Library (MSAL) for Python](https://pypi.org/project/msal/)

> Note: access token is getting acquired via [Client Credential flow](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow)
> in the provided examples. Other forms of token acquisition can be found here: https://msal-python.readthedocs.io/en/latest/

```python
import msal
from office365.graph_client import GraphClient

def acquire_token():
    """
    Acquire token via MSAL
    """
    authority_url = 'https://login.microsoftonline.com/{tenant_id_or_name}'
    app = msal.ConfidentialClientApplication(
        authority=authority_url,
        client_id='{client_id}',
        client_credential='{client_secret}'
    )
    token = app.acquire_token_for_client(scopes=["https://graph.microsoft.com/.default"])
    return token

client = GraphClient(acquire_token)

```

But in terms of Microsoft Graph API authentication, another OAuth spec compliant libraries
such as [adal](https://github.com/AzureAD/azure-activedirectory-library-for-python)
are supported as well.

Using [ADAL Python](https://adal-python.readthedocs.io/en/latest/#)

Usage

```python
import adal
from office365.graph_client import GraphClient

def acquire_token_func():
    authority_url = 'https://login.microsoftonline.com/{tenant_id_or_name}'
    auth_ctx = adal.AuthenticationContext(authority_url)
    token = auth_ctx.acquire_token_with_client_credentials(
        "https://graph.microsoft.com",
        "{client_id}",
        "{client_secret}")
    return token

client = GraphClient(acquire_token_func)

```

#### Example

The example demonstrates how to send an email via [Microsoft Graph endpoint](https://docs.microsoft.com/en-us/graph/api/user-sendmail?view=graph-rest-1.0&tabs=http).

> Note: access token is getting acquired  via [Client Credential flow](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow)

```python
from office365.graph_client import GraphClient

client = GraphClient(acquire_token_func)

client.me.send_mail(
    subject="Meet for lunch?",
    body="The new cafeteria is open.",
    to_recipients=["fannyd@contoso.onmicrosoft.com"]
).execute_query()

```

Additional examples & scenarios:

-  [download a message](examples/outlook/messages/download.py)
-  [list messages](examples/outlook/messages/list_all.py)
-  [move messages to a different folder](examples/outlook/messages/move.py)
-  [search messages](examples/outlook/messages/search.py)
-  [send messages](examples/outlook/messages/send.py)
-  [send messages with attachments](examples/outlook/messages/send_with_attachment.py)
-  [enable sending emails on behalf of another user in your organization](https://learn.microsoft.com/en-us/microsoft-365/solutions/allow-members-to-send-as-or-send-on-behalf-of-group)

Refer to [examples section](examples/outlook) for other scenarios

# Working with OneDrive and SharePoint v2 APIs

#### Documentation

[OneDrive Graph API reference](https://docs.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0)

#### Authentication

[The Microsoft Authentication Library (MSAL) for Python](https://pypi.org/project/msal/) which comes as a dependency
is used to obtain token

```python
import msal

def acquire_token_func():
    """
    Acquire token via MSAL
    """
    authority_url = 'https://login.microsoftonline.com/{tenant_id_or_name}'
    app = msal.ConfidentialClientApplication(
        authority=authority_url,
        client_id='{client_id}',
        client_credential='{client_secret}'
    )
    token = app.acquire_token_for_client(scopes=["https://graph.microsoft.com/.default"])
    return token
```

#### Examples

##### Example: list available drives

The example demonstrates how to enumerate and print drive's url
which corresponds to [`list available drives` endpoint](https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/drive_list?view=odsp-graph-online)

> Note: access token is getting acquired  via [Client Credential flow](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow)

```python
from office365.graph_client import GraphClient

tenant_name = "contoso.onmicrosoft.com"
client = GraphClient(acquire_token_func)
drives = client.drives.get().execute_query()
for drive in drives:
    print("Drive url: {0}".format(drive.web_url))
```

##### Example: download the contents of a DriveItem(folder facet)

```python
from office365.graph_client import GraphClient
client = GraphClient(acquire_token_func)
# retrieve drive properties
drive = client.users["{user_id_or_principal_name}"].drive.get().execute_query()
# download files from OneDrive into local folder
with tempfile.TemporaryDirectory() as path:
    download_files(drive.root, path)
```

where

```python
def download_files(remote_folder, local_path):
    drive_items = remote_folder.children.get().execute_query()
    for drive_item in drive_items:
        if drive_item.file is not None:  # is file?
            # download file content
            with open(os.path.join(local_path, drive_item.name), 'wb') as local_file:
                drive_item.download(local_file).execute_query()
```

Additional examples:

-  [create list column](examples/onedrive/columns/create_text.py)
-  [download file](examples/onedrive/files/download.py)
-  [export files](examples/onedrive/files/export.py)
-  [upload folder](examples/onedrive/folders/upload.py)
-  [list drives](examples/onedrive/drives/list.py)
-  [list files](examples/onedrive/folders/list_files.py)

Refer to [OneDrive examples section](examples/onedrive) for more examples.

# Working with Microsoft Teams API

#### Authentication

[The Microsoft Authentication Library (MSAL) for Python](https://pypi.org/project/msal/) which comes as a dependency
is used to obtain token

#### Examples

##### Example: create a new team under a group

The example demonstrates how create a new team under a group
which corresponds to [`Create team` endpoint](https://docs.microsoft.com/en-us/graph/api/team-put-teams?view=graph-rest-1.0&tabs=http)

```python
from office365.graph_client import GraphClient
client = GraphClient(acquire_token_func)
new_team = client.groups["{group_id}"].add_team().execute_query_retry()
```

Additional examples:

-  [create a team](examples/teams/create_team.py)
-  [create team from group](examples/teams/create_from_group.py)
-  [list all teams](examples/teams/list_all.py)
-  [list my teams](examples/teams/list_my_teams.py)
-  [send messages](examples/teams/send_message.py)

Refer to [examples section](examples/teams) for other scenarios

# Working with Microsoft Onenote API

The library supports OneNote API in terms of calls to a user's OneNote notebooks, sections, and pages in a personal or organization account

Example: Create a new page

```python
from office365.graph_client import GraphClient
client = GraphClient(acquire_token_func)

files = {}
with open("./MyPage.html", 'rb') as f, \
    open("./MyImage.png", 'rb') as img_f, \
    open("./MyDoc.pdf", 'rb') as pdf_f:
    files["imageBlock1"] = img_f
    files["fileBlock1"] = pdf_f
    page = client.me.onenote.pages.add(presentation_file=f, attachment_files=files).execute_query()
```

# Working with Microsoft Planner API

The example demonstrates how to create a new planner task
which corresponds to [`Create plannerTask` endpoint](https://docs.microsoft.com/en-us/graph/api/planner-post-tasks?view=graph-rest-1.0):

```python
from office365.graph_client import GraphClient

client = GraphClient(acquire_token_func)
task = client.planner.tasks.add(title="New task", planId="--plan id goes here--").execute_query()
```

# Third Party Libraries and Dependencies
The following libraries will be installed when you install the client library:
* [requests](https://github.com/kennethreitz/requests)
* [Microsoft Authentication Library (MSAL) for Python](https://pypi.org/project/msal/)


===============================
 Gnocchi - Metric as a Service
===============================

.. image:: https://badge.fury.io/py/gnocchi.svg
    :target: https://badge.fury.io/py/gnocchi

.. image:: doc/source/_static/gnocchi-logo.png

Gnocchi is an open-source time series database.

The problem that Gnocchi solves is the storage and indexing of time series
data and resources at a large scale. This is useful in modern cloud platforms
which are not only huge but also are dynamic and potentially multi-tenant.
Gnocchi takes all of that into account.

Gnocchi has been designed to handle large amounts of aggregates being stored
while being performant, scalable and fault-tolerant. While doing this, the goal
was to be sure to not build any hard dependency on any complex storage system.

Gnocchi takes a unique approach to time series storage: rather than storing
raw data points, it aggregates them before storing them. This built-in feature
is different from most other time series databases, which usually support
this mechanism as an option and compute aggregation (average, minimum, etc.) at
query time.

Because Gnocchi computes all the aggregations at ingestion, getting the data
back is extremely fast, as it just needs to read back the pre-computed results.

You can read the full documentation online at http://gnocchi.osci.io.


# pytorch-pfn-extras

[![PyPI](https://img.shields.io/pypi/v/pytorch-pfn-extras)](https://pypi.python.org/pypi/pytorch-pfn-extras)
[![Docs](https://img.shields.io/readthedocs/pytorch-pfn-extras)](https://pytorch-pfn-extras.readthedocs.io/)
[![License](https://img.shields.io/github/license/pfnet/pytorch-pfn-extras)](https://github.com/pfnet/pytorch-pfn-extras/blob/master/LICENSE)

Supplementary components to accelerate research and development in PyTorch.

## Installation

```sh
pip install pytorch-pfn-extras

# Use `[onnx]` to use onnx submodule like:
#  pip install "pytorch-pfn-extras[onnx]"

### Optinal dependencies
# For PlotReport / VariableStatisticsPlot extensions
pip install matplotlib

# For IgniteExtensionsManager
pip install pytorch-ignite torchvision

# For CuPy interoperability (see: https://docs.cupy.dev/en/stable/install.html)
pip install cupy  # or cupy-cudaXXX
```

## Requirements

* Python 3.8+
* PyTorch 1.10+

Optional dependencies:

* CuPy 8.0+ for PyTorch/CuPy interoperatbility

## Documentation

Refer to [Read The Docs](https://pytorch-pfn-extras.readthedocs.io/) for the complete documentation.

Below are some quick-links to the most important features of the library.

* [Extensions Manager](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/extensions.html)
* [Reporting](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/reporting.html)
* [Lazy Modules](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/lazy.html)
* [Distributed Snapshot](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/snapshot.html)
* [Config System](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/config.html)
* [ONNX Utils](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/onnx.html)
* [CUDA Utils (CuPy Interoperability)](https://pytorch-pfn-extras.readthedocs.io/en/latest/user_guide/cuda.html)

## Examples

* [Custom training loop](example/mnist.py)
* [Ignite integration](example/ignite-mnist.py)

## Contribution Guide

You can contribute to this project by sending a pull request.
After approval, the pull request will be merged by the reviewer.

Before making a contribution, please confirm that:

- Code quality stays consistent across the script, module or package.
- Code is covered by unit tests.
- API is maintainable.

## License

MIT License


# pajbot ![Python 4HEad](https://github.com/pajbot/pajbot/workflows/Python%204HEad/badge.svg)

pajbot is a twitch chat bot created by [pajlada](http://twitch.tv/pajlada).
[Website](https://pajbot.com)

Note: pajbot is in **maintenance mode**.
This means we focus on keeping the project alive by not allowing major overhauls of any pajbot system or any major features.
Fixing bugs, updating dependencies and ensuring that code interacting with external APIs still function will be our main goal.
Feature requests will not be accepted unless someone is willing to own the feature, and even then some features that change too much of the architecture won't be allowed.
Current minimal supported Python version is **3.9**.

## Python versioning

We use [pyenv](https://github.com/pyenv/pyenv) to manage Python versions. Get familiar with this tool.
Quick install of pyenv on Linux systems: `curl https://pyenv.run | bash`

If you don't want to use pyenv's version of Python in any of our scripts, set the `SKIP_PYENV` environment variable to `1`.

## Quick install

1. Install library requirements by typing `./scripts/venvinstall.sh` in the root folder
2. Copy `./configs/example.ini` to `./config.ini` and change the relevant lines in the file.
3. Run the bot! `./main.py`

## Detailed install

You can find a detailed installation guide for **pajbot** in the [`install-docs` directory](./install-docs) of this repository.

## Run-time options

Some values can be set to apply to your bot without modifying the config file, these are mostly for out-of-bot things.
They are configured using environment variables. The following options are available:

- `PB1_LOG_HIDE_TIMESTAMPS`
   If this option is set to `1`, all log entries will be printed without a timestamp prefix.


An open source platform for developing always-on, voice-controlled applications.

	Based on the previous work of Jasper.

  Key Features •
  Documentation •
  Download •
  Contribution •
  Sponsor Naomi •
  Credits •
  License

## Key Features

* Control Anything
  * Use your voice to ask for information, update social networks, control your home, and more.
* Always Listening
  * Naomi is always on, always listening for commands, and you can speak from feet away.
* 100% Open Source
  * Build it yourself with off-the-shelf hardware, and use our documentation to write your own modules.
* Cross Platform Coming Soon
  * Raspberry Pi and Linux currently.
* Thorough Documentation
  * Documentation is kept up to date.

## Documentation

All the documentation is available on our [website](https://projectnaomi.com/dev/docs)!

## Download

You can [download](https://projectnaomi.com/download) the latest version of Naomi from our website.

## Contribution

Please make sure to read the [Contributing Guide](https://github.com/NaomiProject/Naomi/blob/naomi-dev/CONTRIBUTING.md) before making a pull request.

Thank you to all the people who already contributed to Naomi!

## Sponsor Naomi

Naomi is an MIT-licensed open source project with its ongoing development made possible entirely by the support of these awesome [backers](https://github.com/naomiproject/naomi/blob/naomi-dev/BACKERS.md). If you'd like to join them, please consider:

- [Become a backer or sponsor on Open Collective](https://opencollective.com/projectnaomi).
- [One-time donation via PayPal or crypto-currencies.](https://projectnaomi.com/support-naomi/#One-time-Donations)

Sponsors via Open Collective

Platinum

Gold

## Credits

This software uses code from the initial Jasper Project & the Jasper2fork.

* [JasperProject](http://jasperproject.github.io/)
* [Jasper2Fork](https://github.com/andweber/j2f)

## License

MIT

---

> [projectnaomi.com](https://projectnaomi.com)  ·
> GitHub [@naomiproject](https://github.com/naomiproject)


blinkpy |Build Status| |Coverage Status| |PyPi Version| |Codestyle|
=============================================================================================
A Python library for the Blink Camera system (Python 3.9+)

Like the library? Consider buying me a cup of coffee!

`Buy me a Coffee! <https://buymeacoffee.com/kevinfronczak>`__

**BREAKING CHANGE WARNING:**
As of ``0.22.0`` the library uses asyncio which will break any user scripts used prior to this version. Please see the updated examples below and the ``blinkapp.py`` or ``blinksync.py`` examples in the ``blinkapp/`` directory for examples on how to migrate.

**Disclaimer:**
Published under the MIT license - See LICENSE file for more details.

"Blink Wire-Free HS Home Monitoring & Alert Systems" is a trademark owned by Immedia Inc., see www.blinkforhome.com for more information.
I am in no way affiliated with Blink, nor Immedia Inc.

Original protocol hacking by MattTW : https://github.com/MattTW/BlinkMonitorProtocol

API calls faster than 60 seconds is not recommended as it can overwhelm Blink's servers.  Please use this module responsibly.

Installation
-------------
``pip install blinkpy``

Installing Development Version
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the current development version, perform the following steps.  Note that the following will create a blinkpy directory in your home area:

.. code:: bash

    $ cd ~
    $ git clone https://github.com/fronzbot/blinkpy.git
    $ cd blinkpy
    $ pip install .

If you'd like to contribute to this library, please read the `contributing instructions <https://github.com/fronzbot/blinkpy/blob/dev/CONTRIBUTING.rst>`__.

Purpose
-------
This library was built with the intention of allowing easy communication with Blink camera systems, specifically to support the `Blink component <https://home-assistant.io/components/blink>`__ in `homeassistant <https://home-assistant.io/>`__.

Quick Start
=============
The simplest way to use this package from a terminal is to call ``await Blink.start()`` which will prompt for your Blink username and password and then log you in.  In addition, http requests are throttled internally via use of the ``Blink.refresh_rate`` variable, which can be set at initialization and defaults to 30 seconds.

.. code:: python

    import asyncio
    from aiohttp import ClientSession
    from blinkpy.blinkpy import Blink

    async def start():
        blink = Blink(session=ClientSession())
        await blink.start()
        return blink

    blink = asyncio.run(start())

This flow will prompt you for your username and password.  Once entered, if you likely will need to send a 2FA key to the blink servers (this pin is sent to your email address).  When you receive this pin, enter at the prompt and the Blink library will proceed with setup.

Starting blink without a prompt
-------------------------------
In some cases, having an interactive command-line session is not desired.  In this case, you will need to set the ``Blink.auth.no_prompt`` value to ``True``.  In addition, since you will not be prompted with a username and password, you must supply the login data to the blink authentication handler.  This is best done by instantiating your own auth handler with a dictionary containing at least your username and password.

.. code:: python

    import asyncio
    from aiohttp import ClientSession
    from blinkpy.blinkpy import Blink
    from blinkpy.auth import Auth

    async def start():
        blink = Blink(session=ClientSession())
        # Can set no_prompt when initializing auth handler
        auth = Auth({"username": <your username>, "password": <your password>}, no_prompt=True)
        blink.auth = auth
        await blink.start()
        return blink

    blink = asyncio.run(start())

Since you will not be prompted for any 2FA pin, you must call the ``blink.auth.send_auth_key`` function.  There are two required parameters: the ``blink`` object as well as the ``key`` you received from Blink for 2FA:

.. code:: python

    await auth.send_auth_key(blink, <your key>)
    await blink.setup_post_verify()

Supplying credentials from file
--------------------------------
Other use cases may involved loading credentials from a file.  This file must be ``json`` formatted and contain a minimum of ``username`` and ``password``.  A built in function in the ``blinkpy.helpers.util`` module can aid in loading this file.  Note, if ``no_prompt`` is desired, a similar flow can be followed as above.

.. code:: python

    import asyncio
    from aiohttp import ClientSession
    from blinkpy.blinkpy import Blink
    from blinkpy.auth import Auth
    from blinkpy.helpers.util import json_load

    async def start():
        blink = Blink()
        auth = Auth(await json_load("<File Location>"))
        blink.auth = auth
        await blink.start()
        return blink

    blink = asyncio.run(start())

Saving credentials
-------------------
This library also allows you to save your credentials to use in future sessions.  Saved information includes authentication tokens as well as unique ids which should allow for a more streamlined experience and limits the frequency of login requests.  This data can be saved as follows (it can then be loaded by following the instructions above for supplying credentials from a file):

.. code:: python

    await blink.save("<File location>")

Getting cameras
----------------
Cameras are instantiated as individual ``BlinkCamera`` classes within a ``BlinkSyncModule`` instance.  All of your sync modules are stored within the ``Blink.sync`` dictionary and can be accessed using the name of the sync module as the key (this is the name of your sync module in the Blink App).

The below code will display cameras and their available attributes:

.. code:: python

    for name, camera in blink.cameras.items():
      print(name)                   # Name of the camera
      print(camera.attributes)      # Print available attributes of camera

The most recent images and videos can be accessed as a bytes-object via internal variables.  These can be updated with calls to ``Blink.refresh()`` but will only make a request if motion has been detected or other changes have been found.  This can be overridden with the ``force`` flag, but this should be used for debugging only since it overrides the internal request throttling.

.. code:: python

    camera = blink.cameras['SOME CAMERA NAME']
    await blink.refresh(force=True)  # force a cache update USE WITH CAUTION
    camera.image_from_cache  # bytes-like image object (jpg)
    camera.video_from_cache  # bytes-like video object (mp4)

The ``blinkpy`` api also allows for saving images and videos to a file and snapping a new picture from the camera remotely:

.. code:: python

    camera = blink.cameras['SOME CAMERA NAME']
    await camera.snap_picture()       # Take a new picture with the camera
    await blink.refresh()             # Get new information from server
    await camera.image_to_file('/local/path/for/image.jpg')
    await camera.video_to_file('/local/path/for/video.mp4')

Arming Blink
-------------
Methods exist to arm/disarm the sync module, as well as enable/disable motion detection for individual cameras.  This is done as follows:

.. code:: python

    # Arm a sync module
    await blink.sync["SYNC MODULE NAME"].async_arm(True)

    # Disarm a sync module
    await blink.sync["SYNC MODULE NAME"].async_arm(False)

    # Print arm status of a sync module - a system refresh should be performed first
    await blink.refresh()
    sync = blink.sync["SYNC MODULE NAME"]
    print(f"{sync.name} status: {sync.arm}")

Similar methods exist for individual cameras:

.. code:: python

   camera = blink.cameras["SOME CAMERA NAME"]

   # Enable motion detection on a camera
   await camera.async_arm(True)

   # Disable motion detection on a camera
   await camera.async_arm( False)

   # Print arm status of a sync module - a system refresh should be performed first
   await blink.refresh()
   print(f"{camera.name} status: {camera.arm}")

Download videos
----------------
You can also use this library to download all videos from the server.  In order to do this, you must specify a ``path``.  You may also specifiy a how far back in time to go to retrieve videos via the ``since=`` variable (a simple string such as ``"2017/09/21"`` is sufficient), as well as how many pages to traverse via the ``stop=`` variable.  Note that by default, the library will search the first ten pages which is sufficient in most use cases.  Additionally, you can specify one or more cameras via the ``camera=`` property.  This can be a single string indicating the name of the camera, or a list of camera names.  By default, it is set to the string ``'all'`` to grab videos from all cameras. If you are downloading many items, setting the ``delay`` parameter is advised in order to throttle sequential calls to the API. By default this is set to ``1`` but can be any integer representing the number of seconds to delay between calls.

Example usage, which downloads all videos recorded since July 4th, 2018 at 9:34am to the ``/home/blink`` directory with a 2s delay between calls:

.. code:: python

    await blink.download_videos('/home/blink', since='2018/07/04 09:34', delay=2)

Sync Module Local Storage
=========================

Description of how I think the local storage API is used by Blink
-----------------------------------------------------------------

Since local storage is within a customer's residence, there are no guarantees for latency
and availability.  As a result, the API seems to be built to deal with these conditions.

In general, the approach appears to be this:  The Blink app has to query the sync
module for all information regarding the stored clips.  On a click to view a clip, the app asks
for the full list of stored clips, finds the clip in question, uploads the clip to the
cloud, and then downloads the clip back from a cloud URL. Each interaction requires polling for
the response since networking conditions are uncertain.  The app also caches recent clips and the manifest.

API steps
---------
1. Request the local storage manifest be created by the sync module.

   * POST **{base_url}/api/v1/accounts/{account_id}/networks/{network_id}/sync_modules/{sync_id}/local_storage/manifest/request**
   * Returns an ID that is used to get the manifest.

2. Retrieve the local storage manifest.

   * GET **{base_url}/api/v1/accounts/{account_id}/networks/{network_id}/sync_modules/{sync_id}/local_storage/manifest/request/{manifest_request_id}**
   * Returns full manifest.
   * Extract the manifest ID from the response.

3. Find a clip ID in the clips list from the manifest to retrieve, and request an upload.

   * POST **{base_url}/api/v1/accounts/{account_id}/networks/{network_id}/sync_modules/{sync_id}/local_storage/manifest/{manifest_id}/clip/request/{clip_id}**
   * When the response is returned, the upload has finished.

4. Download the clip using the same clip ID.

   * GET **{base_url}/api/v1/accounts/{account_id}/networks/{network_id}/sync_modules/{sync_id}/local_storage/manifest/{manifest_id}/clip/request/{clip_id}**

.. |Build Status| image:: https://github.com/fronzbot/blinkpy/workflows/build/badge.svg
   :target: https://github.com/fronzbot/blinkpy/actions?query=workflow%3Abuild
.. |Coverage Status| image:: https://codecov.io/gh/fronzbot/blinkpy/branch/dev/graph/badge.svg
    :target: https://codecov.io/gh/fronzbot/blinkpy
.. |PyPi Version| image:: https://img.shields.io/pypi/v/blinkpy.svg
    :target: https://pypi.python.org/pypi/blinkpy
.. |Codestyle| image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black


# spotpy
A Statistical Parameter Optimization Tool for Python

---

[![PyPI Version][pypi-v-image]][pypi-v-link]
[![Python Versions][pypi-pyv-image]][pypi-pyv-link]
[![Build Status][travis-image]][travis-link]
[![License][license-image]][license-link]
[![Coverage Status](https://coveralls.io/repos/github/thouska/spotpy/badge.svg?branch=master)](https://coveralls.io/github/thouska/spotpy?branch=master)
[![Documentation Status](https://readthedocs.org/projects/spotpy/badge/?version=latest)](https://readthedocs.org/projects/spotpy/badge/?version=latest)
[![DOI](https://zenodo.org/badge/47562322.svg)](https://zenodo.org/badge/latestdoi/47562322)

[pypi-v-image]: https://img.shields.io/pypi/v/spotpy.png
[pypi-v-link]: https://pypi.python.org/pypi/spotpy
[pypi-pyv-image]: https://img.shields.io/pypi/pyversions/spotpy.png
[pypi-pyv-link]: https://img.shields.io/pypi/pyversions/spotpy
[travis-image]: https://img.shields.io/travis/thouska/spotpy/master.png
[travis-link]: https://travis-ci.org/thouska/spotpy
[license-image]: https://img.shields.io/badge/license-MIT-blue.png
[license-link]: http://opensource.org/licenses/MIT

Purpose
=================

SPOTPY is a Python framework that enables the use of Computational optimization techniques for calibration, uncertainty
and sensitivity analysis techniques of almost every (environmental-) model. The package is puplished in the open source journal PLoS One:

Houska, T., Kraft, P., Chamorro-Chavez, A. and Breuer, L.: SPOTting Model Parameters Using a Ready-Made Python Package, PLoS ONE,
10(12), e0145180, doi:[10.1371/journal.pone.0145180](http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0145180 "SPOTting Model Parameters Using a Ready-Made Python Package"), 2015

The simplicity and flexibility enables the use and test of different
algorithms of almost any model, without the need of complex codes::

	sampler = spotpy.algorithms.sceua(model_setup())     # Initialize your model with a setup file
	sampler.sample(10000)                                # Run the model
	results = sampler.getdata()                          # Load the results
	spotpy.analyser.plot_parametertrace(results)         # Show the results

Features
=================

Complex algorithms bring complex tasks to link them with a model.
We want to make this task as easy as possible.
Some features you can use with the SPOTPY package are:

* Fitting models to evaluation data with different algorithms.
  Available algorithms are:

  * Monte Carlo (`MC`)
  * Markov-Chain Monte-Carlo (`MCMC`)
  * Maximum Likelihood Estimation (`MLE`)
  * Latin-Hypercube Sampling (`LHS`)
  * Simulated Annealing (`SA`)
  * Shuffled Complex Evolution Algorithm (`SCE-UA`)
  * Differential Evolution Markov Chain Algorithm (`DE-MCz`)
  * Differential Evolution Adaptive Metropolis Algorithm (`DREAM`)
  * RObust Parameter Estimation (`ROPE`)
  * Fourier Amplitude Sensitivity Test (`FAST`)
  * Artificial Bee Colony (`ABC`)
  * Fitness Scaled Chaotic Artificial Bee Colony (`FSCABC`)
  * Dynamically Dimensioned Search algorithm (`DDS`)
  * Pareto Archived - Dynamicallly Dimensioned Search algorithm (`PA-DDS`)
  * Fast and Elitist Multiobjective Genetic Algorithm (`NSGA-II`)

* Wide range of objective functions (also known as loss function, fitness function or energy function) to validate the sampled results. Available functions are

  * Bias
  * PBias
  * Nash-Sutcliffe (`NSE`)
  * logarithmic Nash-Sutcliffe (`logNSE`)
  * logarithmic probability (`logp`)
  * Correlation Coefficient (`r`)
  * Coefficient of Determination (`r^2`)
  * Mean Squared Error (`MSE`)
  * Root Mean Squared Error (`RMSE`)
  * Mean Absolute Error (`MAE`)
  * Relative Root Mean Squared Error (`RRMSE`)
  * Agreement Index (`AI`)
  * Covariance, Decomposed MSE (`dMSE`)
  * Kling-Gupta Efficiency (`KGE`)
  * Non parametric Kling-Gupta Efficiency (`KGE_non_parametric`)

* Wide range of hydrological signatures functions to validate the sampled results:

  * Slope
  * Flooding/Drought events
  * Flood/Drought frequency
  * Flood/Drought duration
  * Flood/Drought variance
  * Mean flow
  * Median flow
  * Skewness
  * compare percentiles of discharge

* Prebuild parameter distribution functions:

  * Uniform
  * Normal
  * logNormal
  * Chisquare
  * Exponential
  * Gamma
  * Wald
  * Weilbull

* Wide range to adapt algorithms to perform uncertainty-, sensitivity analysis or calibration
  of a model.

* Multi-objective support

* MPI support for fast parallel computing

* A progress bar monitoring the sampling loops. Enables you to plan your coffee brakes.

* Use of NumPy functions as often as possible. This makes your coffee brakes short.

* Different databases solutions: `ram` storage for fast sampling a simple , `csv` tables
  the save solution for long duration samplings.

* Automatic best run selecting and plotting

* Parameter trace plotting

* Parameter interaction plot including the Gaussian-kde function

* Regression analysis between simulation and evaluation data

* Posterior distribution plot

* Convergence diagnostics with Gelman-Rubin and the Geweke plot

Install
=================

Classical Python options exist to install SPOTPY:

From PyPi:

	pip install spotpy

From Conda-Forge:

	conda config --add channels conda-forge
	conda config --set channel_priority strict
	conda install spotpy

From [Source](https://pypi.python.org/pypi/spotpy):

	python setup.py install

Support
=================

* Documentation: https://spotpy.readthedocs.io/en/latest/

* Feel free to contact the authors of this tool for any support questions.

* Please contact the authors in case of any bug.

* If you use this package for a scientific research paper, please cite SPOTPY. It is [peer-reviewed](http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0145180 "SPOTting Model Parameters Using a Ready-Made Python Package").

* Patches/enhancements and any other contributions to this package are very welcome!

Getting started
=================
Have a look at https://github.com/thouska/spotpy/tree/master/spotpy/examples and https://spotpy.readthedocs.io/en/latest/getting_started/

Contributing
=================
Patches/enhancements/new algorithms and any other contributions to this package are very welcome!

1. Fork it ( http://github.com/thouska/spotpy/fork )
2. Create your feature branch (``git checkout -b my-new-feature``)
3. Add your modifications
4. Add short summary of your modifications on ``CHANGELOG.md``
5. Commit your changes (``git commit -m "Add some feature"``)
6. Push to the branch (``git push origin my-new-feature``)
7. Create new Pull Request

Papers citing SPOTPY
=====================
See [Google Scholar](https://scholar.google.de/scholar?cites=17155001516727704728&as_sdt=2005&sciodt=0,5&hl=de) for a continuously updated list.


ControllerX

[![hacs_badge](https://img.shields.io/badge/HACS-Default-orange.svg?style=for-the-badge)](https://github.com/hacs/integration)
[![github-check-status](https://img.shields.io/github/checks-status/xaviml/controllerx/main?style=for-the-badge)](https://github.com/xaviml/controllerx/actions?query=branch%3Amain)
[![last-release](https://img.shields.io/github/v/release/xaviml/controllerx.svg?style=for-the-badge)](https://github.com/xaviml/controllerx/releases)
[![downloads-latest](https://img.shields.io/github/downloads/xaviml/controllerx/latest/total?style=for-the-badge)](http://github.com/xaviml/controllerx/releases/latest)
[![code-coverage](https://img.shields.io/codecov/c/gh/xaviml/controllerx/main?style=for-the-badge&token=7PUZHL97IG)](https://app.codecov.io/gh/xaviml/controllerx/branch/main)
[![community-topic](https://img.shields.io/badge/community-topic-blue?style=for-the-badge)](https://community.home-assistant.io/t/controllerx-bring-full-functionality-to-light-and-media-player-controllers/148855)
[![buy-me-a-beer](https://img.shields.io/badge/sponsor-Buy%20me%20a%20beer-orange?style=for-the-badge)](https://www.buymeacoffee.com/xaviml)

_Create controller-based automations with ease to control your home devices and scenes._

## Quick example

With just this configuration, you can have the E1810 controller from IKEA (5 buttons one) connected to the bedroom light and be able to change the brightness and color temperature or color.

```yaml
livingroom_controller:
  module: controllerx
  class: E1810Controller
  controller: sensor.livingroom_controller_action
  integration: z2m
  light: light.livingroom
```

## Documentation

You can check the documentation in [here](https://xaviml.github.io/controllerx/).

If you have any question, you can either [open an issue](https://github.com/xaviml/controllerx/issues/new/choose) or comment in [this topic](https://community.home-assistant.io/t/controllerx-bring-full-functionality-to-light-and-media-player-controllers/148855) from the Home Assistant community forum.

If you like this project, don't forget to star it :)

## Contributing

If you want to contribute to this project, check [CONTRIBUTING.md](https://github.com/xaviml/controllerx/blob/main/CONTRIBUTING.md).


    Galactic Dynamics in python

[galpy](http://www.galpy.org) is a Python package for galactic dynamics. It supports orbit integration in a variety of potentials, evaluating and sampling various distribution functions, and the calculation of action-angle coordinates for all static potentials. `galpy` is an [astropy](http://www.astropy.org/) [affiliated package](http://www.astropy.org/affiliated/) and provides full support for astropy’s [Quantity](http://docs.astropy.org/en/stable/api/astropy.units.Quantity.html) framework for variables with units.

[![image](https://github.com/jobovy/galpy/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/jobovy/galpy/actions/workflows/build.yml) [![image](https://github.com/jobovy/galpy/actions/workflows/build_windows.yml/badge.svg?branch=main)](https://github.com/jobovy/galpy/actions/workflows/build_windows.yml) [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/jobovy/galpy/main.svg)](https://results.pre-commit.ci/latest/github/jobovy/galpy/main)
[![image](http://codecov.io/gh/jobovy/galpy/coverage.svg?branch=main)](http://app.codecov.io/gh/jobovy/galpy?branch=main) [![image](https://readthedocs.org/projects/galpy/badge/?version=latest)](http://docs.galpy.org/en/latest/)
[![image](http://img.shields.io/pypi/v/galpy.svg)](https://pypi.python.org/pypi/galpy/) [![image](https://img.shields.io/pypi/pyversions/galpy?logo=python&logoColor=white)](https://pypi.python.org/pypi/galpy/) [![image](https://anaconda.org/conda-forge/galpy/badges/version.svg)](https://anaconda.org/conda-forge/galpy) [![image](https://img.shields.io/github/commits-since/jobovy/galpy/latest)](https://github.com/jobovy/galpy/commits/main)
[![image](http://img.shields.io/badge/license-New%20BSD-brightgreen.svg)](https://github.com/jobovy/galpy/blob/main/LICENSE) [![image](http://img.shields.io/badge/DOI-10.1088/0067%2D%2D0049/216/2/29-blue.svg)](http://dx.doi.org/10.1088/0067-0049/216/2/29) [![image](http://img.shields.io/badge/powered%20by-AstroPy-orange.svg?style=flat)](http://www.astropy.org/) [![image](https://img.shields.io/badge/join-slack-E01563.svg?style=flat&logo=slack&logoWidth=10)](https://join.slack.com/t/galpy/shared_invite/zt-p6upr4si-mX7u8MRdtm~3bW7o8NA_Ww)

AUTHOR
======

Jo Bovy - bovy at astro dot utoronto dot ca

See
[AUTHORS.txt](https://github.com/jobovy/galpy/blob/main/AUTHORS.txt)
for a full list of contributors.

If you find this code useful in your research, please let me know. **If
you use galpy in a publication, please cite** [Bovy
(2015)](http://adsabs.harvard.edu/abs/2015ApJS..216...29B) **and link to
http://github.com/jobovy/galpy**. See [the acknowledgement documentation
section](http://docs.galpy.org/en/latest/index.html#acknowledging-galpy)
for a more detailed guide to citing parts of the code. Thanks!

LOOKING FOR HELP?
=================

The latest documentation can be found
[here](http://docs.galpy.org/en/latest/). You can also join the
[galpy slack community](https://galpy.slack.com/) for any questions
related to `galpy`; join
[here](https://join.slack.com/t/galpy/shared_invite/zt-p6upr4si-mX7u8MRdtm~3bW7o8NA_Ww).

If you find *any* bug in the code, please report these using the [Issue
Tracker](http://github.com/jobovy/galpy/issues) or by joining the [galpy
slack community](https://galpy.slack.com/).

If you are having issues with the installation of `galpy`, please first
consult the [Installation
FAQ](http://docs.galpy.org/en/latest/installation.html#installation-faq).

PYTHON VERSIONS AND DEPENDENCIES
================================

`galpy` supports Python 3. Specifically, galpy supports Python 3.8, 3.9, 3.10, 3.11,
and 3.12. GitHub Actions CI builds regularly check support for
Python 3.12 (and of 3.8, 3.9, 3.10, and 3.11 using a more limited, core set of tests)
on Linux and Windows (and 3.12 on Mac OS). Python 2.7 is no longer supported.

This package requires [Numpy](https://numpy.org/),
[Scipy](http://www.scipy.org/), and
[Matplotlib](http://matplotlib.sourceforge.net/). Certain advanced
features require the GNU Scientific Library
([GSL](http://www.gnu.org/software/gsl/)), with action calculations
requiring version 1.14 or higher. Other optional dependencies include:

* Support for providing inputs and getting outputs as Quantities with units is provided through
[`astropy`](http://www.astropy.org/).
* Querying SIMBAD for the coordinates of an object in the `Orbit.from_name` initialization method requires [`astroquery`](https://astroquery.readthedocs.io/en/latest/).
* Displaying a progress bar for certain operations (e.g., orbit integration of multiple objects at once) requires [`tqdm`](https://github.com/tqdm/tqdm).
* Plotting arbitrary functions of Orbit attributes requires [`numexpr`](https://github.com/pydata/numexpr).
* Speeding up the evaluation of certain functions in the C code requires [`numba`](https://numba.pydata.org/).
* Constant-anisotropy DFs in `galpy.df.constantbetadf` require [`JAX`](https://github.com/google/jax).
* Use of `SnapshotRZPotential` and `InterpSnapshotRZPotential` requires [`pynbody`](https://github.com/pynbody/pynbody).

Other parts of the code may require additional packages and you will be alerted by the code if they are
not installed.

CONTRIBUTING TO GALPY
=====================

If you are interested in contributing to galpy\'s development, take a
look at [this brief
guide](https://github.com/jobovy/galpy/wiki/Guide-for-new-contributors)
on the wiki. This will hopefully help you get started!

Some further development notes can be found on the
[wiki](http://github.com/jobovy/galpy/wiki/). This includes a list of
small and larger extensions of galpy that would be useful
[here](http://github.com/jobovy/galpy/wiki/Possible-galpy-extensions) as
well as a longer-term roadmap
[here](http://github.com/jobovy/galpy/wiki/Roadmap). Please let the main
developer know if you need any help contributing!


[![CI Build Status](https://github.com/googlefonts/gftools/workflows/Test/badge.svg?branch=main)](https://github.com/googlefonts/gftools/actions/workflows/ci.yml?query=workflow%3ATest+branch%3Amain)
[![PyPI](https://img.shields.io/pypi/v/gftools.svg)](https://pypi.org/project/gftools/)

# Google Fonts Tools

This project contains tools used for working with the Google Fonts collection, plus **Google Fonts Glyph Set Documentation** in the [/encodings](https://github.com/googlefonts/gftools/tree/main/Lib/gftools/encodings) subdirectory. While these tools are primarily intended for contributors to the Google Fonts project, anyone who works with fonts could find them useful.

The tools and files under this directory are available under the Apache License v2.0, for details see [LICENSE](LICENSE)

## Google Fonts Official Glyph Sets

The glyph sets useful for type designers that were previously hosted in this repository have been moved to:

<https://github.com/googlefonts/glyphsets/tree/main/GF_glyphsets>

## Tool Usage Examples

Compare fonts:

    gftools compare-font font1.ttf font2.ttf

Add a METADATA.pb to a family directory

    gftools add-font ../ofl/newfamily

Sanity check a family directory:

    gftools sanity-check --repair_script=/tmp/fix.py ../ofl/josefinsans
    gftools sanity-check --repair_script=/tmp/fix.py --fix_type=fsSelection ../ufl

Check a font family against the same family hosted on Google Fonts:

    gftools qa [fonts.ttf] -gfb -a -o qa

Check a variable font family against the same family as static fonts:

    gftools qa -f [vf_fonts] -fb [static_fonts] --diffenator --diffbrowsers -o ~/path/out

Fix a non hinted font

    gftools fix-nonhinting font_in.ttf font_out.ttf

Package and PR a family update to google/fonts. Much more detailed [documentation](./docs/gftools-packager).

    gftools packager "Family Sans" path/to/local/google/fonts -py

## Tool Installation

**Please note that gftools requires [Python 3.7](http://www.python.org/download/) or later.**

Please install these tools using pip:

    pip install gftools

If you would like to use `gftools qa`:

    pip install 'gftools[qa]'

### Tool Requirements and Dependencies

`gftools packager` needs the command line `git` program in a version >= Git 2.5 (Q2 2015) in order to perform a shallow clone (`--depth 1`) of the font upstream repository and branch. This is not supported by pygit2/libgit2 yet.

`gftools manage-traffic-jam` requires two private files which contain sensitive data. Ask m4rc1e for them.

### Google Fonts API Key

In order to use the scripts **gftools qa** and **gftools family-html-snippet**, you will need to generate a Google Fonts api key, https://developers.google.com/fonts/. You will then need to create a new text file located on your system at `~/.gf-api-key` (where ~ is your home directory), which contains the following:

```
[Credentials]
key = your-newly-generated-googlefonts-api-key

```

**Upstream project repos**

* https://github.com/google/protobuf
* https://github.com/behdad/fonttools


# Default repositories

This is where HACS gets its default repositories from to show in the "store".

If you want to publish your repositories as default in HACS have a look here:

- https://hacs.xyz/docs/publish/start
- https://hacs.xyz/docs/publish/include


[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Downloads](https://pepy.tech/badge/happytransformer)](https://pepy.tech/project/happytransformer)
[![Website shields.io](https://img.shields.io/website-up-down-green-red/http/shields.io.svg)](http://happytransformer.com)
![PyPI](https://img.shields.io/pypi/v/happytransformer)
[![](https://github.com/EricFillion/happy-transformer/workflows/build/badge.svg)](https://github.com/EricFillion/happy-transformer/actions)

# Happy Transformer
**Documentation and news: [happytransformer.com](http://happytransformer.com)**

Join our Discord server: [![Support Server](https://img.shields.io/discord/839263772312862740.svg?label=Discord&logo=Discord&colorB=7289da&style=?style=flat-square&logo=appveyor)](https://discord.gg/psVwe3wfTb)

![HappyTransformer](logo.png)

Happy Transformer makes it easy to fine-tune and perform inference with NLP Transformer models.

## 3.0.0
1. DeepSpeed for training
2. Apple's MPS for training and inference
3. WandB to track training runs
4. Data supplied for training is automatically split into portions for training and evaluating
5. Push models directly to Hugging Face's Model Hub

Read about the full 3.0.0 update including breaking changes [here](https://happytransformer.com/news/).

## Tasks

| Tasks                    | Inference | Training   |
|--------------------------|-----------|------------|
| Text Generation          | ✔         | ✔          |
| Text Classification      | ✔         | ✔          |
| Word Prediction          | ✔         | ✔          |
| Question Answering       | ✔         | ✔          |
| Text-to-Text             | ✔         | ✔          |
| Next Sentence Prediction | ✔         |            |
| Token Classification     | ✔         |            |

## Quick Start
```sh
pip install happytransformer
```

```python

from happytransformer import HappyWordPrediction
#--------------------------------------#
happy_wp = HappyWordPrediction()  # default uses distilbert-base-uncased
result = happy_wp.predict_mask("I think therefore I [MASK]")
print(result)  # [WordPredictionResult(token='am', score=0.10172799974679947)]
print(result[0].token)  # am
```

## Maintainers
- [Eric Fillion](https://github.com/ericfillion)  Lead Maintainer
- [Ted Brownlow](https://github.com/ted537) Maintainer

## Tutorials
[Text generation with training (GPT-Neo)](https://youtu.be/GzHJ3NUVtV4)

[Text classification (training)](https://www.vennify.ai/train-text-classification-transformers/)

[Text classification (hate speech detection)](https://youtu.be/jti2sPQYzeQ)

[Text classification (sentiment analysis)](https://youtu.be/Ew72EAgM7FM)

[Word prediction with training (DistilBERT, RoBERTa)](https://youtu.be/AWe0PHsPc_M)

[Top T5 Models ](https://www.vennify.ai/top-t5-transformer-models/)

[Grammar Correction](https://www.vennify.ai/grammar-correction-python/)

[Fine-tune a Grammar Correction Model](https://www.vennify.ai/fine-tune-grammar-correction/)


 Stone Soup

[![PyPI](https://img.shields.io/pypi/v/stonesoup?style=flat)](https://pypi.org/project/stonesoup)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/stonesoup.svg)](https://anaconda.org/conda-forge/stonesoup)
[![CircleCI branch](https://img.shields.io/circleci/project/github/dstl/Stone-Soup/main.svg?label=tests&style=flat)](https://circleci.com/gh/dstl/Stone-Soup)
[![Codecov](https://img.shields.io/codecov/c/github/dstl/Stone-Soup.svg)](https://codecov.io/gh/dstl/Stone-Soup)
[![Read the Docs](https://img.shields.io/readthedocs/stonesoup.svg?style=flat)](https://stonesoup.readthedocs.io/en/latest/?badge=latest)
[![Gitter](https://img.shields.io/gitter/room/dstl/Stone-Soup.svg?color=informational&style=flat)](https://gitter.im/dstl/Stone-Soup?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.4663993-informational)](https://doi.org/10.5281/zenodo.4663993)

## Background
Stone Soup is a software project to provide the target tracking and state estimation
community with a framework for the development and testing of tracking and state
estimation algorithms.

An article is [available](https://www.gov.uk/government/news/dstl-shares-new-open-source-framework-initiative) that details the background to the project, and contains links to sample data.

Please see the
[Stone Soup documentation](https://stonesoup.readthedocs.org/) for more
information.

Please see the [tutorials](https://stonesoup.readthedocs.io/en/latest/auto_tutorials/index.html),
[examples](https://stonesoup.readthedocs.io/en/latest/auto_examples/index.html),
and [demonstrations](https://stonesoup.readthedocs.io/en/latest/auto_demos/index.html),
which you can also try out on Binder: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/dstl/Stone-Soup/main?filepath=notebooks)

## Dependencies
Stone Soup uses the following dependencies:

| Name | License |
| ---- | ------- |
| [Python](https://www.python.org/) (v3.8+) | PSFL |
| [numpy](https://numpy.org/) | BSD |
| [SciPy](https://www.scipy.org/) | BSD |
| [matplotlib](https://matplotlib.org/) | [PSF/BSD-compatible](https://matplotlib.org/users/license.html) |
| [ruamel.yaml](https://yaml.readthedocs.io/) | MIT |
| [pymap3d](https://github.com/scivision/pymap3d) | MIT |
| [utm](https://github.com/Turbo87/utm) | MIT |
| [ordered-set](https://github.com/LuminosoInsight/ordered-set) | MIT |
| [setuptools](https://github.com/pypa/setuptools) | MIT |
| [rtree](https://github.com/Toblerity/rtree) | MIT |

### Development

#### Testing
These dependencies are required for running Stone Soup tests.

| Name | License |
| ---- | ------- |
| [pytest](https://docs.pytest.org/) | MIT |
| [Flake8](https://flake8.pycqa.org/) | MIT |
| [Coverage.py](https://coverage.readthedocs.io/) | Apache 2.0 |

#### Documentation
These dependencies are required for building Stone Soup documentation.

| Name | License |
| ---- | ------- |
| [Sphinx](https://www.sphinx-doc.org/) | BSD |
| [sphinx-gallery](https://sphinx-gallery.github.io/) | BSD |
| [pillow](https://pillow.readthedocs.io/en/stable/index.html) | [PIL Software License](https://pillow.readthedocs.io/en/stable/about.html#license) |
| [folium](https://python-visualization.github.io/folium/) | MIT |

## License
Stone Soup is released under MIT License. Please see [License](LICENSE) for details.


CloudBridge provides a consistent layer of abstraction over different
Infrastructure-as-a-Service cloud providers, reducing or eliminating the need
to write conditional code for each cloud.

Documentation
~~~~~~~~~~~~~
Detailed documentation can be found at http://cloudbridge.cloudve.org.

Build Status Tests
~~~~~~~~~~~~~~~~~~
.. image:: https://github.com/CloudVE/cloudbridge/actions/workflows/integration.yaml/badge.svg
   :target: https://github.com/CloudVE/cloudbridge/actions/
   :alt: Integration Tests

.. image:: https://codecov.io/gh/CloudVE/cloudbridge/graph/badge.svg?token=w0LAfAIVdd
   :target: https://codecov.io/gh/CloudVE/cloudbridge
   :alt: Code Coverage

.. image:: https://img.shields.io/pypi/v/cloudbridge.svg
   :target: https://pypi.python.org/pypi/cloudbridge/
   :alt: latest version available on PyPI

.. image:: https://readthedocs.org/projects/cloudbridge/badge/?version=latest
   :target: http://cloudbridge.readthedocs.org/en/latest/?badge=latest
   :alt: Documentation Status

.. image:: https://img.shields.io/pypi/dm/cloudbridge
   :target: https://pypistats.org/packages/cloudbridge
   :alt: Download stats

.. |aws-py38| image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/nuwang/d354f151eb8c9752da13e6dec012fb07/raw/cloudbridge_py3.8_aws.json
              :target: https://github.com/CloudVE/cloudbridge/actions/

.. |azure-py38| image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/nuwang/d354f151eb8c9752da13e6dec012fb07/raw/cloudbridge_py3.8_azure.json
                :target: https://github.com/CloudVE/cloudbridge/actions/

.. |gcp-py38| image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/nuwang/d354f151eb8c9752da13e6dec012fb07/raw/cloudbridge_py3.8_gcp.json
              :target: https://github.com/CloudVE/cloudbridge/actions/

.. |mock-py38| image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/nuwang/d354f151eb8c9752da13e6dec012fb07/raw/cloudbridge_py3.8_mock.json
              :target: https://github.com/CloudVE/cloudbridge/actions/

.. |os-py38| image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/nuwang/d354f151eb8c9752da13e6dec012fb07/raw/cloudbridge_py3.8_openstack.json
             :target: https://github.com/CloudVE/cloudbridge/actions/

+---------------------------+----------------+
| **Provider/Environment**  | **Python 3.8** |
+---------------------------+----------------+
| **Amazon Web Services**   | |aws-py38|     |
+---------------------------+----------------+
| **Google Cloud Platform** | |gcp-py38|     |
+---------------------------+----------------+
| **Microsoft Azure**       | |azure-py38|   |
+---------------------------+----------------+
| **OpenStack**             | |os-py38|      |
+---------------------------+----------------+
| **Mock Provider**         | |mock-py38|    |
+---------------------------+----------------+

Installation
~~~~~~~~~~~~
Install the latest release from PyPi:

.. code-block:: shell

  pip install cloudbridge[full]

For other installation options, see the `installation page`_ in
the documentation.

Usage example
~~~~~~~~~~~~~

To `get started`_ with CloudBridge, export your cloud access credentials
(e.g., AWS_ACCESS_KEY and AWS_SECRET_KEY for your AWS credentials) and start
exploring the API:

.. code-block:: python

  from cloudbridge.factory import CloudProviderFactory, ProviderList

  provider = CloudProviderFactory().create_provider(ProviderList.AWS, {})
  print(provider.security.key_pairs.list())

The exact same command (as well as any other CloudBridge method) will run with
any of the supported providers: ``ProviderList.[AWS | AZURE | GCP | OPENSTACK]``!

Citation
~~~~~~~~

N. Goonasekera, A. Lonie, J. Taylor, and E. Afgan,
"CloudBridge: a Simple Cross-Cloud Python Library,"
presented at the Proceedings of the XSEDE16 Conference on Diversity, Big Data, and Science at Scale, Miami, USA, 2016.
DOI: http://dx.doi.org/10.1145/2949550.2949648

Quick Reference
~~~~~~~~~~~~~~~
The following object graph shows how to access various provider services, and the resource
that they return.

.. image:: http://cloudbridge.readthedocs.org/en/latest/_images/object_relationships_detailed.svg
   :target: http://cloudbridge.readthedocs.org/en/latest/?badge=latest#quick-reference
   :alt: CloudBridge Quick Reference

Design Goals
~~~~~~~~~~~~

1. Create a cloud abstraction layer which minimises or eliminates the need for
   cloud specific special casing (i.e., Not require clients to write
   ``if EC2 do x else if OPENSTACK do y``.)

2. Have a suite of conformance tests which are comprehensive enough that goal
   1 can be achieved. This would also mean that clients need not manually test
   against each provider to make sure their application is compatible.

3. Opt for a minimum set of features that a cloud provider will support,
   instead of  a lowest common denominator approach. This means that reasonably
   mature clouds like Amazon and OpenStack are used as the benchmark against
   which functionality & features are determined. Therefore, there is a
   definite expectation that the cloud infrastructure will support a compute
   service with support for images and snapshots and various machine sizes.
   The cloud infrastructure will very likely support block storage, although
   this is currently optional. It may optionally support object storage.

4. Make the CloudBridge layer as thin as possible without compromising goal 1.
   By wrapping the cloud provider's native SDK and doing the minimal work
   necessary to adapt the interface, we can achieve greater development speed
   and reliability since the native provider SDK is most likely to have both
   properties.

Contributing
~~~~~~~~~~~~
Community contributions for any part of the project are welcome. If you have
a completely new idea or would like to bounce your idea before moving forward
with the implementation, feel free to create an issue to start a discussion.

Contributions should come in the form of a pull request. We strive for 100% test
coverage so code will only be accepted if it comes with appropriate tests and it
does not break existing functionality. Further, the code needs to be well
documented and all methods have docstrings. We are largely adhering to the
`PEP8 style guide`_ with 80 character lines, 4-space indentation (spaces
instead of tabs), explicit, one-per-line imports among others. Please keep the
style consistent with the rest of the project.

Conceptually, the library is laid out such that there is a factory used to
create a reference to a cloud provider. Each provider offers a set of services
and resources. Services typically perform actions while resources offer
information (and can act on itself, when appropriate). The structure of each
object is defined via an abstract interface (see
``cloudbridge/providers/interfaces``) and any object should implement the
defined interface. If adding a completely new provider, take a look at the
`provider development page`_ in the documentation.

.. _`installation page`: http://cloudbridge.readthedocs.org/en/
   latest/topics/install.html
.. _`get started`: http://cloudbridge.readthedocs.org/en/latest/
    getting_started.html
.. _`PEP8 style guide`: https://www.python.org/dev/peps/pep-0008/
.. _`provider development page`: http://cloudbridge.readthedocs.org/
   en/latest/
    topics/provider_development.html