# SATOSA

[![PyPI](https://img.shields.io/pypi/v/SATOSA.svg)](https://pypi.python.org/pypi/SATOSA)

A configurable proxy for translating between different authentication protocols
such as SAML2, OpenID Connect and OAuth2.

# Table of Contents

- [Installation](doc/README.md#installation)
  - [Docker](doc/README.md#docker)
  - [Manual installation](doc/README.md#manual-installation)
    - [Dependencies](doc/README.md#dependencies)
    - [Instructions](doc/README.md#instructions)
- [Configuration](doc/README.md#configuration)
  - [SATOSA proxy configuration: proxy_conf.yaml.example](doc/README.md#satosa-proxy-configuration-proxy_confyamlexample)
  - [Attribute mapping configuration: internal_attributes.yaml](doc/README.md#attribute-mapping-configuration-internal_attributesyaml)
    - [attributes](doc/README.md#attributes)
    - [user_id_from_attrs](doc/README.md#user_id_from_attrs)
    - [user_id_to_attr](doc/README.md#user_id_to_attr)
- [Plugins](doc/README.md#plugins)
  - [SAML2 plugins](doc/README.md#saml2-plugins)
    - [Metadata](doc/README.md#metadata)
    - [AuthnContextClassRef](doc/README.md#providing-authncontextclassref)
    - [Frontend](doc/README.md#saml2-frontend)
      - [Custom attribute release](doc/README.md#custom-attribute-release)
      - [Policy](doc/README.md#policy)
    - [Backend](doc/README.md#saml2-backend)
      - [Name ID Format](doc/README.md#name-id-format)
      - [Discovery service](doc/README.md#use-a-discovery-service)
      - [ForceAuthn option](doc/README.md#mirror-the-saml-forceauthn-option)
      - [Memorize IdP](doc/README.md#memorize-the-idp-selected-through-the-discovery-service)
  - [OpenID Connect plugins](doc/README.md#openid-connect-plugins)
    - [Frontend](doc/README.md#oidc-frontend)
    - [Backend](doc/README.md#oidc-backend)
    - [Social login plugins](doc/README.md#social-login-plugins)
      - [Google](doc/README.md#google)
      - [Facebook](doc/README.md#facebook)
  - [Dummy adapters](doc/README.md#dummy-adapters)
  - [Micro-services](doc/README.md#micro-services)
- [Generating proxy metadata](doc/README.md#generate-proxy-metadata)
- [Running the proxy application](doc/README.md#running-the-proxy-application)
- [External contributions](doc/README.md#external-contributions)

# Use cases

In this section a set of use cases for the proxy is presented.

## SAML2<->SAML2

There are SAML2 service providers for example Box which is not able to handle
multiple identity providers. For more information about how to set up,
configure and run such a proxy instance please visit [Single Service
Provider<->Multiple Identity providers](doc/one-to-many.md)

If an identity provider can not communicate with service providers in for
example a federation, they can convert requests and make the communication
possible.

## SAML2<->Social logins

This setup makes it possible to connect a SAML2 service provider to multiple
social media identity providers such as Google and Facebook. The proxy makes it
possible to mirror an identity provider by generating SAML2 metadata
corresponding to that provider and create dynamic endpoints which are connected to
a single identity provider.

For more information about how to set up, configure and run such a proxy
instance please read [SAML2<->Social logins](doc/SAML2-to-Social_logins.md)

## SAML2<->OIDC

The proxy is able to act as a proxy between a SAML2 service provider and a
OpenID connect provider [SAML2<->OIDC](doc/saml2-to-oidc.md)

# Contact

If you have any questions regarding operations/deployment of SATOSA please use
the satosa-users [mailing list](https://lists.sunet.se/postorius/lists/idpy-discuss.lists.sunet.se/).


.. image:: https://img.shields.io/pypi/v/dimod.svg
    :target: https://pypi.org/project/dimod

.. image:: https://img.shields.io/pypi/pyversions/dimod.svg
    :target: https://pypi.python.org/pypi/dimod

.. image:: https://circleci.com/gh/dwavesystems/dimod.svg?style=svg
    :target: https://circleci.com/gh/dwavesystems/dimod

.. image:: https://codecov.io/gh/dwavesystems/dimod/branch/main/graph/badge.svg
    :target: https://codecov.io/gh/dwavesystems/dimod

dimod
=====

.. index-start-marker1

`dimod` is a shared API for samplers. It provides:

* classes for quadratic models---such as the binary quadratic model (BQM) class that
  contains Ising and QUBO models used by samplers such as the D-Wave system---and
  higher-order (non-quadratic) models.
* reference examples of samplers and composed samplers.
* `abstract base classes <https://docs.python.org/3/library/abc.html>`_ for
  constructing new samplers and composed samplers.

.. index-end-marker1

(For explanations of the terminology, see the
`Ocean glossary <https://docs.ocean.dwavesys.com/en/stable/concepts/index.html>`_.)

Example Usage
-------------

.. index-start-marker2

>>> import dimod
...
>>> # Construct a problem
>>> bqm = dimod.BinaryQuadraticModel({0: -1, 1: 1}, {(0, 1): 2}, 0.0, dimod.BINARY)
...
>>> # Use dimod's brute force solver to solve the problem
>>> sampleset = dimod.ExactSolver().sample(bqm)
>>> print(sampleset)
   0  1 energy num_oc.
1  1  0   -1.0       1
0  0  0    0.0       1
3  0  1    1.0       1
2  1  1    2.0       1
['BINARY', 4 rows, 4 samples, 2 variables]

.. index-end-marker2

See the `documentation <https://docs.ocean.dwavesys.com/en/stable/docs_dimod/>`_
for more examples.

Installation
------------

.. installation-start-marker

Installation from `PyPI <https://pypi.org/project/dimod>`_:

.. code-block:: bash

    pip install dimod

.. installation-end-marker

License
-------

Released under the Apache License 2.0. See LICENSE file.

Contributing
------------

Ocean's `contributing guide <https://docs.ocean.dwavesys.com/en/stable/contributing.html>`_
has guidelines for contributing to Ocean packages.

dimod includes some formatting customization in the
`.clang-format <.clang-format>`_ and `setup.cfg <setup.cfg>`_ files.

Release Notes
~~~~~~~~~~~~~

dimod makes use of `reno <https://docs.openstack.org/reno/>`_ to manage its
release notes.

When making a contribution to dimod that will affect users, create a new
release note file by running

.. code-block:: bash

    reno new your-short-descriptor-here

You can then edit the file created under ``releasenotes/notes/``.
Remove any sections not relevant to your changes.
Commit the file along with your changes.

See reno's `user guide <https://docs.openstack.org/reno/latest/user/usage.html>`_
for details.


The E2E sample implements aspects of smart city sensing, analytics and management features as follows:

- **Camera Provisioning**: Tag and calibrate cameras for installation locations, calibration parameters and other usage pattern information.
- **Camera Discovery**: Discover and register IP cameras on specified IP blocks. Registered cameras automatically participate into the analytics activities. See [Sensor Simulation and Discovery](sensor/README.md) for additional details.
- **Recording**: Record and manage segmented camera footage for preview or review (at a later time) purpose.
- **Analytics**: Perform analytics on the live/recorded camera streams. Latency-sensitive analytics are performed on Edge while others are on cloud.
- **Triggers and Alerts**: Manage triggers on analytics data. Respond with actions on triggered alerts.
- **Smart Upload and Archive**: Transcode and upload only critical data to cloud for archival or further offline analysis.
- **Stats**: Calculate statistics for planning/monitoring purpose on analytical data.
- **UI**: Present above data to users/administrators/city planners.

### Scenarios

The sample implements the Smart-City [`traffic`](https://github.com/OpenVisualCloud/Smart-City-Sample/wiki/Smart-City:-Traffic-Scenario) and [`stadium`](https://github.com/OpenVisualCloud/Smart-City-Sample/wiki/Smart-City:-Stadium-Scenario) scenarios. The [`traffic`](https://github.com/OpenVisualCloud/Smart-City-Sample/wiki/Smart-City:-Traffic-Scenario) scenario measures vehicle/pedestrian flow for planning purpose. The [`stadium`](https://github.com/OpenVisualCloud/Smart-City-Sample/wiki/Smart-City:-Stadium-Scenario) scenario focuses on different access control techniques, including entrance people counting, service-point queue counting, and stadium seating zone crowd counting.

| [Traffic](https://www.youtube.com/watch?v=BWU0SEqEfbo") | Stadium |
|:-------:|:-------:|
|||

### Software Stacks

The sample is powered by the following OpenVisualCloud software stacks:
- **Edge Low-latency Analytics**:
  - [The GStreamer-based media analytics stack](https://github.com/OpenVisualCloud/Dockerfiles/tree/master/Xeon/ubuntu-20.04/analytics/gst) is used for object detection, people-counting, queue-counting and crowd-counting on camera streams. The software stack is optimized for [Intel® Xeon® Scalable Processors](https://github.com/OpenVisualCloud/Dockerfiles/tree/master/Xeon/ubuntu-20.04/analytics/gst).

- **Smart Upload with Transcoding**:
  - [The FFmpeg-based media transcoding stack](https://github.com/OpenVisualCloud/Dockerfiles/tree/master/Xeon/ubuntu-20.04/media/ffmpeg) is used to transcode recorded content before uploading to cloud. The software stack is optimized for [Intel Xeon Scalable Processors](https://github.com/OpenVisualCloud/Dockerfiles/tree/master/Xeon/ubuntu-20.04/media/ffmpeg).

### Install Prerequisites:

- **Time Zone**: Check that the timezone setting of your host machine is correctly configured. Timezone is used during build. If you plan to run the sample on a cluster of machines managed by Docker Swarm or Kubernetes, please make sure to synchronize time among the manager/master node and worker nodes.

- **Build Tools**: Install `cmake`, `make`, `m4`, `wget` and `gawk` if they are not available on your system.

- **Docker Engine**:
  - Install [docker engine](https://docs.docker.com/install). Minimum version required: `17.05`. Make sure you [setup](https://docs.docker.com/install/linux/linux-postinstall) docker to run as a regular user.
  - Setup [docker swarm](https://docs.docker.com/engine/swarm), if you plan to deploy through docker swarm. See [Docker Swarm Setup](deployment/docker-swarm/README.md) for additional setup details.
  - Setup [Kubernetes](https://kubernetes.io/docs/setup), if you plan to deploy through Kubernetes. See [Kubernetes Setup](deployment/kubernetes/README.md) for additional setup details.
  - Setup docker proxy as follows if you are behind a firewall:

```bash
sudo mkdir -p /etc/systemd/system/docker.service.d
printf "[Service]\nEnvironment=\"HTTPS_PROXY=$https_proxy\" \"NO_PROXY=$no_proxy\"\n" | sudo tee /etc/systemd/system/docker.service.d/proxy.conf
sudo systemctl daemon-reload
sudo systemctl restart docker
```

### Build Sample:

Use the following commands to build the sample. By default, the sample builds to the `traffic` scenario. To enable the `stadium` scenario, use `cmake -DSCENARIO=stadium ..`. See also: [Build Options](doc/cmake.md).

```bash
mkdir build
cd build
cmake ..
make
```

### Start/stop Sample:

Use the following commands to start/stop services via docker swarm:

```bash
make update # optional for private registry
make start_docker_swarm
make stop_docker_swarm
```

See also:  [Docker Swarm Setup](deployment/docker-swarm/README.md).

Use the following commands to start/stop Kubernetes services:

```
make update # optional for private registry
make start_kubernetes
make stop_kubernetes
```

See also: [Kubernetes Setup](deployment/kubernetes/README.md).

### Launch Sample UI:

Launch your browser and browse to ```https://<hostname>``` for the sample UI.

---

* For Kubernetes/Docker Swarm, ```<hostname>``` is the hostname of the manager/master node.
* If you see a browser warning of self-signed certificate, please accept it to proceed to the sample UI.

---

### See Also

- [Configuration Options](doc/cmake.md)
- [Docker Swarm Setup](deployment/docker-swarm/README.md)
- [Kubernetes Setup](deployment/kubernetes/README.md)
- [REST API List](doc/restapi.md)
- [Demo Video](https://www.youtube.com/watch?v=BWU0SEqEfbo)


Tethys Platform
===============

.. image:: https://github.com/tethysplatform/tethys/actions/workflows/tethys.yml/badge.svg
    :target: https://github.com/tethysplatform/tethys/actions
    :alt: Test Status

.. image:: https://coveralls.io/repos/github/tethysplatform/tethys/badge.svg
    :target: https://coveralls.io/github/tethysplatform/tethys
    :alt: Coverage Status

.. image:: https://readthedocs.org/projects/tethys-platform/badge/?version=stable
    :target: http://docs.tethysplatform.org/en/stable/?badge=stable
    :alt: Documentation Status

Tethys Platform provides both a development environment and a hosting environment for water resources web apps.

Documentation can be found here: `<http://docs.tethysplatform.org/>`_


# Welcome to the Cloud Rapid Experimentation and Analysis Toolkit

Cloud Rapid Experimentation and Analysis Tool (aka CBTOOL) is a framework that automates IaaS cloud benchmarking through the running of controlled experiments.

[![Gitter](https://badges.gitter.im/ibmcb-project/community.svg)](https://gitter.im/ibmcb-project/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

Subscribe to our mailing list:

- Users: https://groups.google.com/forum/#!forum/cbtool-users

- Development: https://groups.google.com/forum/#!forum/cbtool-devel

## New! CloudBench is now released as a component of [SPEC Cloud IaaS 2018](http://spec.org/cloud_iaas2018)

## To get started:

1. [Learn more about the tool](https://github.com/ibmcb/cbtool/wiki/DOC:-Table-of-Contents)

2. [Perform the initial installation](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Initial-Installation)

3. [Run the tool for the first time](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Running-the-tool-for-the-first-time)

4. [Prepare your cloud to be driven by the tool](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Preparing-your-cloud-to-be-driven-by-CBTOOL)

5. [Prepare a VM image to be used with the tool](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Preparing-a-VM-to-be-used-with-CBTOOL-on-a-real-cloud)

====> [Are you impatient? Use our workloads instead.](https://github.com/ibmcb/cbtool/wiki/DOC:-Zero-configuration-(almost)-benchmarking-w--Nested-containers)

6. [Deploy your first Virtual Application](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Deploy-your-first-virtual-application)

7. [Run simple experiments](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Run-simple-experiments)

8. [If needed, debug the initial setup](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Debug-initial-setup)

9. [Configure CBTOOL to run outside of the cloud (or with multiple tenant networks)](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Run-the-CloudBench-orchestrator-outside-of-the-cloud-(or-with-multiple-tenant-networks))

    1. [Detailed instructions on how to use OpenVPN support](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Use-VPN-support-with-your-benchmarks)

10. Look at some [auto generated plots](https://github.com/ibmcb/cbtool/wiki/HOWTO:-Save-Monitoring-Data-on-the-Command-Line) made from the data collected.

11. Read our latest paper [IC2E 2013](http://dl.acm.org/citation.cfm?id=2497243).

12. Try administrating the tool with the [Graphical Environment](https://github.ibm.com/marcios/cbtool/wiki/HOWTO:-Using-the-Graphical-Environment)

    1. Then try [monitoring your experiments with the Graphical Environment](https://github.ibm.com/marcios/cbtool/wiki/HOWTO:-Monitoring-with-the-Graphical-Environment)

    2. Try to [customize your dashboard monitoring data with filters in the Graphical Environment](https://github.ibm.com/marcios/cbtool/wiki/HOWTO:-Customize-Dashboard-Monitoring-in-the-Graphical-Environment)

    3. You can also try to use the [Graphical Wizard](https://github.ibm.com/marcios/cbtool/wiki/HOWTO:-Using-the-Wizard-for-first-time-connection) for a first-time connection.

13. Read the [Frequently Asked Questions](https://github.com/ibmcb/cbtool/wiki/FAQ)

## Features:
- Automatic deployment and controlled execution of multiple multi-tier applications.
Each individual application can have a specific load profile using probability distributions.
- Adapters for multiple clouds (EC2 and OpenStack, among others), with a plugin structure that allows new cloud models to be added incrementally.
- Can orchestrate different arrival rates and lifetimes for VMs using probability distributions.
- Collects application and system (OS) performance data from hosts and guests in real time.
- It is designed from the ground up to be highly scalable and parallel.

## [Supported Clouds](https://github.com/ibmcb/cbtool/wiki/DOC:-Supported-Clouds):

1. Amazon EC2
2. OpenStack (and RackSpace)
3. Google Compute Engine
4. DigitalOcean
5. Linode (Akamai Connected Cloud), NOTE: libcloud version 3.9.0 only: https://github.com/apache/libcloud/pull/1946
6. Docker/Swarm
7. LXD/LXC
8. Kubernetes
9. Libvirt+KVM
10. VMWare vCloud (NOT actively maintained)
11. CloudStack (NOT actively maintained)
12. SoftLayer
13. Nutanix

Want to add support for a new Cloud? Take a look at our [Frequently Asked Questions](https://github.com/ibmcb/cbtool/wiki/FAQ#development-)

## [Supported Workloads](https://github.com/ibmcb/cbtool/wiki/DOC:-Supported-Virtual-Applications)

To get the most current list, start CBTOOL and type ```typelist``` on the CLI. To get more information about a given workload, ```typeshow <workload name>```:

> (MYCLOUD) typelist

> AIs with the following types can be attached to to this experiment (Cloud MYSIMCLOUD) :

1. synthetic:
2.  bonnie (default, full)
3.   btest (default)
4.  coremark (default)
5.  ddgen (default)
6.  filebench (fileserver, oltp_noism, varmail, videoserver, webproxy)
7.  fio (default)
8.  iperf (tcp, udp)
9.  mlg (default)
10.  netperf (tcp_stream, tcp_maerts, udp_stream, tcp_rr, tcp_cc, tcp_crr, udp_rr)
11.  nuttcp (tcp, udp)
12.  postmark (default)
13.  unixbench (arithmetic, dhrystone, whetstone, load, misc, speed, oldsystem, system, fs, shell, index)
14.  xping (icmp)

> application-stress:
1.  memtier (default)
2.  oldisim (default)
3.  wrk (default)

> scientific:
1.  hpcc (default)
2.  linpack (default)
3.  multichase (simple, work:N, t0-2, nta, movdqa, mvntdqa, parallel2-10, critword:N, critword2:N)
4.  parboil (histo, bfs, stencil, mri-q, mri-gridding, lbm, tpacf, sad, spmv, sgemm, cutcp)
5.  scimark (default)

> transactional:
1.  cassandra_ycsb (workloada, workloadb, workloadc, workloadd, workloade, workloadf)
2.  ibm_daytrader (default)
3.  mongo_ycsb (workloada, workloadb, workloadc, workloadd, workloade, workloadf)
4.  open_daytrader (default)
5.  redis_ycsb (workloada, workloadb, workloadc, workloadd, workloade, workloadf)
6.  specjbb (preset, hbir)
7.  sysbench (simple, complex, nontrx, sp)

> data-centric:
1.  giraph (pagerank, topkpagerank)
2.  hadoop (sort, wordcount, terasort, dfsioe, nutchindexing, pagerank, bayes, kmeans, hivebench)

> fake:
1.  nullworkload (default)

Contacts:

Marcio Silva marcios@us.ibm.com
Michael Galaxy mgalaxy@akamai.com


# Deezer Python Client

---

**Documentation**: https://deezer-python.readthedocs.io

**Source Code**: https://github.com/browniebroke/deezer-python

---

A friendly Python wrapper around the [Deezer API](https://developers.deezer.com/api).

## Installation

The package is published on
[PyPI](https://pypi.org/project/deezer-python/) and can be installed by running:

    pip install deezer-python

## Basic Use

Easily query the Deezer API from you Python code. The data returned by the Deezer
API is mapped to python resources:

```pycon
>>> client = deezer.Client()
>>> client.get_album(680407).title
'Monkey Business'
```

Ready for more? Look at our whole [documentation](http://deezer-python.readthedocs.io/)
on Read The Docs or have a play in pre-populated Jupyter notebook
[on Binder](https://mybinder.org/v2/gh/browniebroke/deezer-python/main?filepath=demo.ipynb).

## Contributors

Bruno Alla
💻 📖 🤔 🚧 📦 ⚠

misuzu
💻 📖 🤔 ⚠

Pascal
💻 ⚠

khamaileon
📖

Nikolay Sheregeda
💻 ⚠

Matheus Horstmann
💻 📖

Kieran Wynne
💻

Jonathan Virga
💻 📖

Hugo
💻

allcontributors[bot]
🚇

nrebena
🐛 💻 ⚠

Pavel
🐛

Samuel Gaist
💻 ⚠ 🛡

Thomas Morelli
🐛 💻 🤔

Naveen M K
💻 🚇

Carababusha
💻

Bertrand Jacquin
💻

Pierre
💻 📖

Jonathan
💻 📖 ⚠

This project follows the [all-contributors](https://allcontributors.org) specification.
Contributions of any kind are welcome!


# tskit  <img align="right" width="145" height="90" src="https://github.com/tskit-dev/administrative/blob/main/tskit_logo.svg">

[![License](https://img.shields.io/github/license/tskit-dev/tskit)](https://github.com/tskit-dev/tskit/blob/main/LICENSE)
[![Contributors](https://img.shields.io/github/contributors/tskit-dev/tskit)](https://github.com/tskit-dev/tskit/graphs/contributors)
[![Commit activity](https://img.shields.io/github/commit-activity/m/tskit-dev/tskit)](https://github.com/tskit-dev/tskit/commits/main)
[![Coverage](https://codecov.io/gh/tskit-dev/tskit/branch/main/graph/badge.svg)](https://codecov.io/gh/tskit-dev/tskit)
![OS](https://img.shields.io/badge/OS-linux%20%7C%20OSX%20%7C%20win--64-steelblue)

Succinct tree sequences are a highly efficient way of storing a set of related DNA
sequences by encoding their ancestral history as a set of correlated trees along the
genome. The tree sequence format is output by a number of software libraries and programs
(such as [msprime](https://github.com/tskit-dev/msprime),
[SLiM](https://github.com/MesserLab/SLiM),
[fwdpp](http://molpopgen.github.io/fwdpp/), and
[tsinfer](https://tsinfer.readthedocs.io/en/latest/)) that either simulate or infer
the evolutionary history of genetic sequences. The evolutionary history of genetic
sequences is often technically referred to as an Ancestral Recombination Graph (ARG);
succinct tree sequences are fully compatible with this formulation, and tskit is a
therefore a powerful platform for processing ARGs.

The `tskit` library provides the underlying functionality used to load, examine, and
manipulate tree sequences, including efficient methods for calculating genetic
statistics. It often forms part of an installation of other software packages such as
those listed above. Please see the
[documentation](https://tskit.dev/tskit/docs/latest/) for further details, which
includes
[installation instructions](https://tskit.dev/tskit/docs/latest/installation.html).
Also see the [road map](https://github.com/tskit-dev/tskit/blob/main/ROADMAP.md) for
planned improvements and additions to the library.

To get started with tskit, tutorials and other content are at http://tskit.dev. For help
and support from the community you can use
[discussions](https://github.com/tskit-dev/tskit/discussions) here on github, or raise an
issue for a specific bug or feature request.

We warmly welcome contributions from the community. Raise an issue if you have an
idea you'd like to work on, or submit a PR for comments and help.

The base `tskit` library provides both a Python and C API. A Rust API is provided in the
[tskit-rust](https://github.com/tskit-dev/tskit-rust) repository.

#### Python API
[![PyPI version](https://img.shields.io/pypi/v/tskit.svg)](https://pypi.org/project/tskit/)
[![Supported Python Versions](https://img.shields.io/pypi/pyversions/tskit.svg)](https://pypi.org/project/tskit/)
[![Wheel](https://img.shields.io/pypi/wheel/tskit)](https://pypi.org/project/tskit/)
[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Travis](https://img.shields.io/travis/tskit-dev/tskit)](https://travis-ci.org/github/tskit-dev/tskit)

Most users of `tskit` will use the python API as it provides a convenient, high-level API
to access, analyse and create tree sequences. Full documentation is
[here](https://tskit.dev/tskit/docs/latest/python-api.html).

#### C API
[![C99](https://img.shields.io/badge/Language-C99-steelblue.svg)](https://en.wikipedia.org/wiki/C99)
[![CircleCI](https://circleci.com/gh/tskit-dev/tskit.svg?style=shield)](https://circleci.com/gh/tskit-dev/tskit)

The `tskit` C API provides comprehensive, low-level methods for manipulating and
processing tree-sequences. Written to the C99 standard and fully thread-safe, it can be
used with either C or C++. Full documentation is
[here](https://tskit.dev/tskit/docs/latest/c-api.html).


.. _pycatia.readthedocs.io: https://pycatia.readthedocs.io
.. _installation: https://pycatia.readthedocs.io/en/latest/installation.html
.. _introduction: https://pycatia.readthedocs.io/en/latest/introduction.html
.. _examples: https://pycatia.readthedocs.io/en/latest/examples.html
.. _user_scripts: https://pycatia.readthedocs.io/en/latest/user_scripts.html
.. _pypi.org: https://pypi.org/project/pycatia/
.. _pycatia-tools: https://github.com/evereux/pycatia-tools

pycatia
=======

alpha software
--------------

This is alpha software.

All the test cases and examples work but there will be many issues outside of
the test framework. The CATIA com interface is huge and I've predominantly
only worked / tested on those I can figure out how to use.

The entire framework is now in place for others to contribute so if you know
CATIA and python (being already familiar with automation using the more
traditional methods will help) please do contribute. Bonus points for adding
tests too.

I have limited access to CATIA licences / workbenches. Also, there's are many
modules that I simply don't know what they do. Thus my ability to test and
support can be limited.

All this said many bugs that exist should be for the most part be quite easy to
fix using those methods that do work as a reference.

Requirements
------------

* python >= 3.9
* **CATIA V5** running on Windows.
* see requirements.txt

Installation
------------

see installation_.

Usage
-----

See the introduction_,  examples_ and user_scripts_ for a good over view on how
to use pycatia.

Links
-----

Releases: pycatia @ pypi.org_

Examples And Scripts
--------------------

See the documentation @ examples_ and user_scripts_.

A GUI based application that uses pycatia - pycatia-tools_.

Asking Questions
----------------

Please read the following with regards to raising questions: https://github.com/evereux/pycatia/issues/28

Contributing
------------

See CONTRIBUTING.md in root of github repository.

Running The Tests
-----------------

Prior to running the tests please ensure CATIA V5 has the following
configuration settings:

* CGR cache system must be disabled.
* Do not activate default shapes on open must be disabled.
* Parameter names must not have back ticks enabled. Tools > Options > General > Parameters and Measure > Knowledge > Parameter Names > Surrounded by the \` symbol.

On the first run, during the running of the tests, the test suite will create
the CATIA drawing, products and part it requires to run in the folder
tests/cat_files.

CATIA V5 should already be running and have NO documents already open.

To run the tests with coverage (-v is verbosity):

.. code-block:: python

    py.test -v --cov-report term-missing --cov=pycatia

To run a specific test:

.. code-block:: python

    py.test -v tests/test_product.py::test_move

To stop tests running after first failure.

    py.test -vx

Release process
---------------

A reminder for @evereux. I don't do this often and forget ...

* Check version is correct.

* Check changelog has been updated.

* Run the tests. `pytest -v tests`
   * Ensure cache is disabled and
   * Fix any issues.

* Run the examples.
   * Fix any issues.

* Build the docs. `cd docs` `./make html`
   * Fix any issues.

* Run mypy over module. `mypy pycatia`

* Build source. ``python setup.py sdist bdist_wheel``
   * Check source contents.

* Build pycatia exe ``python -m nuitka --standalone pycatia-exe.py``.
  * use 64 env.
  * rename pycatia-exe.exe.
  * copy build to win_32 folder and zip.

* Merge changes with master branch and upload.

* Upload to pypi. ``twine upload dist/*``.

* Update github releases.


# RayTracing

by the DCC/M Lab group http://www.dccmlab.ca, guided by Prof. [Daniel Côté](mailto:dccote@cervo.ulaval.ca?subject=Raytracing%20python%20module).

**[Update September 2023]: What are we up to? You will notice the master branch has not changed in some time. However, other branches are actively being worked on, including a GPU-accelerated branch that promises to open up possibilities for Monte Carlo calculations. More extensive tutorials, with practical lab-related calculations, are also being worked on.  Stay tuned.**

This code aims to provide a simple ray tracing module for calculating various properties of optical paths (object, image, aperture stops, field stops).  It makes use of ABCD matrices and does not consider spherical aberrations but can compute chromatic aberrations for simple cases when the materials are known. Since it uses the ABCD formalism (or Ray matrices, or Gauss matrices) it can perform tracing of rays and gaussian laser beams.

It is not a package to do "Rendering in 3D with raytracing".

The code has been developed first for teaching purposes and is used in my "[Optique](https://itunes.apple.com/ca/book/optique/id949326768?mt=11)" Study Notes (french only), but also for actual use in my research. As of January 21st, 2021, there is an extensive, freely accessible, peer-reviewed tutorial in Journal of Neurophotonics:

> ["Tools and tutorial on practical ray tracing for microscopy"](https://doi.org/10.1117/1.NPh.8.1.010801)
>
> by V. Pineau Noël\*, S. Masoumi\*, E. Parham\*, G. Genest, L. Bégin, M.-A. Vigneault, D. C. Côté,
> Neurophotonics, 8(1), 010801 (2021).
> *Equal contributions.
> Permalink: https://doi.org/10.1117/1.NPh.8.1.010801

The published tutorial assumes version 1.3.x.  There are video [tutorials](https://www.youtube.com/playlist?list=PLUxTghemi4Ft0NzQwuufpU-EGgkmaInAf) (in english or french, with english subtitles when in french) on YouTube. We have made no attempts at making high performance code.  **Readability** and **simplicity of usage** are the key here. It is a module with a few files, and only `matplotlib` and `numpy` as dependent modules.

## Where do I get started?
   * If you want to use the module, keep reading.
   * If you have a suggestion or a bug report, go to [Issues](https://github.com/DCC-Lab/RayTracing/issues).
   * If you want to read and contribute to the code, go to the [Wiki](https://github.com/DCC-Lab/RayTracing/wiki) for general considerations. We plan to have a roadmap in the near future.

## Getting started
The module defines `Ray` , `Matrix`, `MatrixGroup` and `ImagingPath` as the main elements for tracing rays.  `Matrix` and `MatrixGroup` are either one or a sequence of many matrices into which `Ray` will propagate. `ImagingPath` is also a sequence of elements, with an object at the front edge.  Specific subclasses of `Matrix` exists: `Space`, `Lens`, `ThicklLens`, and `Aperture`. Finally, a ray fan is a collection of rays, originating from a given point with a range of angles.

We have tried to separate the calculation code (i.e. the matrices and subclasses) from the drawing code (figures and graphics). One can use the calculation code without any graphics calls.

If you want to perform calculations with coherent laser beams, then you use `GaussianBeam` and `LaserPath`. Everything is essentially the same, except that the formalism does not allow for the gaussian beam to be "blocked", hence any calculation of stops with aperture are not available in `LaserPath`. That part of the code is less developed, but it is nevertheless available.

## What's new?

To get information about what is new, currently the best place is the [release page on GitHub.](https://github.com/DCC-Lab/RayTracing/releases)

There is a **[Frequently Asked Questions](./FAQ.md)** page.

The article above is fully compatible with all 1.3.x versions.  As long as the API does not change, versions will be 1.3.x.

## Installing and upgrading

You need `matplotlib`, which is a fairly standard Python module. If you do not have it,  installing [Anaconda](https://www.anaconda.com/download/) is your best option. Python 3.6 or later is required. There are several ways to install the module:

1. Simplest: `pip install raytracing` or `pip install --upgrade raytracing`
   1. If you need to install `pip`, download [getpip.py](https://bootstrap.pypa.io/get-pip.py) and run it with `python getpip.py`
2. If you download the [source](https://pypi.org/project/raytracing/) of the module, then you can type: `python setup.py install`
3. From GitHub, you can get the latest version (including bugs, which are 153% free!) and then type `python setup.py install`
4. If you are completely lost, copying the folder `raytracing` (the one that includes `__init__.py`) from the source file into the same directory as your own script will work.
5. Watch the tutorial with subtitles [here.](https://www.youtube.com/playlist?list=PLUxTghemi4Ft0NzQwuufpU-EGgkmaInAf)

## Getting started

The simplest way to import the package in your own scripts after installing it:

```python
from raytracing import *
```

This will import `Ray` , `GaussianBeam`,  and several `Matrix` elements such as `Space`, `Lens`, `ThickLens`, `Aperture`, `DielectricInterface`, but also `MatrixGroup` (to group elements together),  `ImagingPath` (to ray trace with an object at the front edge), `LaserPath` (to trace a gaussian laser beam from the front edge) and a few predefined other such as `Objective` (to create a very thick lens that mimicks an objective).

You create an `ImagingPath` or a `LaserPath`, which you then populate with optical elements such as `Space`, `Lens` or `Aperture` or vendor lenses. You can then adjust the path properties (object height in `ImagingPath` for instance or inputBeam for `LaserPath`) and display in matplotlib. You can create a group of elements with `MatrixGroup` for instance a telescope, a retrofocus or any group of optical elements you would like to treat as a "group".  The Thorlabs and Edmund optics lenses, for instance, are defined as `MatrixGroups`.

This will show you a list of examples of things you can do (more on that in the Examples section):

```shell
python -m raytracing -l           # List examples
python -m raytracing -e all       # Run all of them
python -m raytracing -e 1,2,4,6   # Only run 1,2,4 and 6
python -m raytracing -t           # Run all the tests.  Some performance tests can take up to a minute, but they should all pass.
```

or request help with:

```
python -m raytracing -h
```

In your code, you would do this:

```python
from raytracing import *

path = ImagingPath()
path.append(Space(d=50))
path.append(Lens(f=50, diameter=25))
path.append(Space(d=120))
path.append(Lens(f=70))
path.append(Space(d=100))
path.display()
```

You can also call `display()` on an element to see the cardinal points, principal planes, BFL (back focal length, or the distance between the last interface and the focal point after the lens) and FFL (front focal length, or the distance between the focal point before the lens and the first interface). You can do it with any single `Matrix` element but also with `MatrixGroup`.

```python
from raytracing import *

thorlabs.AC254_050_A().display()
eo.PN_33_921().display()
```

![e0](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/e0.png)

![thorlabs](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/thorlabs.png)

Finally, an addition as of 1.2.0 is the ability to obtain the intensity profile of a given source from the object plane at the exit plane of an `OpticalPath`. This is in fact really simple: by tracing a large number of rays, with the number of rays at y and θ being proportionnal to the intensity, one can obtain the intensity profile by plotting the histogram of rays reaching a given height at the image plane. `Rays` are small classes that return a `Ray` that satisfies the condition of the class.  Currently, there is `UniformRays`,`RandomUniformRays` `LambertianRays` and `RandomLambertianRays` (a Lambertian distribution follows a cosθ distribution, it is a common diffuse surface source).  They appear like iterators and can easily be used like this example script:

```python
from raytracing import *
from numpy import *
import matplotlib.pyplot as plt

# Kohler illumination with these variables
fobj = 5
dObj = 5
f2 = 200
d2 = 50
f3 = 100
d3 = 50

# We build the path (i.e. not an Imaging path)
path = OpticalPath()
path.append(Space(d=f3))
path.append(Lens(f=f3, diameter=d3))
path.append(Space(d=f3))
path.append(Space(d=f2))
path.append(Lens(f=f2, diameter=d2))
path.append(Space(d=f2))
path.append(Space(d=fobj))
path.append(Lens(f=fobj, diameter=dObj))
path.append(Space(d=fobj))

# Obtaining the intensity profile
nRays = 1000000 # Increase for better resolution
inputRays = RandomLambertianRays(yMax=2.5, maxCount=nRays)
inputRays.display("Input profile")
outputRays = path.traceManyThrough(inputRays, progress=True)
# On macOS and Linux, you can do parallel computations
# outputRays = path.traceManyThroughInParallel(inputRays, progress=True, processes=8)
outputRays.display("Output profile")

```

and you will get the following ray histograms:

Finally, it is possible to obtain the chromatic aberrations for compound lenses (achromatic doublets from Thorlabs and Edmund optics, and singlet lens because the materials are known). The following command will give you the focal shift as a function of wavelength (as a graph or values):
```python
from raytracing import *

thorlabs.AC254_100_A().showChromaticAberrations()
wavelengths, shifts = thorlabs.AC254_100_A().focalShifts()
```

## Documentation

All the documentation is [available online](https://raytracing.readthedocs.io/).

The class hierarchy for optical elements (with parameters and defaults) is:

![class](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/hierarchy.png)

You may obtain help by:

1. Reading the documentation for the code ([API reference](https://raytracing.readthedocs.io/en/latest/reference.html)).
   1. Core:
      1. `Ray`: a ray for geometrical optics with a height and angle $y$ and $\theta$.
      2. `Rays`: ray distributions to ray trace an object through the optical system.
         1.  `UniformRays`, `RandomUniformRays`, `LambertianRays` and `RandomLambertianRays` are currently available.  See example above.
      3. `GaussianBeam`: a gaussian laser beam with complex radius of curvature $q$.
      4. `Matrix`: any 2x2 matrix.
      5. `MatrixGroup`: treats a group of matrix as a unit (draws it as a unit too)
      6. `ImagingPath`: A `MatrixGroup` with an object at the front for geometrical optics
      7. `LaserPath`: A `MatrixGroup` with a laser beam input at the front or a Resonator.
   2. Optical elements: `Aperture`, `Space`, `Lens`, `DielectricInterface`, `DielectricSlab`, `ThickLens`
   3. Specialty lenses: Defines a general achromat and objective lens
   4. Thorlabs lenses: Achromat doublet lenses from Thorlabs.
   5. Edmund Optics lenses: Achromat doublet lenses from Edmund Optics
   6. Olympus objectives: A few objectives from Olympus.
   7. Glasses: A few glasses used by Thorlabs to make achromatic doublets. They all have a single function `n(wavelength)` that returns the index at that wavelength.  All data obtained from http://refractiveindex.info.
   8. Zemax ZMX file reader: to read text-based Zemax files of lenses.
2. typing (interactively): `help(Matrix)`,`help(MatrixGroup)` `help(Ray)`,`help(ImagingPath)` to get the API,
3. look at the examples with `python -m raytracing`
4. simply look at the code.

```python
python
>>> help(Matrix)
Help on class Matrix in module raytracing.abcd:

class Matrix(builtins.object)
 |  Matrix(A, B, C, D, physicalLength=0, apertureDiameter=inf, label='')
 |
 |  A matrix and an optical element that can transform a ray or another
 |  matrix.
 |
 |  The general properties (A,B,C,D) are defined here. The operator "*" is
 |  overloaded to allow simple statements such as:
 |
 |  ray2 = M1 * ray
 |  or
 |  M3 = M2 * M1
 |
 |  The physical length is included in the matrix to allow simple management of
 |  the ray tracing. IF two matrices are multiplied, the resulting matrice
 |  will have a physical length that is the sum of both matrices.
 |
 |  In addition finite apertures are considered: if the apertureDiameter
 |  is not infinite (default), then the object is assumed to limit the
 |  ray height to plus or minus apertureDiameter/2 from the front edge to the back
 |  edge of the element.
 |
 |  Methods defined here:
 |
 |  __init__(self, A, B, C, D, physicalLength=0, apertureDiameter=inf, label='')
 |      Initialize self.  See help(type(self)) for accurate signature.
 |
 |  __mul__(self, rightSide)
 |      Operator overloading allowing easy to read matrix multiplication
 |
 |      For instance, with M1 = Matrix() and M2 = Matrix(), one can write
 |      M3 = M1*M2. With r = Ray(), one can apply the M1 transform to a ray
 |      with r = M1*r
 |
 |  __str__(self)
 |      String description that allows the use of print(Matrix())
 |
 |  backwardConjugate(self)
 |      With an image at the back edge of the element,
 |      where is the object ? Distance before the element by
 |      which a ray must travel to reach the conjugate plane at
 |      the back of the element. A positive distance means the
 |      object is "distance" in front of the element (or to the
 |      left, or before).
 |
 |      M2 = M1*Space(distance)
 |      # M2.isImaging == True

```

## Examples

You can list several examples `python -m raytracing -l`:

```shell
All example code on your machine is found at: /somedirectory/on/your/machine
 1. ex01.py A single lens f = 50 mm, infinite diameter
 2. ex02.py Two lenses, infinite diameters
 3. ex03.py Finite-diameter lens
 4. ex04.py Aperture behind lens acting as Field Stop
 5. ex05.py Simple microscope system
 6. ex06.py Kohler illumination
 7. ex07.py Focussing through a dielectric slab
 8. ex08.py Virtual image at -f with object at f/2
 9. ex09.py Infinite telecentric 4f telescope
10. ex10.py Retrofocus $f_e$={0:.1f} cm, and BFL={1:.1f}
11. ex11.py Thick diverging lens computed from the Lensmaker equation
12. ex12.py Thick diverging lens built from individual elements
13. ex13.py Obtain the forward and backward conjugates
14. ex14.py Generic objectives
15. ex15.py Model Olympus objective LUMPlanFL40X
16. ex16.py Commercial doublets from Thorlabs and Edmund
17. ex17.py An optical system with vendor lenses
18. ex18.py Laser beam and vendor lenses
19. ex19.py Cavity round trip and calculated laser modes
.... and more complete examples at /somedirectory/on/your/machine
```

 You can run them all with `python -m raytracing -e all` (see them all below) to get a flavour of what is possible (note: in the US, it will give you a flavor of what is possible instead). Notice the command will tell you where the directory with all the tests is on your machine. **You will find more complete examples** in that [examples](https://github.com/DCC-Lab/RayTracing/tree/master/raytracing/examples) directory, distributed with the module.  For instance, `illuminator.py` to see a Kohler illuminator, and `invariant.py` to see an example of the role of lens diameters to determine the field of view.

![ex01](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex01.png)

![ex02](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex02.png)

![ex03](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex03.png)

![ex04](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex04.png)

![ex05](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex05.png)

![ex06](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex06.png)

![ex07](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex07.png)

![ex08](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex08.png)

![ex09](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex09.png)

![ex10](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex10.png)

![ex11](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex11.png)

![ex12](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex12.png)

![ex14](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex14.png)

![ex15](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex15.png)

![ex16.1](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex16.1.png)

![ex16.2](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex16.2.png)

![ex16.3](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex16.3.png)

![ex17](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex17.png)

![ex18](https://github.com/DCC-Lab/RayTracing/raw/master/README.assets/ex18.png)

## Known limitations

There are no known bugs in the actual calculations, but there are bugs or limitations in the display:

1. It is not easy to put several labels on a graph without any overlap. We are still working on it.
2. It is also not easy to figure out what "the right size" should be for an arrow head, the font, the position of a label, the size of the "ticks" on the aperture.
3. Labelling focal points with appropriate secondary labels should be possible, maybe a superscript?
4. The y-scale is not always set appropriately when the elements have infinite diameters: the rays will go beyond the element drawn on the figure.

## Licence

This code is provided under the [MIT License](./LICENSE).


> [!NOTE]
> This project is currently under development.
> - Version 1.0.0 will support Python ≥3.7 and is developed in the [release_prep branch](https://github.com/riga/law/tree/release_prep).
> - There will be a legacy branch with Python 2.7 and ≤3.6 support that, however, won't receive any new features.
> - The release is targetted for spring 2024.

Use law to build complex and large-scale task workflows.
It is build on top of [luigi](https://github.com/spotify/luigi) and adds abstractions for **run locations**, **storage locations** and **software environments**.
Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.

Key features:

- CLI with auto-completion and interactive status and dependency inspection.
- Remote targets with automatic retries and local caching
  - WebDAV, HTTP, Dropbox, SFTP, all WLCG protocols (srm, xrootd, dcap, gsiftp, webdav, ...)
- Automatic submission to batch systems from within tasks
  - HTCondor, LSF, gLite, ARC, Slurm, CMS-CRAB
- Environment sandboxing, configurable on task level
  - Docker, Singularity, Sub-Shells, Virutal envs

## Contents

- [First steps](#first-steps)
   - [Installation and dependencies](#installation-and-dependencies)
   - [Usage at CERN](#usage-at-cern)
   - [Overcomplete example config](#overcomplete-example-config)
- [Projects using law](#projects-using-law)
- [Examples](#examples)
- [Further topics](#further-topics)
   - [Auto completion on the command-line](#auto-completion-on-the-command-line)
- [Development](#development)
   - [Tests](#tests)
   - [Contributors](#contributors)

## First steps

### Installation and dependencies

Install via [pip](https://pypi.python.org/pypi/law)

```shell
pip install law
```

or [conda / (micro)mamba](https://anaconda.org/conda-forge/law)

```shel
conda install -c conda-forge law
```

If you plan to use remote targets, the (default) implementation also requires [gfal2](https://dmc-docs.web.cern.ch/dmc-docs/gfal2/gfal2.html) and [gfal2-python](https://pypi.python.org/pypi/gfal2-python) (optional) to be installed, either via pip or conda / (micro)mamba.

```shell
conda install -c conda-forge gfal2 gfal2-util
```

### Usage at CERN

See the [wiki](https://github.com/riga/law/wiki/Usage-at-CERN).

### Overcomplete example config

See [law.cfg.example](https://github.com/riga/law/tree/master/law.cfg.example).

## Projects using law

- CMS Di-Higgs Inference Tools:
  - Basis for statistical analysis for all Di-Higgs searches in CMS, starting at datacard-level
  - [repo](https://gitlab.cern.ch/hh/tools/inference), [docs](https://cms-hh.web.cern.ch/cms-hh/tools/inference/index.html)
- columnflow ([+ all analyses using it](https://github.com/columnflow/columnflow#projects-using-columnflow)):
  - Python based, fully automated, columnar framework, including job submission, resolution of systematics and ML pipelines, starting at NanoAOD-level with an optimized multi-threaded column reader
  - [repo](https://github.com/columnflow/columnflow), [docs](http://columnflow.readthedocs.io), [task structure](https://github.com/columnflow/columnflow/wiki#default-task-graph)
- CMS B-Tag SF Measurement:
  - Automated workflow for deriving shape-calibrating b-tag scale factors, starting at MiniAOD-level
  - [repo](https://github.com/cms-btv-pog/jet-tagging-sf)
- CMS Tau POG ML Tools:
  - Preprocessing pipeline for ML trainings in the TAU group
  - [repo](https://github.com/cms-tau-pog/TauMLTools)
- CMS HLT Config Parser:
  - Collects information from various databases (HLT, bril, etc.) and shows menus, triggers paths, filter names for configurable MC datasets or data runs
  - [repo](https://github.com/riga/cms-hlt-parser)
- RWTH-CMS Analysis Framework:
  - Basis for multiple CMS analyses ranging from Di-Higgs, to single Higgs and b-tag SF measurements, starting at NanoAOD-level and based on coffea processors
  - [repo](https://git.rwth-aachen.de/3pia/cms_analyses/common/-/tree/master/)
- CIEMAT-CMS Analysis Framework:
  - Python and RDataFrame based framework starting from NanoAOD and targetting multiple CMS analyses
  - [repo](https://gitlab.cern.ch/cms-phys-ciemat/nanoaod_base_analysis/)
- CMS 3D Z+jet 13TeV analysis
  - Analysis workflow management from NTuple production to final plots and fits
  - [repo](https://gitlab.etp.kit.edu/cverstege/zjet-analysis)
- NP-correction derivation tool
  - MC generation with Herwig and analysis of generated events with Rivet
  - [repo](https://github.com/HerrHorizontal/herwig-run)
- CMS SUSY Searches at DESY
  - Analysis framework for CMS SUSY searches going from custom NanoAODs -> NTuple production -> DNN-based inference -> final plots and fits
  - [repo](https://github.com/frengelk/Susy1LeptonAnalysis)
- Kingmaker (CMS Ntuple Production with CROWN)
  - Ntuple conversion from CMS nanoAOD to analysis Ntuples using the CROWN framework. Also includes the training of an event classifier on those ntuples.
  - [repo](https://github.com/KIT-CMS/KingMaker), [CROWN](https://github.com/KIT-CMS/CROWN)

If your project uses law but is not yet listed here, feel free to open a pull request or mention your project details in a new [issue](https://github.com/riga/law/issues/new?assignees=riga&labels=docs&template=register-project.md) and it will be added.

## Examples

All examples can be run either in a Jupyter notebook or a dedicated docker container.
For the latter, do

```shell
docker run -ti riga/law:example <example_name>
```

- [loremipsum](https://github.com/riga/law/tree/master/examples/loremipsum): The *hello world* example of law.
- [workflows](https://github.com/riga/law/tree/master/examples/workflows): Law workflows.
- [workflow_parameters](https://github.com/riga/law/tree/master/examples/workflows): Alternative way of parametrizing workflows with explicit branch parameters.
- [notebooks](https://github.com/riga/law/tree/master/examples/notebooks): Examples showing how to use and work with law in notebooks.
- [dropbox_targets](https://github.com/riga/law/tree/master/examples/dropbox_targets): Working with targets that are stored on Dropbox.
- [wlcg_targets](https://github.com/riga/law/tree/master/examples/wlcg_targets): Working with targets that are stored on WLCG storage elements (dCache, EOS, ...). TODO.
- [htcondor_at_vispa](https://github.com/riga/law/tree/master/examples/htcondor_at_vispa): HTCondor workflows at the [VISPA service](https://vispa.physik.rwth-aachen.de).
- [htcondor_at_cern](https://github.com/riga/law/tree/master/examples/htcondor_at_cern): HTCondor workflows at the CERN batch infrastructure.
- [CMS Crab at CERN](https://github.com/riga/law_example_CMSCrabWorkflows): CMS Crab workflows executed from lxplus at CERN.
- [sequential_htcondor_at_cern](https://github.com/riga/law/tree/master/examples/sequential_htcondor_at_cern): Continuation of the [htcondor_at_cern](https://github.com/riga/law/tree/master/examples/htcondor_at_cern) example, showing sequential jobs that eagerly start once jobs running previous requirements succeeded.
- [htcondor_at_naf](https://github.com/riga/law/tree/master/examples/htcondor_at_naf): HTCondor workflows at German [National Analysis Facility (NAF)](https://confluence.desy.de/display/IS/NAF+-+National+Analysis+Facility).
- [slurm_at_maxwell](https://github.com/riga/law/tree/master/examples/slurm_at_maxwell): Slurm workflows at the [Desy Maxwell cluster](https://confluence.desy.de/display/MXW/Maxwell+Cluster).
- [grid_at_cern](https://github.com/riga/law_example_WLCG): Workflows that run jobs and store data on the WLCG.
- [lsf_at_cern](https://github.com/riga/law/tree/master/examples/lsf_at_cern): LSF workflows at the CERN batch infrastructure.
- [docker_sandboxes](https://github.com/riga/law/tree/master/examples/docker_sandboxes): Environment sandboxing using Docker. TODO.
- [singularity_sandboxes](https://github.com/riga/law/tree/master/examples/singularity_sandboxes): Environment sandboxing using Singularity. TODO.
- [subshell_sandboxes](https://github.com/riga/law/tree/master/examples/subshell_sandboxes): Environment sandboxing using Subshells. TODO.
- [parallel_optimization](https://github.com/riga/law/tree/master/examples/parallel_optimization): Parallel optimization using [scikit optimize](https://scikit-optimize.github.io).
- [notifications](https://github.com/riga/law/tree/master/examples/notifications): Demonstration of slack and telegram task status notifications..
- [CMS Single Top Analysis](https://github.com/riga/law_example_CMSSingleTopAnalysis): Simple physics analysis using law.

## Further topics

### Auto completion on the command-line

**bash**

```shell
source "$( law completion )"
```

**zsh**

zsh is able to load and evaluate bash completion scripts via `bashcompinit`.
In order for `bashcompinit` to work, you should run `compinstall` to enable completion scripts:

```shell
autoload -Uz compinstall && compinstall
```

After following the instructions, these lines should be present in your `~/.zshrc`:

```shell
# The following lines were added by compinstall
zstyle :compinstall filename '~/.zshrc'

autoload -Uz +X compinit && compinit
autoload -Uz +X bashcompinit && bashcompinit
# End of lines added by compinstall
```

If this is the case, just source the law completion script (which internally enables `bashcompinit`) and you're good to go:

```shell
source "$( law completion )"
```

## Development

- Source hosted at [GitHub](https://github.com/riga/law)
- Report issues, questions, feature requests on [GitHub Issues](https://github.com/riga/law/issues)

### Tests

To run and test law, there are various docker `riga/law` images available on the [DockerHub](https://cloud.docker.com/u/riga/repository/docker/riga/law), corresponding to different OS and Python versions (based on [micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)).
Start them via

```shell
docker run -ti riga/law:<the_tag>
```

|      OS     | Python |                    Tags                   |
| ----------- | ------ | ----------------------------------------- |
| AlmaLinux 9 |   3.11 | a9-py311, a9-py3, a9, py311, py3, latest  |
| AlmaLinux 9 |   3.10 | a9-py310, py310                           |
| AlmaLinux 9 |    3.9 | a9-py39, py39                             |
| AlmaLinux 9 |    3.8 | a9-py38, py38                             |
| AlmaLinux 9 |    3.7 | a9-py37, py37                             |
| CentOS 8    |   3.11 | c8-py311, c8-py3, c8                      |
| CentOS 8    |   3.10 | c8-py310                                  |
| CentOS 8    |    3.9 | c8-py39                                   |
| CentOS 8    |    3.8 | c8-py38                                   |
| CentOS 8    |    3.7 | c8-py37                                   |
| CentOS 7    |   3.10 | c7-py310, c7-py3, c7                      |
| CentOS 7    |    3.9 | c7-py39                                   |
| CentOS 7    |    3.8 | c7-py38                                   |
| CentOS 7    |    3.7 | c7-py37                                   |
| CentOS 7    |    3.6 | c7-py36, py36 (removed soon)              |
| CentOS 7    |    2.7 | c7-py27, c7-py2, py27, py2 (removed soon) |

### Contributors

Marcel Rieger
💻 👀 🚧 📖

Peter Fackeldey
💻

Yannik Rath
💻

Jaime Leon Holgado
💻

Louis Moureaux
💻

Lukas Geiger
💻

Valentin Iovene
💻

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification.


cob_command_tools
===========

## ROS Distro Support

|         | Indigo | Jade | Kinetic | Melodic |
|:-------:|:------:|:----:|:-------:|:-------:|
| Branch  | [`indigo_dev`](https://github.com/ipa320/cob_command_tools/tree/indigo_dev) | [`indigo_dev`](https://github.com/ipa320/cob_command_tools/tree/indigo_dev) | [`indigo_dev`](https://github.com/ipa320/cob_command_tools/tree/indigo_dev) | [`indigo_dev`](https://github.com/ipa320/cob_command_tools/tree/indigo_dev) |
| Status  | supported | not supported | supported | supported |
| Version | [version](http://repositories.ros.org/status_page/ros_indigo_default.html?q=cob_command_tools) | [version](http://repositories.ros.org/status_page/ros_jade_default.html?q=cob_command_tools) | [version](http://repositories.ros.org/status_page/ros_kinetic_default.html?q=cob_command_tools) | [version](http://repositories.ros.org/status_page/ros_melodic_default.html?q=cob_command_tools) |

## Travis - Continuous Integration

Status: [![Build Status](https://app.travis-ci.com/ipa320/cob_command_tools.svg?branch=indigo_dev)](https://app.travis-ci.com/ipa320/cob_command_tools)

## ROS Buildfarm

|         | Indigo Source | Indigo Debian | Jade Source | Jade Debian | Kinetic Source | Kinetic Debian | Melodic Source | Melodic Debian |
|:-------:|:-------------:|:-------------:|:-----------:|:-----------:|:--------------:|:--------------:|:--------------:|:--------------:|
| cob_command_tools | [![not released](http://build.ros.org/buildStatus/icon?job=Isrc_uT__cob_command_tools__ubuntu_trusty__source)](http://build.ros.org/view/Isrc_uT/job/Isrc_uT__cob_command_tools__ubuntu_trusty__source/) | [![not released](http://build.ros.org/buildStatus/icon?job=Ibin_uT64__cob_command_tools__ubuntu_trusty_amd64__binary)](http://build.ros.org/view/Ibin_uT64/job/Ibin_uT64__cob_command_tools__ubuntu_trusty_amd64__binary/) | [![not released](http://build.ros.org/buildStatus/icon?job=Jsrc_uT__cob_command_tools__ubuntu_trusty__source)](http://build.ros.org/view/Jsrc_uT/job/Jsrc_uT__cob_command_tools__ubuntu_trusty__source/) | [![not released](http://build.ros.org/buildStatus/icon?job=Jbin_uT64__cob_command_tools__ubuntu_trusty_amd64__binary)](http://build.ros.org/view/Jbin_uT64/job/Jbin_uT64__cob_command_tools__ubuntu_trusty_amd64__binary/) | [![not released](http://build.ros.org/buildStatus/icon?job=Ksrc_uX__cob_command_tools__ubuntu_xenial__source)](http://build.ros.org/view/Ksrc_uX/job/Ksrc_uX__cob_command_tools__ubuntu_xenial__source/) | [![not released](http://build.ros.org/buildStatus/icon?job=Kbin_uX64__cob_command_tools__ubuntu_xenial_amd64__binary)](http://build.ros.org/view/Kbin_uX64/job/Kbin_uX64__cob_command_tools__ubuntu_xenial_amd64__binary/) | [![not released](http://build.ros.org/buildStatus/icon?job=Msrc_uB__cob_command_tools__ubuntu_bionic__source)](http://build.ros.org/view/Msrc_uB/job/Msrc_uB__cob_command_tools__ubuntu_bionic__source/) | [![not released](http://build.ros.org/buildStatus/icon?job=Mbin_uB64__cob_command_tools__ubuntu_bionic_amd64__binary)](http://build.ros.org/view/Mbin_uB64/job/Mbin_uB64__cob_command_tools__ubuntu_bionic_amd64__binary/) |

Installation instructions and tutorials can be found at http://wiki.ros.org/care-o-bot.


# a2ml - Automation of AutoML

[![CircleCI](https://img.shields.io/circleci/build/gh/augerai/a2ml/master)](https://circleci.com/gh/augerai/a2ml)
[![Join the chat](https://img.shields.io/gitter/room/augerai/a2ml.svg)](https://gitter.im/augerai/a2ml)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://raw.githubusercontent.com/augerai/a2ml/master/LICENSE)
[![Python](https://img.shields.io/pypi/pyversions/a2ml.svg)](https://pypi.org/project/a2ml/)
[![PyPI - A2ML Versions](https://img.shields.io/pypi/v/a2ml.svg)](https://pypi.org/project/a2ml/)

The A2ML ("Automate AutoML") project is a Python API and set of command line tools to automate Automated Machine Learning tools from multiple vendors. The intention is to provide a common API for all Cloud-oriented AutoML vendors.  Data scientists can then train their datasets against multiple AutoML models to get the best possible predictive model.  May the best "algorithm/hyperparameter search" win.  Full documentation for A2ML is available at [a2ml.org](http://a2ml.org)

## The PREDIT Pipeline

Every AutoML vendor has their own API to manage the datasets and create and
manage predictive models.  They are similar but not identical APIs.  But they share a
common set of stages:

* Importing data for training
* Train models with multiple algorithms and hyperparameters
* Evaluate model performance and choose one or more for deployment
* Deploy selected models
* Predict results with new data against deployed models
* Review performance of deployed models

Since ITEDPR is hard to remember we refer to this pipeline by its conveniently mnemonic anagram: "PREDIT" (French for "predict"). The A2ML project provides classes which implement this pipeline for various Cloud AutoML providers
and a command line interface that invokes stages of the pipeline.

## Setup

A2ML is distributed as a python package, so to install it:

```sh
$ pip install -U a2ml
```

It will install Auger provider.

To use Azure AutoML:

### Mac:

```sh
$ brew install libomp

```
#### For Mac OS High Sierra and below:

```sh
$ SKLEARN_NO_OPENMP=1 pip install "scikit-learn==0.21.3"
$ pip install "a2ml[azure]" --ignore-installed onnxruntime onnx nimbusml
```

### Linux:

```sh
$ apt-get update && apt-get -y install gcc g++ libgomp1
```

```sh
$ pip install "a2ml[azure]"
```

To use Google Cloud:

```sh
$ pip install "a2ml[google]"
```

To install everything including testing and server code:

```sh
$ pip install "a2ml[all]"
```

## Development

To release a new version the flow should be:

1. Change the `__version__` variable in `a2ml/__init__.py` to match what you want to release, minus the "v". By default it would be "<current-milestone>.dev0", for example "0.3.0.dev0". This ensures we don’t accidentally release a dev version to pypi.org. So for when we’re ready to release 0.3.0, the   `__version__` variable should simply be "0.3.0".

2. Commit and push the changes above.

```sh
git tag v<the-version> (for example: git tag v0.3.0)
git push --tags
```

3. verify circleci build passed and docker image tag exists:

```sh
pip install -U a2ml==0.3.0
docker pull augerai/a2ml:v0.3.0
```

4. Increment the `__version__` variable in `a2ml/__init__.py` to the next version in the current milestone. For example, "0.3.1.dev0"


# Jupyter kernels for the Spyder console

[![Windows status](https://github.com/spyder-ide/spyder-kernels/workflows/Windows%20tests/badge.svg)](https://github.com/spyder-ide/spyder-kernels/actions?query=workflow%3A%22Windows+tests%22)
[![Linux status](https://github.com/spyder-ide/spyder-kernels/workflows/Linux%20tests/badge.svg)](https://github.com/spyder-ide/spyder-kernels/actions?query=workflow%3A%22Linux+tests%22)
[![Macos status](https://github.com/spyder-ide/spyder-kernels/workflows/Macos%20tests/badge.svg)](https://github.com/spyder-ide/spyder-kernels/actions?query=workflow%3A%22Macos+tests%22)
[![codecov](https://codecov.io/gh/spyder-ide/spyder-kernels/branch/master/graph/badge.svg)](https://codecov.io/gh/spyder-ide/spyder-kernels/branch/master)

Package that provides Jupyter kernels for use with the consoles of Spyder, the
Scientific Python Development Environment.

These kernels can be launched either through Spyder itself or in an independent
Python session, and allow for interactive or file-based execution of Python
code inside Spyder.

To learn about creating, connecting to and using these kernels with the Spyder
console, please read our [documentation](https://docs.spyder-ide.org/current/panes/ipythonconsole.html).

For advice on managing packages and environments with `spyder-kernels`, please read this
[FAQ](http://docs.spyder-ide.org/current/faq.html#using-existing-environment) in our docs.

## Installation

To install this package, you can use either the ``pip`` or ``conda`` package
managers, as follows:

Using conda (the recommended way!):

```
conda install spyder-kernels
```

Using pip:

```
pip install spyder-kernels
```

## Dependencies

This project depends on:

* [ipykernel](https://github.com/ipython/ipykernel/)
* [cloudpickle](https://github.com/cloudpipe/cloudpickle)
* [wurlitzer](https://github.com/minrk/wurlitzer) (only on Linux and macOS).

## Changelog

Visit our [CHANGELOG](CHANGELOG.md) file to know more about our new features
and improvements.

## Development and contribution

To start contributing to this project you can execute

```
pip install -e .
```

in your git clone and then test your changes in Spyder. We follow PEP8 and
PEP257 style guidelines.

## Sponsors

Spyder and its subprojects are funded thanks to the generous support of

[![Quansight](https://user-images.githubusercontent.com/16781833/142477716-53152d43-99a0-470c-a70b-c04bbfa97dd4.png)](https://www.quansight.com/)[![Numfocus](https://i2.wp.com/numfocus.org/wp-content/uploads/2017/07/NumFocus_LRG.png?fit=320%2C148&ssl=1)](https://numfocus.org/)

and the donations we have received from our users around the world through [Open Collective](https://opencollective.com/spyder/):

[![Sponsors](https://opencollective.com/spyder/sponsors.svg)](https://opencollective.com/spyder#support)


# grahampugh-recipes

A few recipes not found elsewhere. To use them, enter the following command:

    autopkg repo-add grahampugh/recipes-yaml

Note that some recipes in this repo are designed to help package up software that is not publicly available. As such, those recipes have no `.download` recipe. Those recipes have their own `README` files to give further instruction.

## Wait, these are all yaml files!

I write my recipes as `yaml` files, and then convert them to `plist` format when I'm done. I find constructing the recipes much nicer in `yaml` format, and it ensures a consistent `plist` structure (e.g. alphabetised input keys).

If, on the other hand, you are interested in this technique, take a look at https://github.com/grahampugh/plist-yaml-plist


![SolveBio Python Package](https://github.com/solvebio/solvebio-python/workflows/SolveBio%20Python%20Package/badge.svg)

SolveBio Python Client
======================

This is the SolveBio Python package and command-line interface (CLI).
This module is tested against Python 2.7, 3.6, 3.7, 3.8, 3.10, PyPy and PyPy3.

Developer documentation is available at [docs.solvebio.com](https://docs.solvebio.com). For more information about SolveBio visit [www.solvebio.com](https://www.solvebio.com).

Installation & Setup
--------------------

Install `solvebio` using `pip`:

    pip install solvebio

For interactive use, we recommend installing `IPython` and `gnureadline`:

    pip install ipython
    pip install gnureadline

To log in, type:

    solvebio login

Enter your SolveBio credentials and you should be good to go!

Install from Git
----------------

    pip install -e git+https://github.com/solvebio/solvebio-python.git#egg=solve

Development
-----------

    git clone https://github.com/solvebio/solvebio-python.git
    cd solve-python/
    python setup.py develop

Or install `tox` and run:

    pip install tox
    tox

Releasing
---------

You will need to [configure Twine](https://twine.readthedocs.io/en/latest/#installation) in order to push to PyPI.

Maintainers can release solvebio-python to PyPI with the following steps:

    bumpversion <major|minor|patch>
    git push --tags
    make changelog
    make release

Support
-------

Developer documentation is available at [docs.solvebio.com](https://docs.solvebio.com).

If you experience problems with this package, please [create a GitHub Issue](https://github.com/solvebio/solvebio-python/issues).

For all other requests, please [email SolveBio Support](mailto:support@solvebio.com).

Configuring the Client
-------

The SolveBio python client can be configured by setting system environment variables.
Supported environment variables are:

`SOLVEBIO_API_HOST`
- The URL of the target API backend.
If not specified the value from the local credentials file will be used.

`SOLVEBIO_ACCESS_TOKEN`
- The OAuth2 access token for authenticating with the API.

`SOLVEBIO_API_KEY`
- The API Key to use for authenticating with the API.

The lookup order for credentials is:
1. Access Token
2. API Key
3. Local Credentials file

`SOLVEBIO_LOGLEVEL`
- The log level at which to log messages.
If not specified the default log level will be WARN.

`SOLVEBIO_LOGFILE`
- The file in which to write log messages.
If the file does not exist it will be created.
If not specified '~/.solvebio/solvebio.log' will be used by default.

`SOLEVBIO_RETRY_ALL`
- Flag for enabling aggressive retries for failed requests to the API.
When truthy, the client will attempt to retry a failed request regardless of the type of operation.
This includes idempotent and nonidempotent operations:
"HEAD", "GET", "PUT", "POST", "PATCH", "DELETE", "OPTIONS", "TRACE"
If this value is not set it will default to false and retries will only be enabled for idempotent operations.


# intern

[![PyPI version](https://img.shields.io/pypi/v/intern?style=for-the-badge)](https://pypi.org/project/intern/)
[![Python 3.6/3.7 Tests](https://img.shields.io/github/actions/workflow/status/jhuapl-boss/intern/tests.yml?style=for-the-badge)](https://github.com/jhuapl-boss/intern/actions?query=workflow%3A%22Test+Python+Package%22)

**intern** (Integrated Toolkit for Extensible and Reproducible Neuroscience) is a Python 3 module that enables big-data neuroscience. Currently, it provides an interface to common big-data neuroimaging databases such as [BossDB](https://bossdb.org), CloudVolume, DVID, and other standard formats.

## Getting Started

To quickly get started with intern, check out the wiki: [https://github.com/jhuapl-boss/intern/wiki](https://github.com/jhuapl-boss/intern/wiki)

## Documentation

Full detailed documentation can be found here: [https://jhuapl-boss.github.io/intern/](https://jhuapl-boss.github.io/intern/)

## Installation

-   It's always a good idea to use virtualenv to isolate your work from your system Python installation:

-   Using [virtualenv](https://virtualenv.pypa.io/en/stable/):

```shell
virtualenv intern
. intern/bin/activate
```

-   Using [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/):

```shell
mkvirtualenv intern
```

-   (Preferred) Install via pypi

```shell
pip install intern
```

-   Install via git

Clone the repository from https://github.com/jhuapl-boss/intern and run
`pip install -r requirements.txt` from the repository's location on your
system.

Add `<repository location>` to your `PYTHONPATH`.

For example, on a \*nix system, if intern was cloned to ~/intern:

`export PYTHONPATH=$PYTHONPATH:~/intern`

> **For Python 2 support, you will need to install intern v0.10.0 or earlier.**

### Optional Dependencies
To install depedencies required to use the [cloud-volume](https://github.com/seung-lab/cloud-volume) remote, run the command:

```shell
pip install intern[cloudvolume]
```

## Contributing

Please submit bug reports, or get in touch using GitHub Issues.

## Citation

If you find this library useful to your work, please consider citing the following:

https://www.biorxiv.org/content/10.1101/2020.05.15.098707v1

```
@article{intern,
	doi = {10.1101/2020.05.15.098707},
	url = {https://www.biorxiv.org/content/10.1101/2020.05.15.098707v1},
	year = 2020,
	month = {may},
	publisher = {BiorXiv},
	author = {Matelsky, Jordan K and Rodriguez, Luis and Xenes, Daniel and Gion, Timothy and Hider Jr., Robert and Wester, Brock and Gray-Roncal, William},
	title = {{intern: Integrated Toolkit for Extensible and Reproducible Neuroscience}},
	journal = {BiorXiv}
}
```

## The Boss Legal Notes

Use or redistribution of the Boss system in source and/or binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code or binary forms must adhere to the terms and conditions of any applicable software licenses.
2. End-user documentation or notices, whether included as part of a redistribution or disseminated as part of a legal or scientific disclosure (e.g. publication) or advertisement, must include the following acknowledgement: The Boss software system was designed and developed by the Johns Hopkins University Applied Physics Laboratory (JHU/APL).
3. The names "The Boss", "JHU/APL", "Johns Hopkins University", "Applied Physics Laboratory", "MICrONS", or "IARPA" must not be used to endorse or promote products derived from this software without prior written permission. For written permission, please contact BossAdmin@jhuapl.edu.
4. This source code and library is distributed in the hope that it will be useful, but is provided without any warranty of any kind.


## CVMFS Singularly Image Repository

To add images to the repository for use on OSG, follow the instructions on the
documentation pages below:

 - [Containers Overview](https://portal.osg-htc.org/documentation/htc_workloads/using_software/containers-singularity/)
 - [Containers - Docker](https://portal.osg-htc.org/documentation/htc_workloads/using_software/containers-docker/)


[![Test](https://github.com/nginx-proxy/nginx-proxy/actions/workflows/test.yml/badge.svg)](https://github.com/nginx-proxy/nginx-proxy/actions/workflows/test.yml)
[![GitHub release](https://img.shields.io/github/v/release/nginx-proxy/nginx-proxy)](https://github.com/nginx-proxy/nginx-proxy/releases)
![nginx 1.27.0](https://img.shields.io/badge/nginx-1.27.0-brightgreen.svg)
[![Docker Image Size](https://img.shields.io/docker/image-size/nginxproxy/nginx-proxy?sort=semver)](https://hub.docker.com/r/nginxproxy/nginx-proxy "Click to view the image on Docker Hub")
[![Docker stars](https://img.shields.io/docker/stars/nginxproxy/nginx-proxy.svg)](https://hub.docker.com/r/nginxproxy/nginx-proxy "DockerHub")
[![Docker pulls](https://img.shields.io/docker/pulls/nginxproxy/nginx-proxy.svg)](https://hub.docker.com/r/nginxproxy/nginx-proxy "DockerHub")

nginx-proxy sets up a container running nginx and [docker-gen](https://github.com/nginx-proxy/docker-gen). docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped.

See [Automated Nginx Reverse Proxy for Docker](http://jasonwilder.com/blog/2014/03/25/automated-nginx-reverse-proxy-for-docker/) for why you might want to use this.

### Usage

To run it:

```console
docker run --detach \
    --name nginx-proxy \
    --publish 80:80 \
    --volume /var/run/docker.sock:/tmp/docker.sock:ro \
    nginxproxy/nginx-proxy:1.6
```

Then start any containers (here an nginx container) you want proxied with an env var `VIRTUAL_HOST=subdomain.yourdomain.com`

```console
docker run --detach \
    --name your-proxied-app \
    --env VIRTUAL_HOST=foo.bar.com \
    nginx
```

Provided your DNS is setup to resolve `foo.bar.com` to the host running nginx-proxy, a request to `http://foo.bar.com` will then be routed to a container with the `VIRTUAL_HOST` env var set to `foo.bar.com` (in this case, the **your-proxied-app** container).

The containers being proxied must :

- [expose](https://docs.docker.com/engine/reference/run/#expose-incoming-ports) the port to be proxied, either by using the `EXPOSE` directive in their `Dockerfile` or by using the `--expose` flag to `docker run` or `docker create`.
- share at least one Docker network with the nginx-proxy container: by default, if you don't pass the `--net` flag when your nginx-proxy container is created, it will only be attached to the default bridge network. This means that it will not be able to connect to containers on networks other than bridge.

Note: providing a port number in `VIRTUAL_HOST` isn't suported, please see [virtual ports](https://github.com/nginx-proxy/nginx-proxy/tree/main/docs#virtual-ports) or [custom external HTTP/HTTPS ports](https://github.com/nginx-proxy/nginx-proxy/tree/main/docs#custom-external-httphttps-ports) depending on what you want to achieve.

### Image variants

The nginx-proxy images are available in two flavors.

#### Debian based version

This image is based on the nginx:mainline image, itself based on the debian slim image.

```console
docker pull nginxproxy/nginx-proxy:1.6
```

#### Alpine based version (`-alpine` suffix)

This image is based on the nginx:alpine image.

```console
docker pull nginxproxy/nginx-proxy:1.6-alpine
```

#### :warning: a note on `latest` and `alpine`:

It is not recommended to use the `latest` (`nginxproxy/nginx-proxy`, `nginxproxy/nginx-proxy:latest`) or `alpine` (`nginxproxy/nginx-proxy:alpine`) tag for production setups.

[Those tags point](https://hub.docker.com/r/nginxproxy/nginx-proxy/tags) to the latest commit in the `main` branch. They do not carry any promise of stability, and using them will probably put your nginx-proxy setup at risk of experiencing uncontrolled updates to non backward compatible versions (or versions with breaking changes). You should always specify the version you want to use explicitly to ensure your setup doesn't break when the image is updated.

### Additional documentation

Please check the [docs section](https://github.com/nginx-proxy/nginx-proxy/tree/main/docs).


===================
django-crispy-forms
===================

.. image:: https://github.com/django-crispy-forms/django-crispy-forms/actions/workflows/main.yml/badge.svg

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/psf/black

.. image:: http://codecov.io/github/django-crispy-forms/django-crispy-forms/coverage.svg?branch=main
   :target: http://codecov.io/github/django-crispy-forms/django-crispy-forms?branch=main

The best way to have Django_ DRY forms. Build programmatic reusable layouts out of components, having full control of the rendered HTML without writing HTML in templates. All this without breaking the standard way of doing things in Django, so it plays nice with any other form application.

`django-crispy-forms` supports Django 4.2+ with Python 3.8+.

Looking for Bootstrap 5 support? See the `crispy-bootstrap5 package`_.

.. _`crispy-bootstrap5 package`: https://github.com/django-crispy-forms/crispy-bootstrap5

The application mainly provides:

* A filter named ``|crispy`` that will render elegant div based forms. Think of it as the built-in methods: ``as_table``, ``as_ul`` and ``as_p``. You cannot tune up the output, but it is easy to start using it.
* A tag named ``{% crispy %}`` that will render a form based on your configuration and specific layout setup. This gives you amazing power without much hassle, helping you save tons of time.

Django-crispy-forms supports several frontend frameworks, such as Twitter `Bootstrap`_ (versions 2, 3, and 4), `tailwind`_, `Bulma`_ and Foundation. You can also easily adapt your custom company's one, creating your own, `see the docs`_ for more information. You can easily switch among them using ``CRISPY_TEMPLATE_PACK`` setting variable.

.. _`tailwind`: https://github.com/django-crispy-forms/crispy-tailwind
.. _`Bootstrap`: https://getbootstrap.com
.. _`Bulma`: https://github.com/ckrybus/crispy-bulma
.. _`see the docs`: https://django-crispy-forms.readthedocs.io

Authors
=======

django-crispy-forms is the new django-uni-form. django-uni-form was an application created by `Daniel Greenfeld`_ that I led since version 0.8.0. The name change tries to better explain the purpose of the application, which changed in a significant way since its birth.

If you are upgrading from django-uni-form, we have `instructions`_ for helping you.

* Lead developer: `Miguel Araujo`_

.. _`Daniel Greenfeld`: https://github.com/pydanny
.. _`Miguel Araujo`: https://github.com/maraujop
.. _`instructions`: https://django-crispy-forms.readthedocs.io/en/latest/install.html#moving-from-django-uni-form-to-django-crispy-forms

Example
=======

This is a teaser of what you can do with latest django-crispy-forms. `Find here the gist`_ for generating this form:

.. image:: http://i.imgur.com/LSREg.png

.. _`Find here the gist`: https://gist.github.com/1838193

Documentation
=============

For extensive documentation see the ``docs`` folder or `read it on readthedocs`_

.. _`read it on readthedocs`: https://django-crispy-forms.readthedocs.io/en/latest/index.html

Special thanks
==============

* To Daniel Feldroy (`@pydanny`_) for his support, time and the opportunity given to me to do this.
* The name of the project was suggested by the fantastic Audrey Feldroy (`@audreyfeldroy`_)
* To Kenneth Love (`@kennethlove`_) for creating django-uni-form-contrib from which bootstrap template pack was started.

.. _`@audreyfeldroy`: https://github.com/audreyfeldroy
.. _`@pydanny`: https://github.com/pydanny
.. _`@kennethlove`: https://github.com/kennethlove

.. _Django: http://djangoproject.com


# Lektor

[![Tests master](https://github.com/lektor/lektor/workflows/Tests%20master/badge.svg)](https://github.com/lektor/lektor/actions?query=workflow%3A%22Tests+master%22)
[![Code Coverage](https://codecov.io/gh/lektor/lektor/branch/master/graph/badge.svg)](https://codecov.io/gh/lektor/lektor)
[![PyPI version](https://badge.fury.io/py/Lektor.svg)](https://pypi.org/project/Lektor/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/Lektor.svg)](https://pypi.org/project/Lektor/)
[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Join the chat at https://gitter.im/lektor/lektor](https://badges.gitter.im/lektor/lektor.svg)](https://gitter.im/lektor/lektor?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Lektor is a static website generator. It builds out an entire project
from static files into many individual HTML pages and has a built-in
admin UI and minimal desktop app.

<img src="https://raw.githubusercontent.com/lektor/lektor-assets/master/screenshots/admin.png" width="100%">

To see how it works look at the top-level `example/` folder, which contains
a showcase of the wide variety of Lektor's features.

For a more complete example look at the [lektor/lektor-website](https://github.com/lektor/lektor-website)
repository, which contains the sourcecode for the official lektor website.

## How do I use this?

For installation instructions head to the official documentation:

- [Installation](https://www.getlektor.com/docs/installation/)
- [Quickstart](https://www.getlektor.com/docs/quickstart/)

## Want to develop on Lektor?

This gets you started (assuming you have Python, pip, npm, and pre-commit
installed):

```bash
$ git clone https://github.com/lektor/lektor
$ cd lektor
$ python -m venv _venv
$ . _venv/bin/activate

# pip>=21.3 is required for PEP 610 support
$ pip install -U "pip>=21.3"

$ pip install --editable .

# If you plan on committing:
$ pre-commit install

# Run the Lektor server
$ export LEKTOR_DEV=1
$ cp -r example example-project
$ lektor --project example-project server
```

If you want to run the test suite (you'll need tox installed):

```sh
$ tox
```


# Archery

 SQL 审核查询平台

[![Django CI](https://github.com/hhyo/Archery/actions/workflows/django.yml/badge.svg)](https://github.com/hhyo/Archery/actions/workflows/django.yml)
[![Release](https://img.shields.io/github/release/hhyo/archery.svg)](https://github.com/hhyo/archery/releases/)
[![codecov](https://codecov.io/gh/hhyo/archery/branch/master/graph/badge.svg)](https://codecov.io/gh/hhyo/archery)
[![version](https://img.shields.io/pypi/pyversions/django)](https://img.shields.io/pypi/pyversions/django/)
[![version](https://img.shields.io/badge/django-4.1-brightgreen.svg)](https://docs.djangoproject.com/zh-hans/4.1/)
[![Publish Docker image](https://github.com/hhyo/Archery/actions/workflows/docker-image.yml/badge.svg)](https://github.com/hhyo/Archery/actions/workflows/docker-image.yml)
[![docker_pulls](https://img.shields.io/docker/pulls/hhyo/archery.svg)](https://hub.docker.com/r/hhyo/archery/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](http://github.com/hhyo/archery/blob/master/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

[文档](https://archerydms.com/) | [FAQ](https://github.com/hhyo/archery/wiki/FAQ) | [Releases](https://github.com/hhyo/archery/releases/)

![](https://github.com/hhyo/Archery/wiki/images/dashboard.png)

功能清单
====

| 数据库        | 查询 | 审核 | 执行 | 备份 | 数据字典 | 慢日志 | 会话管理 | 账号管理 | 参数管理 | 数据归档 |
|------------| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MySQL      | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ |
| MsSQL      | √ | × | √ | × | √ | × | × | × | × | × |
| Redis      | √ | × | √ | × | × | × | × | × | × | × |
| PgSQL      | √ | × | √ | × | × | × | × | × | × | × |
| Oracle     | √ | √ | √ | √ | √ | × | √  | × | × | × |
| MongoDB    | √ | √  | √  | × | × | × | √  | √ | × | × |
| Phoenix    | √ | ×  | √  | × | × | × | × | × | × | × |
| ODPS       | √ | ×  | ×  | × | × | × | × | × | × | × |
| ClickHouse | √ | √  | √  | × | × | × | × | × | × | × |
| Cassandra  | √ | ×  | √  | × | × | × | × | × | × | × |
| Doris      | √ | ×  | √  | × | × | × | × | × | × | × |

快速开始
===============
### 系统体验
[在线体验](https://demo.archerydms.com)

| 账号 | 密码 |
| --- | --- |
| archer | archer |

### Docker
参考 https://github.com/hhyo/archery/wiki/docker

手动安装
===============
[部署说明](https://github.com/hhyo/archery/wiki/manual)

运行测试
===============
```
python manage.py test -v 3
```

依赖清单
===============
### 框架
- [Django](https://github.com/django/django)
- [Bootstrap](https://github.com/twbs/bootstrap)
- [jQuery](https://github.com/jquery/jquery)
### 前端组件
- 菜单栏 [metisMenu](https://github.com/onokumus/metismenu)
- 主题 [sb-admin-2](https://github.com/BlackrockDigital/startbootstrap-sb-admin-2)
- 编辑器 [ace](https://github.com/ajaxorg/ace)
- SQL美化 [sql-formatter](https://github.com/zeroturnaround/sql-formatter)
- 表格  [bootstrap-table](https://github.com/wenzhixin/bootstrap-table)
- 表格编辑  [bootstrap-editable](https://github.com/vitalets/x-editable)
- 下拉菜单 [bootstrap-select](https://github.com/snapappointments/bootstrap-select)
- 文件上传 [bootstrap-fileinput](https://github.com/kartik-v/bootstrap-fileinput)
- 时间选择  [bootstrap-datetimepicker](https://github.com/smalot/bootstrap-datetimepicker)
- 日期选择  [daterangepicker](https://github.com/dangrossman/daterangepicker)
- 开关  [bootstrap-switch](https://github.com/Bttstrp/bootstrap-switch)
- Markdown展示  [marked](https://github.com/markedjs/marked)
### 服务端
- 队列任务 [django-q](https://github.com/Koed00/django-q)
- MySQL Connector [mysqlclient-python](https://github.com/PyMySQL/mysqlclient-python)
- MsSQL Connector [pyodbc](https://github.com/mkleehammer/pyodbc)
- Redis Connector [redis-py](https://github.com/andymccurdy/redis-py)
- PostgreSQL Connector [psycopg2](https://github.com/psycopg/psycopg2)
- Oracle Connector [cx_Oracle](https://github.com/oracle/python-cx_Oracle)
- MongoDB Connector [pymongo](https://github.com/mongodb/mongo-python-driver)
- Phoenix Connector [phoenixdb](https://github.com/lalinsky/python-phoenixdb)
- ODPS Connector [pyodps](https://github.com/aliyun/aliyun-odps-python-sdk)
- ClickHouse Connector [clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver)
- SQL解析/切分/类型判断 [sqlparse](https://github.com/andialbrecht/sqlparse)
- MySQL Binlog解析/回滚 [python-mysql-replication](https://github.com/noplay/python-mysql-replication)
- LDAP [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap)
- 序列化 [simplejson](https://github.com/simplejson/simplejson)
- 时间处理 [python-dateutil](https://github.com/paxan/python-dateutil)
### 功能依赖
- 可视化 [pyecharts](https://github.com/pyecharts/pyecharts)
- MySQL审核/执行/备份 [goInception](https://github.com/hanchuanchuan/goInception)|[inception](https://github.com/hhyo/inception)
- MySQL索引优化 [SQLAdvisor](https://github.com/Meituan-Dianping/SQLAdvisor)
- SQL优化/压缩 [SOAR](https://github.com/XiaoMi/soar)
- My2SQL [my2sql](https://github.com/liuhr/my2sql)
- 表结构同步 [SchemaSync](https://github.com/hhyo/SchemaSync)
- 慢日志解析展示 [pt-query-digest](https://www.percona.com/doc/percona-toolkit/3.0/pt-query-digest.html)|[aquila_v2](https://github.com/thinkdb/aquila_v2)
- 大表DDL [gh-ost](https://github.com/github/gh-ost)|[pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/3.0/pt-online-schema-change.html)
- MyBatis XML解析 [mybatis-mapper2sql](https://github.com/hhyo/mybatis-mapper2sql)
- RDS管理 [aliyun-openapi-python-sdk](https://github.com/aliyun/aliyun-openapi-python-sdk)
- 数据加密 [django-mirage-field](https://github.com/luojilab/django-mirage-field)

贡献代码
===============
可查阅主页的开发计划以及依赖清单，在对应Issues中回复认领，或者直接提交PR，感谢你对Archery的贡献

贡献包括但不限于以下方式：
- [Wiki文档](https://github.com/hhyo/Archery/wiki)（开放编辑）
- Bug修复
- 新功能提交
- 代码优化
- 测试用例完善

交流反馈
===============
- 使用咨询、需求沟通：[Discussions](https://github.com/hhyo/Archery/discussions)
- Bug提交：[Issues](https://github.com/hhyo/archery/issues)

致谢
===============
- [archer](https://github.com/jly8866/archer) Archery 项目是基于 archer 二次开发而来
- [goInception](https://github.com/hanchuanchuan/goInception) 一个集审核、执行、备份及生成回滚语句于一身的MySQL运维工具
- [JetBrains Open Source](https://www.jetbrains.com/zh-cn/opensource/?from=archery) 为项目提供免费的 IDE 授权
  [](https://www.jetbrains.com/opensource/)


# Jarvis

[![Build Status](https://travis-ci.org/sukeesh/Jarvis.svg?branch=master)](https://travis-ci.org/sukeesh/Jarvis) [![Join the chat at https://gitter.im/Sukeesh_Jarvis/Lobby](https://badges.gitter.im/Sukeesh_Jarvis/Lobby.svg)](https://gitter.im/Sukeesh_Jarvis/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

A Personal Assistant for Linux, MacOS and Windows

![Jarvis](http://i.imgur.com/xZ8x9ES.jpg)

Jarvis is a simple personal assistant for Linux, MacOS and Windows which works on the command line. He can talk to you if you enable his voice. He can tell you the weather, he can find restaurants and other places near you. He can do some great stuff for you.

## 20 Different Tasks That Jarvis Can Do For You ("Plugin Name" Included)
1. Suggest a random activity for you to do if you're bored ("activity", "bored")
2. Give ideas for what to draw, watch, do, listen to (“prompt”, “top_media”, “taste dive”, “mood music”)
3. Receive up-to-date information about different sports such as team ranking, match starts times, player stats ( “basketball” , “cricket”, “soccer”, “tennis”)
4. Play games (“blackjack”, “connect_four”, “guess_number_game”, “hangman”, “rockpaperscissors”. “roulette”, “tic_tac_toe”,”word_game”, “wordle”)
5. Help you stay fit by giving food nutrition facts, recipes, workout programs, and health trackers (“bmi”, “bmr”, “calories”, “food recipe”, “fruit”, “fruit nutrition”, “workout”)
6. Teach you how to make a cocktail (“cocktail”, “drink”)
7. Generate random lists, numbers, passwords (“random list”, “random number”, “random password”)
8. Perform conversions for binary numbers, money, hexadecimal numbers, distance, mass, speed, strings, temp, time (“binary”, “currencyconv”, “hex”, “lengthconv”, “massconv”, “speedconv”, “string_convert”, “tempconv”, “timveconv”)
9. Take pictures (“open camera”, “screencapture”)
10. Tell you different specifications for your current computer system (“battery”,  “cat his”, “dns forward”, “dns reverse”, “hostinfo”, “ip”, “scan_network”, “speedtest”, “os”, “check ram”, “systeminfo”)
11. Manage files on your computer (“file manage”, “file organize”
12. Upload, edit, and convert images (“imgur”, “image to pdg”,  “image compressor”)
13. Convert webpages to PDF or PDFs to images (“htmptopdg”, “pdf to images”)
14. Tell jokes (“dadjoke”, “joke daily”, “joke chuck”, “joke”)
15. Perform arithmetic and calculations (“calculate”, “factor”, “solve”, “equations”, “plot”, “matrix add”)
16. Generate a QR code to attach to a URL (“qr”)
17. Check the weather (“weather report”)
18. Provide language translations (“translate”)
19. Tell random facts (“fact”, “cat fact”)
20. Display stock market information (“stock”, “cryptotracker”)

## Getting Started With Installation

In order to install Jarvis, follow these steps:
1. Clone [this repository](https://github.com/sukeesh/Jarvis.git) with `git clone https://github.com/sukeesh/Jarvis.git`
2. Run the command `python installer` (or `python3 installer` if that doesn't work) from the terminal.

Run **Jarvis** from anywhere by command `jarvis`, or `./jarvis` from within the project directory to start up Jarvis!

You can start by typing `help` within the Jarvis command line to check what Jarvis can do for you.

### Frequently encountered issues
**Question**: When I run Jarvis, it shows an error relating to module not found

**Platform**: Windows

**Solution 1**: Uninstall and/or install the module package

Example:

Error: `ImportError: DLL load failed while importing win32api: The specified module could not be found.`

Solution:

`pip uninstall pywin32`

`pip install pywin32` or `conda instapll pywin32`

**Solution 2**: add the package to your environment variables system PATH.

**Question**: After cloning the repo in terminal it gives an error when running python3 installer saying please install virtual environemnt
**Solution**:
- Install virtual env using this command "python3 -m pip install virtualenv"
- OR: On Linux use package manager (e.g. Ubuntu sudo apt install python3-venv)
- Restart Installer

If you find other issues and/or have found solutions to them on any platform, please consider adding to this list!

## Youtube Video Showing Jarvis

[Click here](https://www.youtube.com/watch?v=PR-nxqmG3V8)

## Contributing

Check out our [CONTRIBUTING.md](CONTRIBUTING.md) to learn how you can contribute!

### QuickStart: Create a new feature (plugin)

Create new file custom/hello_world.py

```
from plugin import plugin

@plugin("helloworld")
def helloworld(jarvis, s):
    """Repeats what you type"""
    jarvis.say(s)
```

Check it out!
```
./jarvis
Jarvis' sound is by default disabled.
In order to let Jarvis talk out loud type: enable sound
Type 'help' for a list of available actions.

~> Hi, what can I do for you?
helloworld Jarvis is cool!
jarvis is cool
```

### Plugins

[Click here](doc/PLUGINS.md) to learn more about plugins.

### Creating a test

Creating a test is optional but never a bad idea ;).

[Click here](doc/TESTING.md) to learn more about testing.

### How to run tests:

 Run `test.sh`
 ```bash
 ./test.sh
 ```
## Optional Dependencies

- Any pyttsx3 text-to-speech engine (``sapi5, nsss or espeak``) for Jarvis to talk out loud (e.g. Ubuntu do ``sudo apt install espeak``)
- Portaudio + python-devel packages for voice control
- ``notify-send`` on Linux if you want to receive *nice* and desktop-notification instead of *ugly* pop up windows (e.g. Ubuntu do ``sudo apt install libnotify-bin``)
- ``ffmpeg`` if you want ``music`` to download songs as .mp3 instead of .webm

## Docker

Run with docker (docker needs to be installed and running):

```
[sudo] make build_docker
[sudo] make run_docker
```

## Authors

 **sukeesh**

See also the list of [contributors](https://github.com/sukeesh/Jarvis/graphs/contributors) who have participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.