Ansible Modules Hashivault
==========================

Ansible modules for Hashicorp Vault.

.. image:: https://img.shields.io/pypi/v/ansible-modules-hashivault.svg
   :alt: Latest version
   :target: https://pypi.python.org/pypi/ansible-modules-hashivault/
.. image:: https://pepy.tech/badge/ansible-modules-hashivault/month
   :alt: Downloads
   :target: https://pepy.tech/project/ansible-modules-hashivault
.. image:: https://github.com/TerryHowe/ansible-modules-hashivault/actions/workflows/test.yml/badge.svg?branch=main
   :alt: CI
   :target: https://github.com/TerryHowe/ansible-modules-hashivault/actions/workflows/test.yml
.. image:: https://img.shields.io/badge/License-MIT-yellow.svg
   :alt: License: MIT
   :target: https://opensource.org/licenses/MIT

Install this Ansible module:

* via ``pip``:

::

  pip install ansible-modules-hashivault

* via ``ansible-galaxy`` (requires ``hvac>=2.1.0``):

::

  ansible-galaxy install 'git+https://github.com/TerryHowe/ansible-modules-hashivault.git'

..

  Note: The ``hashicorp`` lookup plugin does not work with this last install method (`ansible/ansible#28770 <https://github.com/ansible/ansible/issues/28770>`_).
  You can fallback to the build-in lookup plugin: `hashi_vault <https://docs.ansible.com/ansible/latest/plugins/lookup/hashi_vault.html>`_

In most cases the Hashicorp Vault modules should be run on localhost.

Environmental Variables
-----------------------

The following variables need to be exported to the environment where you run ansible
in order to authenticate to your HashiCorp Vault instance:

  * `VAULT_ADDR`: url for vault
  * `VAULT_SKIP_VERIFY=true`: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
  * `VAULT_AUTHTYPE`: authentication type to use: `token`, `userpass`, `github`, `ldap`, `approle`
  * `VAULT_LOGIN_MOUNT_POINT`: mount point for login defaults to auth type
  * `VAULT_TOKEN`: token for vault
  * `VAULT_ROLE_ID`: (required by `approle`)
  * `VAULT_SECRET_ID`: (required by `approle`)
  * `VAULT_USER`: username to login to vault
  * `VAULT_PASSWORD`: password to login to vault
  * `VAULT_CLIENT_KEY`: path to an unencrypted PEM-encoded private key matching the client certificate
  * `VAULT_CLIENT_CERT`: path to a PEM-encoded client certificate for TLS authentication to the Vault server
  * `VAULT_CACERT`: path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
  * `VAULT_CAPATH`: path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
  * `VAULT_AWS_HEADER`: X-Vault-AWS-IAM-Server-ID Header value to prevent replay attacks
  * `VAULT_NAMESPACE`: specify the Vault Namespace, if you have one

Documentation
-------------

There are a few simple examples in this document, but the full documentation can be found at:

https://terryhowe.github.io/ansible-modules-hashivault/modules/list_of_hashivault_modules.html

Reading and Writing
-------------------

The following example writes the giant secret with two values and then
reads the fie value. The `hashivault_secret` module is kv2 by default::

    ---
    - hosts: localhost
      tasks:
        - hashivault_secret:
            secret: giant
            data:
                foo: foe
                fie: fum
        - hashivault_read:
            secret: giant
            key: fie
            version: 2
          register: vault_read

The lookup plugin::

        - set_fact:
            looky: "{{lookup('hashivault', 'giant', 'foo', version=2)}}"

The hashivault_write, hashivault_read and the lookup plugin assume the
/secret mount point.  If you are accessing another mount point, use `mount_point`::

    ---
    - hosts: localhost
      tasks:
        - hashivault_secret_engine:
            name: stories
            backend: generic
        - hashivault_write:
            mount_point: /stories
            secret: stuart
            data:
                last: 'little'
        - hashivault_read:
            mount_point: /stories
            secret: stuart
            key: last
        - set_fact:
            book: "{{lookup('hashivault', 'stuart', 'last', mount_point='/stories')}}"

Version 2 of KV secret engine is also supported, just add `version: 2`::

    ---
        - hashivault_read:
            mount_point: /stories
            version: 2
            secret: stuart
            key: last
        - set_fact:
            book: "{{lookup('hashivault', 'stuart', 'last', mount_point='/stories', version=2)}}"

Initialization, Seal, and Unseal
--------------------------------

The real strength of this module is all the administrative functions you can do. See the documentation
mentioned above for more, but here is a small sample.

You may init the vault::

    ---
    - hosts: localhost
      tasks:
        - hashivault_init:
          register: vault_init

You may also seal and unseal the vault::

    ---
    - hosts: localhost
      vars:
        vault_keys:  "{{ lookup('env','VAULT_KEYS') }}"
      tasks:
        - hashivault_status:
          register: vault_status
        - block:
            - hashivault_seal:
              register: vault_seal
          when: "{{vault_status.status.sealed}} == False"
        - hashivault_unseal:
            keys: '{{vault_keys}}'

Action Plugin
-------------

If you are not using the VAULT_ADDR and VAULT_TOKEN environment variables,
you may be able to simplify your playbooks with an action plugin.  This can
be some somewhat similar to this `example action plugin <https://terryhowe.wordpress.com/2016/05/02/setting-ansible-module-defaults-using-action-plugins/>`_.

Developer Note
--------------
One of the complicated problems with development and testing of this module is
:code:`ansible/module_utils/hashivault.py` is not a directory in itself which
in my opinion is a problem with ansible.  Because of this limitation with
ansible, :code:`pip install -e .` does not work like it would for other
projects.  Two potential ways to work around this issue are either use the
:code:`link.sh` script in the top level directory or run for every change::

    rm -rf dist; python setup.py sdist
    pip install ./dist/ansible-modules-hashivault-*.tar.gz

License
-------

`MIT <https://github.com/TerryHowe/ansible-modules-hashivault/blob/master/LICENSE>`_.


Open Data Cube Core
===================

.. image:: https://github.com/opendatacube/datacube-core/workflows/build/badge.svg
    :alt: Build Status
    :target: https://github.com/opendatacube/datacube-core/actions

.. image:: https://codecov.io/gh/opendatacube/datacube-core/branch/develop/graph/badge.svg
    :alt: Coverage Status
    :target: https://codecov.io/gh/opendatacube/datacube-core

.. image:: https://readthedocs.org/projects/datacube-core/badge/?version=latest
    :alt: Documentation Status
    :target: http://datacube-core.readthedocs.org/en/latest/

Overview
========

The Open Data Cube Core provides an integrated gridded data
analysis environment for decades of analysis ready earth observation
satellite and related data from multiple satellite and other acquisition
systems.

Documentation
=============

See the `user guide <http://datacube-core.readthedocs.io/en/latest/>`__ for
installation and usage of the datacube, and for documentation of the API.

`Join our Slack <http://slack.opendatacube.org>`__ if you need help
setting up or using the Open Data Cube.

Please help us to keep the Open Data Cube community open and inclusive by
reading and following our `Code of Conduct <code-of-conduct.md>`__.

Requirements
============

System
~~~~~~

-  PostgreSQL 10+
-  Python 3.9+

Developer setup
===============

1. Clone:

   -  ``git clone https://github.com/opendatacube/datacube-core.git``

2. Create a Python environment for using the ODC.  We recommend `Mambaforge <https://mamba.readthedocs.io/en/latest/user_guide/mamba.html>`__ as the
   easiest way to handle Python dependencies.

::

   mamba env create -f conda-environment.yml
   conda activate cubeenv

3. Install a develop version of datacube-core.

::

   cd datacube-core
   pip install --upgrade -e .

4. Install the `pre-commit <https://pre-commit.com>`__ hooks to help follow ODC coding
   conventions when committing with git.

::

   pre-commit install

5. Run unit tests + PyLint

Install test dependencies using:

   ``pip install --upgrade -e '.[test]'``

If install for these fails, please lodge them as issues.

Run unit tests with:

   ``./check-code.sh``

   (this script approximates what is run by GitHub Actions. You can
   alternatively run ``pytest`` yourself).

6. **(or)** Run all tests, including integration tests.

   ``./check-code.sh integration_tests``

   -  Assumes a password-less Postgres database running on localhost called

   ``pgintegration``

   -  Otherwise copy ``integration_tests/integration.conf`` to
      ``~/.datacube_integration.conf`` and edit to customise.

   - For instructions on setting up a password-less Postgres database, see
      the `developer setup instructions <https://datacube-core.readthedocs.io/en/latest/installation/setup/ubuntu.html#postgres-database-configuration>`__.

Alternatively one can use the ``opendatacube/datacube-tests`` docker image to run
tests. This docker includes database server pre-configured for running
integration tests. Add ``--with-docker`` command line option as a first argument
to ``./check-code.sh`` script.

::

   ./check-code.sh --with-docker integration_tests

To run individual tests in a docker container

::

    docker build --tag=opendatacube/datacube-tests-local --no-cache --progress plain -f docker/Dockerfile .

    docker run -ti -v $(pwd):/code opendatacube/datacube-tests-local:latest pytest integration_tests/test_filename.py::test_function_name

Developer setup on Ubuntu
~~~~~~~~~~~~~~~~~~~~~~~~~

Building a Python virtual environment on Ubuntu suitable for development work.

Install dependencies:

::

    sudo apt-get update
    sudo apt-get install -y \
        autoconf automake build-essential make cmake \
        graphviz \
        python3-venv \
        python3-dev \
        libpq-dev \
        libyaml-dev \
        libnetcdf-dev \
        libudunits2-dev

Build the python virtual environment:

::

    pyenv="${HOME}/.envs/odc"  # Change to suit your needs
    mkdir -p "${pyenv}"
    python3 -m venv "${pyenv}"
    source "${pyenv}/bin/activate"
    pip install -U pip wheel cython numpy
    pip install -e '.[dev]'
    pip install flake8 mypy pylint autoflake black


# Pygame GUI
A GUI system for pygame CE.

 - [Documentation](https://pygame-gui.readthedocs.io/)
 - [GitHub](https://github.com/MyreMylar/pygame_gui)
 - [Examples](https://github.com/MyreMylar/pygame_gui_examples)
 - [PyPi](https://pypi.org/project/pygame-gui/)

[![pypi](https://badge.fury.io/py/pygame-gui.svg)](https://pypi.python.org/pypi/pygame-gui) [![Documentation Status](https://readthedocs.org/projects/pygame-gui/badge/?version=latest)](https://pygame-gui.readthedocs.io/en/latest/?badge=latest) [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/release/python-380/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![codecov](https://codecov.io/gh/MyreMylar/pygame_gui/branch/main/graph/badge.svg?token=zZhkvhgTki)](https://codecov.io/gh/MyreMylar/pygame_gui) [![Downloads](https://pepy.tech/badge/pygame-gui)](https://pepy.tech/project/pygame-gui)
## Requirements

 - Python 3.8+
 - Pygame Community Edition 2.4.0+
 - python-i18n (does localization to different languages)

## How to install

1. Run this command in a command prompt:
```
pip install pygame_gui
```
2. If all goes well you should see a message about game_gui being installed successfully and will be able to find pygame_gui in the list of installed packages for your python interpreter (PyCharm displays these as a nice list under File->Settings->Project:project_name->Project Interpreter).
3. Should you need to delete pygame_gui for any reason then PyCharm will also let you do that from the same Project Interpreter settings window using the red minus symbol button.

## How to upgrade to the latest version

Run this command in a command prompt:

```
pip install pygame_gui -U
```

Or, you may be able to use your IDE to update the library (PyCharm lets you update modules in it's interpreter settings).

## Making use of the library

There is documentation available [here](https://pygame-gui.readthedocs.io/en/latest/index.html), you can also try out the examples at the [pygame_gui_examples repository](https://github.com/MyreMylar/pygame_gui_examples).

## Screenshots of Pygame GUI

![pygame_gui_example_1](https://user-images.githubusercontent.com/13382426/69264498-57ec8980-0bbf-11ea-9883-cac9c854615d.png)

![example_2](https://user-images.githubusercontent.com/13382426/69264921-2f18c400-0bc0-11ea-8a11-d9abd4e969b4.png)

![pygame_gui_example_3](https://user-images.githubusercontent.com/13382426/68039142-5ec06480-fcc3-11e9-91f4-3e401f459886.png)

![pygame_gui_example_4](https://user-images.githubusercontent.com/13382426/68041632-e52b7500-fcc8-11e9-8b72-4cf8817c5fa3.png)


# python-hearthstone

[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/hearthsim/python-hearthstone/ci.yml?branch=master)](https://github.com/HearthSim/python-hearthstone/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/hearthstone.svg)](https://pypi.org/project/hearthstone/)

A Hearthstone Python library containing:

* A CardDefs.xml parser (`hearthstone.cardxml`)
* A DbfXml parser (`hearthstone.dbf`)
* A deck code encoder and decoder (`hearthstone.deckstrings`)
* Hearthstone enums as IntEnum (`hearthstone.enums`)

The CardDefs.xml data for the latest build can optionally be installed from the
[python-hearthstone-data repository](https://github.com/HearthSim/python-hearthstone-data)
or on PyPI with `pip install hearthstone_data`. Otherwise, they will be download at runtime.

## Requirements

* Python 3.6+
* lxml

## Installation

* To install from PyPI: `pip install hearthstone`

## License

This project is licensed under the MIT license. The full license text is
available in the LICENSE file.

## Community

This is a [HearthSim](https://hearthsim.info) project.
Join the HearthSim Developer community [on Discord](https://discord.gg/hearthsim-devs).


# `pyani`

Application and Python module for whole-genome classification of microbes using Average Nucleotide Identity.

-----

## Contributing

If you would like to contribute code to the `pyani` project (e.g. a bug fix or new feature), please refer to the [`CONTRIBUTING.md`](CONTRIBUTING.md) guide for more details.

`pyani` is maintained by:

- [Leighton Pritchard](https://pureportal.strath.ac.uk/en/persons/leighton-pritchard)
- [Bailey Harrington](https://pureportal.strath.ac.uk/en/persons/bailey-harrington)

and we are grateful to all who have contributed to this software:

### Contributors ✨  ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

Leighton Pritchard
📖 💻 🎨 💵 🤔 🚇 📆 🔧 ⚠ ✅

Bailey Harrington
💻 📖 🎨 🤔 🚇 🔧

Peter Cock
💻 🤔

Robert Davey
💻 🤔

Nick Waters
📖

YT
💻 🤔

Özcan Esen
💻 🐛

Balázs Brankovics
💻 🐛

sammywinchester19
🐛

Ram Krishna Shrestha
⚠ 💻 🤔 🐛

Donovan Parks
🐛

Zaki Saati Santamaría
🐛

neelam19051
🐛

jthomp28
🐛

louellette
🤔

Seb.
🤔

ninjatacoshell
🤔

Patricia H. Brito
🤔

Zilia Y. Muñoz Ramirez
🐛

nclaesnacw
🤔

Eric Deveaud
🐛

Angelika Kiepas
💻 🎨 ⚠

> (This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!)

-----

## Citing `pyani`

A complete guide to citing `pyani` is included in the file [`CITATIONS`](CITATIONS). Please cite the following manuscript in your work, if you have found `pyani` useful:

> Pritchard *et al.* (2016) "Genomics and taxonomy in diagnostics for food security: soft-rotting enterobacterial plant pathogens" *Anal. Methods* **8**, 12-24
DOI: [10.1039/C5AY02550H](https://doi.org/10.1039/C5AY02550H)

-----

![macOS Python 3.11 build](https://github.com/widdowquinn/pyani/actions/workflows/build-macos.yml/badge.svg)
![Linux Python 3.11 build](https://github.com/widdowquinn/pyani/actions/workflows/build-linux.yml/badge.svg)
[![codecov.io coverage](https://img.shields.io/codecov/c/github/widdowquinn/pyani/master.svg)](https://codecov.io/github/widdowquinn/pyani)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/f3e56b2bf118471aabf09514a3e6af75)](https://www.codacy.com/manual/widdowquinn/pyani?utm_source=github.com&utm_medium=referral&utm_content=widdowquinn/pyani&utm_campaign=Badge_Grade)
[![CodeFactor](https://www.codefactor.io/repository/github/widdowquinn/pyani/badge)](https://www.codefactor.io/repository/github/widdowquinn/pyani)
[![documentation](https://readthedocs.org/projects/pyani/badge/?version=latest)](https://pyani.readthedocs.io/en/latest/?badge=latest)
[![pyani sourcerank](https://img.shields.io/librariesio/sourcerank/pypi/pyani.svg?logo=koding&logoColor=white)](https://libraries.io/pypi/pyani)

[![All Contributors](https://img.shields.io/badge/all_contributors-22-orange.svg?style=flat-square)](#contributors-)

[![pyani PyPi version](https://img.shields.io/pypi/v/pyani "PyPI version")](https://pypi.python.org/pypi/pyani)
[![pyani licence](https://img.shields.io/pypi/l/pyani "PyPI licence")](https://github.com/widdowquinn/pyani/blob/master/LICENSE)
[![pyani PyPi version](https://img.shields.io/pypi/format/pyani "PyPI format")](https://pypi.python.org/pypi/pyani)
[![pyani PyPi version](https://img.shields.io/pypi/pyversions/pyani "Python versions")](https://pypi.python.org/pypi/pyani)
[![pyani PyPi version](https://img.shields.io/pypi/dm/pyani "PyPI downloads")](https://pypi.python.org/pypi/pyani)

[![GitHub Issues](https://img.shields.io/github/issues-closed/widdowquinn/pyani.svg)](https://github.com/widdowquinn/pyani/issues)
[![GitHub Stars](https://img.shields.io/github/stars/widdowquinn/pyani.svg)](https://github.com/widdowquinn/pyani/stargazers)
[![Code Style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

[![Anaconda-Server Badge](https://anaconda.org/bioconda/pyani/badges/version.svg)](https://anaconda.org/bioconda/pyani)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/pyani/badges/latest_release_date.svg)](https://anaconda.org/bioconda/pyani)

[![Anaconda-Server Badge](https://anaconda.org/bioconda/pyani/badges/installer/conda.svg)](https://conda.anaconda.org/bioconda)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/pyani/badges/downloads.svg)](https://anaconda.org/bioconda/pyani)
[![Anaconda-Server Badge](https://anaconda.org/bioconda/pyani/badges/platforms.svg)](https://anaconda.org/bioconda/pyani)

-----

## Table of Contents

- [`pyani`](#pyani)
  - [Contributing](#contributing)
    - [Contributors ✨  (emoji key):](#contributors---emoji-key)
  - [Citing `pyani`](#citing-pyani)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Installation](#installation)
    - [IMPORTANT NOTICE](#important-notice)
    - [Documentation for stable version `pyani` v0.2.x](#documentation-for-stable-version-pyani-v02x)
    - [`conda`](#conda)
    - [`pip`](#pip)
    - [Third-party tools](#third-party-tools)
    - [NOTE: Installing legacy BLAST](#note-installing-legacy-blast)
    - [Note: Installing fastANI](#note-installing-fastani)
        - [Bypassing `conda`](#bypassing-conda)
        - [Troubleshooting](#troubleshooting)
  - [Documentation (v0.3)](#documentation-v03)
    - [Older documentation (v0.2)](#older-documentation-v02)
  - [Bugs, issues, problems and questions](#bugs-issues-problems-and-questions)
  - [Walkthrough: A First Analysis](#walkthrough-a-first-analysis)
    - [1. Download genome data](#1-download-genome-data)
    - [2. Create an analysis database](#2-create-an-analysis-database)
    - [3. Conduct ANI analysis](#3-conduct-ani-analysis)
      - [Rerunning the same analysis](#rerunning-the-same-analysis)
    - [4. Reporting Analyses and Analysis Results](#4-reporting-analyses-and-analysis-results)
    - [5. Generating graphical output for ANI](#5-generating-graphical-output-for-ani)
    - [6. Classifying Genomes from Analysis Results](#6-classifying-genomes-from-analysis-results)
  - [Using a scheduler](#using-a-scheduler)
    - [SGE/OGE](#sgeoge)
      - [Arguments assigned by Pyani](#arguments-assigned-by-pyani)
      - [Modifiable arguments](#modifiable-arguments)
      - [Specifying additional arguments](#specifying-additional-arguments)
  - [Running `pyani` version 0.2.x](#running-pyani-version-02x)
    - [Script: `average_nucleotide_identity.py`](#script-average_nucleotide_identitypy)
    - [Script: `genbank_get_genomes_by_taxon.py`](#script-genbank_get_genomes_by_taxonpy)
  - [Method and Output Description](#method-and-output-description)
    - [Average Nucleotide Identity (ANI)](#average-nucleotide-identity-ani)
  - [Licensing](#licensing)

-----

## Overview

`pyani` is a software package and Python3 module that calculates average nucleotide identity (ANI) and related measures for whole genome comparisons, and renders relevant graphical summary output.

Where available, `pyani` can take advantage of multicore systems, and integrates with [SGE/OGE](http://gridscheduler.sourceforge.net/)-type job schedulers for the sequence comparisons.

`pyani` installs the program `pyani`, which enables command-line based analysis of genomes.

-----

## Installation

### IMPORTANT NOTICE

**THIS REPOSITORY DEFAULT BRANCH CONTAINS A DEVELOPMENT VERSION OF `PYANI` (v0.3+). IT HAS A DIFFERENT COMMAND-LINE INTERFACE THAN THE STABLE `PYANI` VERSION (v0.2.x).**

### Documentation for stable version `pyani` v0.2.x

- [`pyani` v0.2.x Documentation](./README_v_0_2_x.md)

**IF YOU WISH TO INSTALL THE STABLE VERSION OF `PYANI`, PLEASE FOLLOW THESE INSTRUCTIONS FOR INSTALLING v0.2.x**

The easiest way to install `pyani` v0.2 is to use `conda` or  `pip`. `conda` is recommended for the simplest installation of third-party tool dependencies (`mummer` and `BLAST`/`BLAST+`).

### `conda`

You will need to install the `bioconda` channel, following instructions at [https://bioconda.github.io/user/install.html](https://bioconda.github.io/user/install.html). Then, to create a new environment for `pyani` and install the program, issue the following command:

```bash
conda create --name pyani_env python=3.8 -y
conda activate pyani_env
conda install pyani
```

### `pip`

`pip` will install `pyani` and its Python dependencies, but not the third-party tools.

```bash
pip3 install pyani
```

### Third-party tools

Four alignment packages are required, to use all of `pyani`'s methods: `mummer`, `BLAST+`, legacy `BLAST`, and `fastANI`. (*Note that it is not necessary to install all four tools to use `pyani`. You need only install the tools that are required for the method you wish to use.*)

The simplest route to obtaining these tools is to use `conda`/`bioconda`:

```bash
conda install mummer blast legacy-blast fastani -y
```

But they can also be installed by following instructions from the tools' own websites.

- **BLAST+** (for `anib`) [ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/](ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/)
- **legacy BLAST** (for `aniblastall`) [ftp://ftp.ncbi.nlm.nih.gov/blast/executables/release/LATEST/](ftp://ftp.ncbi.nlm.nih.gov/blast/executables/release/LATEST/)
- **MUMmer** (for `anim`) [http://mummer.sourceforge.net/](ftp://ftp.ncbi.nlm.nih.gov/blast/executables/release/LATEST/)
- **fastANI** (for `fastani`) [https://github.com/ParBLiSS/FastANI](https://github.com/ParBLiSS/FastANI)

### NOTE: Installing legacy BLAST

**The legacy BLAST executable available from NCBI will not run on macOS Big Sur.**

If you wish to use `pyani blastall` or the `ANIblastall` method with the legacy `pyani` interface, you will require a locally-installed copy of this package. This is one of the packages linked in the `requirements-thirdparty.txt` file.

### Note: Installing fastANI

**`fastANI` installation (via `conda`) will fail for Python 3.9 on macOS.**

If you are using Python ≥3.9 to run `pyani` on macOS, `fastANI` will fail to install due to errors in the `conda` recipe (for build `h0a26cfa_0`). Using an earlier version of Python may be the simplest solution. However, you can also install `fastANI` yourself—via `conda`—using a slightly altered version of the command produced from the `conda` recipe, as shown below.

*N.B. We have placed the `fastANI` requirement in its own `requirements-fastani.txt` file so that this will not also cause other installations to fail.*

##### Bypassing `conda`

```bash
c++ -O3 -DNDEBUG -std=c++11 -Isrc \
-I ${ENV_DIR}/lib/include -mmacosx-version-min=10.7 \
-stdlib=libc++ -Xpreprocessor -fopenmp -lomp \
-DUSE_BOOST src/cgi/core_genome_identity.cpp -o fastANI \
${ENV_DIR}/lib/lib/libboost_math_c99.a \
-lstdc++ -lz -lm
```

Two alterations need to be made to the file name specified in the fifth line here:

1. One of the `/lib`s should be removed.
1. The file extension should be changed from `.a`, to `.dylib`.

The resultant command ought to look something like this (with ${ENV_DIR} being wherever you're installing `fastANI`):

```bash
c++ -O3 -DNDEBUG -std=c++11 -Isrc \
-I ${ENV_DIR}/lib/include -mmacosx-version-min=10.7 \
-stdlib=libc++ -Xpreprocessor -fopenmp -lomp \
-DUSE_BOOST src/cgi/core_genome_identity.cpp -o fastANI \
${ENV_DIR}/lib/libboost_math_c99.dylib \
-lstdc++ -lz -lm
```

For a more technical overview of the issue, and other solutions that may be shared there, please see https://github.com/widdowquinn/pyani/issues/377.

##### Troubleshooting

This solution is how one of our developers managed to solve the problem. We hope it works for you, but can not guarantee this. Unfortunately, if this does not work for you, we will also not be able to troubleshoot the issue. In this instance, please file an issue at https://github.com/bioconda/bioconda-recipes.

-----

## Documentation (v0.3)

This `README.md` file provides a quick overview and walkthrough for **THE DEVELOPMENT VERSION (v0.3+) OF `pyani`**, and full documentation can be found at the link below:

- [`pyani` v0.3+ Documentation](https://readthedocs.org/projects/pyani/)

**THIS README AND THE DOCUMENTATION AT `READTHEDOCS` REFERS TO A DEVELOPMENT VERSION OF `PYANI` (v0.3+). IT HAS A DIFFERENT COMMAND-LINE INTERFACE THAN THE STABLE `PYANI` VERSION (v0.2.x).**

**THE STABLE VERSION OF `PYANI` (v0.2) DOES NOT HAVE THE `pyani` COMMAND**

### Older documentation (v0.2)

If you are using `pyani` v0.2.x, then please note that the command-line API has changed, and documentation for this version can be found at the following page:

- [`pyani` v0.2.x Documentation](./README_v_0_2_x.md)

## Bugs, issues, problems and questions

If you would like to report a bug or problem with `pyani`, or ask a question of the developer(s), please raise an issue at the link below:

- [`pyani` Issues page](https://github.com/widdowquinn/pyani/issues)

-----

## Walkthrough: A First Analysis

The command-line interface to `pyani` uses subcommands. These separate individual steps of an analysis into separate actions.

1. Download genomes
2. Create a database to hold genome data and analysis results
3. Perform ANI analysis
4. Report and visualise analysis results
5. Generate species hypotheses (classify genomes) using the analysis results

The steps are described in detail with examples, below.

### 1. Download genome data

The first step is to obtain genome data for analysis. `pyani` expects to find each individual genome in its own FASTA file (that file can contain multiple sequences - chromosomes and plasmids; sequenced scaffolds, etc). All the FASTA files for an analysis are expected to be located in a single subdirectory (with optional `labels` and `classes` files). You can arrange your data manually, but `pyani` provides a subcommand that downloads all genomes in a taxon subtree from NCBI, and organises them ready for use with `pyani`.

We'll use the `pyani download` subcommand to download all available genomes for *Candidatus Blochmannia* from NCBI. The taxon ID for this grouping is [203804](https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=203804&lvl=3&lin=f&keep=1&srchmode=1&unlock).

```bash
pyani download -o C_blochmannia --email my.email@my.domain -t 203804 -v -l C_blochmannia_dl.log
```

The first argument is the output directory into which the downloaded genomes will be written (`-o C_blochmannia`). To download anything from NCBI we must provide an email address (`--email my.email@my.domain`), and to specify which taxon subtree we want to download we provide the taxon ID (`-t 203804`).

Here we also request verbose output (`-v`), and write a log file for reproducible research/diagnosing bugs and errors (`-l C_blochmannia_dl.log`).

This produces a new subdirectory (`C_blochmannia`) with the following contents:

```bash
$ tree C_blochmannia
C_blochmannia
├── GCF_000011745.1_ASM1174v1_genomic.fna
├── GCF_000011745.1_ASM1174v1_genomic.fna.gz
├── GCF_000011745.1_ASM1174v1_genomic.fna.md5
├── GCF_000011745.1_ASM1174v1_hashes.txt
├── GCF_000043285.1_ASM4328v1_genomic.fna
├── GCF_000043285.1_ASM4328v1_genomic.fna.gz
├── GCF_000043285.1_ASM4328v1_genomic.fna.md5
├── GCF_000043285.1_ASM4328v1_hashes.txt
├── GCF_000185985.2_ASM18598v2_genomic.fna
├── GCF_000185985.2_ASM18598v2_genomic.fna.gz
├── GCF_000185985.2_ASM18598v2_genomic.fna.md5
├── GCF_000185985.2_ASM18598v2_hashes.txt
├── GCF_000331065.1_ASM33106v1_genomic.fna
├── GCF_000331065.1_ASM33106v1_genomic.fna.gz
├── GCF_000331065.1_ASM33106v1_genomic.fna.md5
├── GCF_000331065.1_ASM33106v1_hashes.txt
├── GCF_000973505.1_ASM97350v1_genomic.fna
├── GCF_000973505.1_ASM97350v1_genomic.fna.gz
├── GCF_000973505.1_ASM97350v1_genomic.fna.md5
├── GCF_000973505.1_ASM97350v1_hashes.txt
├── GCF_000973545.1_ASM97354v1_genomic.fna
├── GCF_000973545.1_ASM97354v1_genomic.fna.gz
├── GCF_000973545.1_ASM97354v1_genomic.fna.md5
├── GCF_000973545.1_ASM97354v1_hashes.txt
├── classes.txt
└── labels.txt
```

Seven genomes have been downloaded, and each is represented by four files:

- `_genomic.fna.gz`: the compressed genome sequence
- `_genomic.fna`: the uncompressed genome sequence
- `_genomic.fna.md5`: an MD5 hash/checksum of the (uncompressed) genome sequence; this was generated during the download
- `_hashes.txt`: a list of MD5 hashes; this is provided by NCBI and is a reference to be sure that the download did not corrupt the genome sequence

There are two additional plain text files: `classes.txt` and `labels.txt`, which provide alternative labels for use in the analysis. These files are generated during the download.

### 2. Create an analysis database

`pyani` uses a database to store genome data and analysis results. This is convenient for data sharing and developing custom analyses, but also makes it easier to extend an existing ANI analysis with new genomes, without having to repeat calculations that were already performed.

To create a new, clean, database in the default location (`.pyani/pyanidb`) issue the command:

```bash
pyani createdb -v -l C_blochmannia_createdb.log
```

As above, the verbose (`-v`) and log file (`-l C_blochmannia_createdb.log`) options allow for reproducible work. The default database location is in the hidden directory (`.pyani`):

```bash
$ tree .pyani
.pyani
└── pyanidb
```

Subsequent `pyani` commands will assume this location for the database, but you can specify the location when creating a database, or using an existing database.

### 3. Conduct ANI analysis

`pyani` provides four subcommands to run ANI analyses:

- `anim`: ANIm
- `anib`: ANIb, using BLAST+
- `aniblastall`: ANIb, using legacy BLAST
- `tetra`: TETRA

In this walkthrough, we'll run ANIm on the downloaded genomes, using the command:

```bash
pyani anim -i C_blochmannia -o C_blochmannia_ANIm -v -l C_blochmannia_ANIm.log \
    --name "C. blochmannia run 1" \
    --labels C_blochmannia/labels.txt --classes C_blochmannia/classes.txt
```

All four analysis commands operate in a similar way. The first two arguments are paths to directories: the first path is to a directory containing input genomes, and the second is the path to an output directory for storing intermediate results. The `-v` and `-l` arguments work as above, specifying verbose output and logging output to a file.

You will probably notice that the verbose output is very verbose, to enable informative identification of any problems. In particular, the verbose output (which is also written to the log file) writes out the command-lines used for the pairwise comparisons so, if something goes wrong, you can test whether a specific comparison can be run at the command-line *at all*, to aid diagnosis of any problems.

#### Rerunning the same analysis

One reason for using a database backend for analysis results is so that, for very large analyses, we do not ever need to recalculate a pairwise genome comparison. All the analysis subcommands check whether input genomes have been used before (using the unique MD5 hash for each genome to identify whether it's been used previously), and whether the comparison of two genomes has been run, with the particular analysis settings that were used. If either genome was not seen before, or if the analysis settings are different, the comparison is performed.

You can test this for yourself by running the analysis command again, as below. You will see a number of messages indicating that genomes have been seen before, and that analyses performed before were skipped:

```bash
$ pyani anim -i C_blochmannia -o C_blochmannia_ANIm -v -l C_blochmannia_ANIm.log \
    --name "C. blochmannia run 2" \
    --labels C_blochmannia/labels.txt --classes C_blochmannia/classes.txt
INFO: command-line: pyani anim -i C_blochmannia -o C_blochmannia_ANIm -v -l C_blochmannia_ANIm.log
INFO: Running ANIm analysis
INFO: Adding analysis information to database .pyani/pyanidb
INFO: Current analysis has ID 2 in this database
INFO: Identifying input genome/hash files:
[…]
INFO: Adding genome data to database...
WARNING: Genome already in database with this hash and path!
WARNING: Using existing genome from database, row 1
[…]
INFO: Complete pairwise comparison list:
    [(1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (2, 3), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6), (4, 5), (4, 6), (5, 6)]
INFO: Excluding pre-calculated comparisons
INFO: Comparisons still to be performed:
    []
INFO: All comparison results already present in database (skipping comparisons)
INFO: Completed. Time taken: 0.211
```

### 4. Reporting Analyses and Analysis Results

Once an analysis is run, the results are placed in a local `SQLite` database, which can be queried for information about the analyses that have been run. You can request information about:

- `--runs`: show all analysis runs with results stored in the database
- `--runs_genomes`: show all the analysis runs with results in the database, and all the genomes analysed in each run
- `--genomes`: show all the genomes used for any analysis in the database
- `--genomes_runs`: for each genome in the database, also list the analysis results it participates in
- `--run_results`: show all the pairwise comparison results for a named run (run IDs can be obtained with the `--runs` argument

The report tables are written to a named directory (compulsory argument), and are written by default to a `.tab` plain-text format, but HTML and Excel format can also be requested with the `--formats` argument:

```bash
$ pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel,stdout
INFO: Processed arguments: Namespace(cmdline='./pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel', dbpath='.pyani/pyanidb', formats='html,excel', func=<function subcmd_report at 0x10c674a60>, logfile=None, outdir='C_blochmannia_ANIm/', run_results=False, show_genomes=False, show_genomes_runs=False, show_runs=True, show_runs_genomes=False, verbose=True)
INFO: command-line: ./pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel
INFO: Creating output in formats: ['excel', 'tab', 'html']
INFO: Using database: .pyani/pyanidb
INFO: Writing table of pyani runs from the database to C_blochmannia_ANIm/runs.*
INFO: Completed. Time taken: 0.937
$ tree -L 1 C_blochmannia_ANIm/
C_blochmannia_ANIm/
├── nucmer_output
├── runs.html
├── runs.tab
└── runs.xlsx
```

To see all of the pairwise results for an individual run, the run ID must be provided. It is possible to get results for more than one run ID by providing a comma-separated list of run IDs (though each run's results will be provided in a separate file):

```bash
$ pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel --run_results 1,2,3,4
INFO: Processed arguments: Namespace(cmdline='./pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel --run_results 1,2,3,4', dbpath='.pyani/pyanidb', formats='html,excel', func=<function subcmd_report at 0x108616a60>, logfile=None, outdir='C_blochmannia_ANIm/', run_results='1,2,3,4', show_genomes=False, show_genomes_runs=False, show_runs=True, show_runs_genomes=False, verbose=True)
INFO: command-line: ./pyani report -v --runs -o C_blochmannia_ANIm/ --formats html,excel --run_results 1,2,3,4
INFO: Creating output in formats: ['tab', 'excel', 'html']
INFO: Using database: .pyani/pyanidb
INFO: Writing table of pyani runs from the database to C_blochmannia_ANIm/runs.*
INFO: Attempting to write results tables for runs: ['1', '2', '3', '4']
INFO: Collecting data for run with ID: 1
INFO: Collecting data for run with ID: 2
INFO: Collecting data for run with ID: 3
INFO: Collecting data for run with ID: 4
INFO: Completed. Time taken: 1.285
```

You can see a run's results in the terminal by specifying the `stdout` format. For example, to see the identity, coverage, and other output matrices, you would specify `--run_matrices <RUN>` and `--formats=stdout` as below:

```bash
$ pyani report -o C_blochmannia_ANIm --formats=stdout --run_matrices 1
TABLE: C_blochmannia_ANIm/matrix_identity_1
                                                    C. Blochmannia pennsylvanicus BPEN  C. Blochmannia floridanus  C. Blochmannia vafer BVAF  C. Blochmannia chromaiodes 640  B. endosymbiont of Polyrhachis (Hedomyrma) turneri 675  B. endosymbiont of Camponotus (Colobopsis) obliquus 757
C. Blochmannia pennsylvanicus BPEN                                            1.000000                   0.834866                   0.836903                        0.980244                                           0.843700                                                0.829509
C. Blochmannia floridanus                                                     0.834866                   1.000000                   0.828733                        0.834916                                           0.847060                                                0.857859
C. Blochmannia vafer BVAF                                                     0.836903                   0.828733                   1.000000                        0.837811                                           0.866015                                                0.844438
C. Blochmannia chromaiodes 640                                                0.980244                   0.834916                   0.837811                        1.000000                                           0.849834                                                0.834769
B. endosymbiont of Polyrhachis (Hedomyrma) turn...                            0.843700                   0.847060                   0.866015                        0.849834                                           1.000000                                                0.844228
B. endosymbiont of Camponotus (Colobopsis) obli...                            0.829509                   0.857859                   0.844438                        0.834769                                           0.844228                                                1.000000

TABLE: C_blochmannia_ANIm/matrix_coverage_1
                                                    C. Blochmannia pennsylvanicus BPEN  C. Blochmannia floridanus  C. Blochmannia vafer BVAF  C. Blochmannia chromaiodes 640  B. endosymbiont of Polyrhachis (Hedomyrma) turneri 675  B. endosymbiont of Camponotus (Colobopsis) obliquus 757
C. Blochmannia pennsylvanicus BPEN                                            1.000000                   0.045736                   0.041404                        1.000306                                           0.017263                                                0.021027
C. Blochmannia floridanus                                                     0.051317                   1.000000                   0.152609                        0.054930                                           0.016366                                                0.010749
C. Blochmannia vafer BVAF                                                     0.045362                   0.149012                   1.000000                        0.046520                                           0.008356                                                0.014706
C. Blochmannia chromaiodes 640                                                1.000856                   0.048983                   0.042485                        1.000000                                           0.014056                                                0.016140
B. endosymbiont of Polyrhachis (Hedomyrma) turn...                            0.018238                   0.015410                   0.008058                        0.014841                                           1.000000                                                0.020416
B. endosymbiont of Camponotus (Colobopsis) obli...                            0.021508                   0.009799                   0.013730                        0.016500                                           0.019766                                                1.000000

TABLE: C_blochmannia_ANIm/matrix_aln_lengths_1
                                                    C. Blochmannia pennsylvanicus BPEN  C. Blochmannia floridanus  C. Blochmannia vafer BVAF  C. Blochmannia chromaiodes 640  B. endosymbiont of Polyrhachis (Hedomyrma) turneri 675  B. endosymbiont of Camponotus (Colobopsis) obliquus 757
C. Blochmannia pennsylvanicus BPEN                                            791654.0                    36207.0                    32778.0                        791896.0                                            13666.0                                                 16646.0
C. Blochmannia floridanus                                                      36207.0                   705557.0                   107674.0                         38756.0                                            11547.0                                                  7584.0
C. Blochmannia vafer BVAF                                                      32778.0                   107674.0                   722585.0                         33615.0                                             6038.0                                                 10626.0
C. Blochmannia chromaiodes 640                                                791896.0                    38756.0                    33615.0                        791219.0                                            11121.0                                                 12770.0
B. endosymbiont of Polyrhachis (Hedomyrma) turn...                             13666.0                    11547.0                     6038.0                         11121.0                                           749321.0                                                 15298.0
B. endosymbiont of Camponotus (Colobopsis) obli...                             16646.0                     7584.0                    10626.0                         12770.0                                            15298.0                                                     NaN

TABLE: C_blochmannia_ANIm/matrix_sim_errors_1
                                                    C. Blochmannia pennsylvanicus BPEN  C. Blochmannia floridanus  C. Blochmannia vafer BVAF  C. Blochmannia chromaiodes 640  B. endosymbiont of Polyrhachis (Hedomyrma) turneri 675  B. endosymbiont of Camponotus (Colobopsis) obliquus 757
C. Blochmannia pennsylvanicus BPEN                                                 0.0                     5979.0                     5346.0                         15645.0                                             2136.0                                                  2838.0
C. Blochmannia floridanus                                                       5979.0                        0.0                    18441.0                          6398.0                                             1766.0                                                  1078.0
C. Blochmannia vafer BVAF                                                       5346.0                    18441.0                        0.0                          5452.0                                              809.0                                                  1653.0
C. Blochmannia chromaiodes 640                                                 15645.0                     6398.0                     5452.0                             0.0                                             1670.0                                                  2110.0
B. endosymbiont of Polyrhachis (Hedomyrma) turn...                              2136.0                     1766.0                      809.0                          1670.0                                                0.0                                                  2383.0
B. endosymbiont of Camponotus (Colobopsis) obli...                              2838.0                     1078.0                     1653.0                          2110.0                                             2383.0                                                     0.0

TABLE: C_blochmannia_ANIm/matrix_hadamard_1
                                                    C. Blochmannia pennsylvanicus BPEN  C. Blochmannia floridanus  C. Blochmannia vafer BVAF  C. Blochmannia chromaiodes 640  B. endosymbiont of Polyrhachis (Hedomyrma) turneri 675  B. endosymbiont of Camponotus (Colobopsis) obliquus 757
C. Blochmannia pennsylvanicus BPEN                                            1.000000                   0.038183                   0.034652                        0.980543                                           0.014564                                                0.017442
C. Blochmannia floridanus                                                     0.042843                   1.000000                   0.126472                        0.045862                                           0.013863                                                0.009221
C. Blochmannia vafer BVAF                                                     0.037964                   0.123491                   1.000000                        0.038975                                           0.007237                                                0.012418
C. Blochmannia chromaiodes 640                                                0.981082                   0.040896                   0.035594                        1.000000                                           0.011945                                                0.013473
B. endosymbiont of Polyrhachis (Hedomyrma) turn...                            0.015387                   0.013053                   0.006978                        0.012613                                           1.000000                                                0.017236
B. endosymbiont of Camponotus (Colobopsis) obli...                            0.017841                   0.008406                   0.011594                        0.013774                                           0.016687                                                1.000000
```

### 5. Generating graphical output for ANI

The output of a `pyani` run can also be represented graphically, using the `plot` subcommand. For example, the command:

```bash
pyani plot -o C_blochmannia_ANIm --run_id 1 -v --formats png,pdf
```

will place `.pdf` and `.png` format output in the `C_blochmannia_ANIm` output directory for the run with ID 1, generated above. Five heatmaps are generated:

- percentage identity
- percentage coverage (for both query and subject)
- alignment length (total aligned length)
- similarity errors (total number of mismatches, not including indels)
- hadamard matrix (dot product of identity and coverage matrices)

The heatmaps also include dendrograms, clustering the rows and columns by overall similarity.

**NOTE:** Running `pyani plot` with a large number of genomes (~500) and the default figure output (`--method seaborn`) may reduce output figure quality:

- labels in `.png` files may be difficult to read
- `.pdf` files may render very slowly due to the large number of numerical values in heatmap cells.

With large datasets, `--method mpl` (matplotlib) is recommended.

Please be aware that the matrix orientation differs for these two options; so, with `seaborn` (the default, `--method seaborn`), the orientation of self-comparisons is top left to bottom right (`\`), while with `matplotlib` (`--method mpl`) the orientation is bottom left to top right (`/`).

### 6. Classifying Genomes from Analysis Results

-----

## Using a scheduler

### SGE/OGE

The `--scheduler SGE` argument allows one to use `pyani` with an an SGE-type scheduler.

In order for this work, one must be able to submit jobs using the `qsub` command. By default, this will batch the pairwise comparisons in array jobs of 10,000, in order to avoid clogging the scheduler queue. Each comparison will be run as a single-core task in an array job.

#### Arguments assigned by Pyani
The following arguments will be automatically set:

```bash
-N job_name  # this is the value passed to `--name`
-cwd
-o ./stdout  # cwd/ + "stdout"
-e ./stderr  # cwd/ + "stderr"
```

#### Modifiable arguments
The number of pairwise comparisons submitted per chunk can be modified using:

```bash
--SGEgroupsize *number*
```

The job prefix to use can be modified using:

```bash
--jobprefix *prefix*
```

#### Specifying additional arguments
Additional SGE arguments may be specified with:

```bash
--SGEargs "<your arguments here>"
```

## Running `pyani` version 0.2.x

### Script: `average_nucleotide_identity.py`

The `average_nucleotide_identity.py` script - installed as part of this package - enables straightforward ANI analysis at the command-line, and uses the `pyani` module behind the scenes.

You can get a summary of available command-line options with `average_nucleotide_identity.py -h`

```bash
$ ./average_nucleotide_identity.py -h
usage: average_nucleotide_identity.py [-h] [-o OUTDIRNAME] [-i INDIRNAME] [-v]
                                      [-f] [-s FRAGSIZE] [-l LOGFILE]
                                      [--skip_nucmer] [--skip_blastn]
                                      [--noclobber] [--nocompress] [-g]
                                      [--gformat GFORMAT] [--gmethod GMETHOD]
                                      [--labels LABELS] [--classes CLASSES]
                                      [-m METHOD] [--scheduler SCHEDULER]
                                      [--workers WORKERS]
                                      [--SGEgroupsize SGEGROUPSIZE]
                                      [--maxmatch] [--nucmer_exe NUCMER_EXE]
                                      [--blastn_exe BLASTN_EXE]
                                      [--makeblastdb_exe MAKEBLASTDB_EXE]
                                      [--blastall_exe BLASTALL_EXE]
                                      [--formatdb_exe FORMATDB_EXE]
                                      [--write_excel] [--subsample SUBSAMPLE]
                                      [--seed SEED] [--jobprefix JOBPREFIX]

[…]
```

Example data and output can be found in the directory `test_ani_data`. The data are chromosomes of four isolates of *Caulobacter*. Basic analyses can be performed with the command lines:

```bash
./average_nucleotide_identity.py -i tests/test_ani_data/ -o tests/test_ANIm_output -m ANIm -g
./average_nucleotide_identity.py -i tests/test_ani_data/ -o tests/test_ANIb_output -m ANIb -g
./average_nucleotide_identity.py -i tests/test_ani_data/ -o tests/test_ANIblastall_output -m ANIblastall -g
./average_nucleotide_identity.py -i tests/test_ani_data/ -o tests/test_TETRA_output -m TETRA -g
```

The graphical output below, supporting assignment of `NC_002696` and `NC_011916` to the same species (*C.crescentus*), and the other two isolates to distinct species (`NC_014100`:*C.segnis*; `NC_010338`:*C.* sp K31), was generated with the command-line:

```bash
./average_nucleotide_identity.py -v -i tests/test_ani_data/ \
    -o tests/test_ANIm_output/ -g --gformat png,pdf,eps \
    --classes tests/test_ani_data/classes.tab \
    --labels tests/test_ani_data/labels.tab
```

![ANIm percentage identity for *Caulobacter* test data](tests/test_ani_data/ANIm_percentage_identity.png "ANIm percentage identity")
![ANIm alignment coverage for *Caulobacter* test data](tests/test_ani_data/ANIm_alignment_coverage.png "ANIm alignment coverage")
![ANIm alignment length for *Caulobacter* test data](tests/test_ani_data/ANIm_alignment_lengths.png "ANIm alignment length")
![ANIm alignment similarity errors for *Caulobacter* test data](tests/test_ani_data/ANIm_similarity_errors.png "ANIm alignment similarity")

### Script: `genbank_get_genomes_by_taxon.py`

The script `genbank_get_genomes_by_taxon.py`, installed by this package, enables download of genomes from NCBI, specified by taxon ID. The script will download all available assemblies for taxa at or below the specified node in the NCBI taxonomy tree.

Command-line options can be viewed using:

```bash
$ genbank_get_genomes_by_taxon.py -h
usage: genbacnk_get_genomes_by_taxon.py [-h] [-o OUTDIRNAME] [-t TAXON] [-v]
                                        [-f] [--noclobber] [-l LOGFILE]
                                        [--format FORMAT] [--email EMAIL]
                                        [--retries RETRIES]
                                        [--batchsize BATCHSIZE]
[…]
```

For example, the NCBI taxonomy ID for *Caulobacter* is 75, so all publicly-available *Caulobacter* sequences can be obtained using the command-line:

```bash
$ genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email me@my.email.domain
INFO: genbank_get_genomes_by_taxon.py: Mon Apr 18 17:22:54 2016
INFO: command-line: /Users/lpritc/Virtualenvs/pyani3/bin/genbank_get_genomes_by_taxon.py -o Caulobacter_downloads -v -t 75 -l Caulobacter_downloads.log --email me@my.email.domain
INFO: Namespace(batchsize=10000, email='me@my.email.domain', force=False, format='gbk,fasta', logfile='Caulobacter_downloads.log', noclobber=False, outdirname='Caulobacter_downloads', retries=20, taxon='75', verbose=True)
INFO: Set NCBI contact email to me@my.email.domain
INFO: Creating directory Caulobacter_downloads
INFO: Output directory: Caulobacter_downloads
INFO: Passed taxon IDs: 75
INFO: Entrez ESearch with query: txid75[Organism:exp]
INFO: Entrez ESearch returns 29 assembly IDs
INFO: Identified 29 unique assemblies
INFO: Taxon 75: 29 assemblies
[…]
INFO: Assembly 639581: 271 contigs
INFO: Assembly 233261: 17 contigs
INFO: Assembly 575291: 48 contigs
INFO: Mon Apr 18 17:25:46 2016
INFO: Done.
```

**NOTE:** You must provide a valid email to identify yourself to NCBI for troubleshooting.

The number of attempted retries for each download, and the size of a batch download can be modified. By default, the script will attempt 20 download retries, and obtain sequences in batches of 10,000.

-----

## Method and Output Description

### Average Nucleotide Identity (ANI)

This module calculates Average Nucleotide Identity (ANI) according to one of a number of alternative methods described in, e.g.

- Richter M, Rossello-Mora R (2009) Shifting the genomic gold standard for the prokaryotic species definition. Proc Natl Acad Sci USA 106: 19126-19131. doi:10.1073/pnas.0906412106. (ANI1020, ANIm, ANIb)
- Goris J, Konstantinidis KT, Klappenbach JA, Coenye T, Vandamme P, et al. (2007) DNA-DNA hybridization values and their relationship to whole-genome sequence similarities. Int J Syst Evol Micr 57: 81-91. doi:10.1099/ijs.0.64483-0.

ANI is proposed to be the appropriate *in silico* substitute for DNA-DNA hybridisation (DDH), and so useful for delineating species boundaries. A typical percentage threshold for species boundary in the literature is 95% ANI (e.g. Richter et al. 2009).

All ANI methods follow the basic algorithm:

- Align the genome of organism 1 against that of organism 2, and identify the matching regions.
- Calculate the percentage nucleotide identity of the matching regions, as an average for all matching regions.

Methods differ on: (1) what alignment algorithm is used, and the choice of parameters (this affects the aligned region boundaries); (2) what the input is for alignment (typically either fragments of fixed size, or the most complete assembly available).

- **ANIm**: uses MUMmer (NUCmer) to align the input sequences
- **ANIb**: uses BLASTN+ to align 1020nt fragments of the input sequences
- **ANIblastall**: uses legacy BLASTN to align 10al20nt fragments of the input sequences
- **TETRA**: calculates tetranucleotide frequencies of each input sequence

The algorithms take as input correctly-formatted FASTA multiple sequence files. All sequences for a single organism should be contained in only one sequence file. Although it is possible to provide new labels for each input genome for rendering graphical output, the names of these files are used for identification so it is best to name them sensibly.

Output is written to a named directory. The output files differ depending on the chosen ANI method.

- **ANIm**: MUMmer/NUCmer `.delta` files, describing each pairwise sequence alignment. Output as tab-separated plain text format tables describing: alignment coverage; total alignment lengths; similarity errors; and percentage identity (ANIm).
- **ANIb** and **ANIblastall**: FASTA sequences describing 1020nt fragments of each input sequence; BLAST nucleotide databases - one for each set of fragments; and BLASTN output files (tab-separated tabular format plain text) - one for each pairwise comparison of input sequences. Output as tab-separated plain text tables describing: alignment coverage; total alignment lengths; similarity errors; and percentage identity (ANIb or ANIblastall).
- **TETRA**: Tab-separated plain text files describing the Pearson correlations between Z-score distributions for each tetranucleotide in each input sequence (TETRA).

If graphical output is chosen, the output directory will also contain PDF, PNG and EPS files representing the various output measures as a heatmap with row and column dendrograms. Other output formats (e.g. SVG) can be specified with the `--gformat` argument.

-----

## Licensing

Unless otherwise indicated, all code is subject to the following agreement:

```text
    (c) The James Hutton Institute 2014-2019
    (c) The University of Strathclyde 2019-present
    Author: Leighton Pritchard

    Contact: leighton.pritchard@strath.ac.uk

    Address:
    Leighton Pritchard,
    Strathclyde Institute of Pharmacy and Biomedical Sciences
    161 Cathedral Street
    Glasgow
    G4 0RE,
    Scotland,
    UK

The MIT License

Copyright (c) 2014-2019 The James Hutton Institute
Copyright (c) 2014-present The James Hutton Institute

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
```


# (PAZ) Perception for Autonomous Systems
[![Unit testing](https://github.com/oarriaga/paz/actions/workflows/python-package.yml/badge.svg)](https://github.com/oarriaga/paz/actions/workflows/python-package.yml)
[![PyPI upload](https://github.com/oarriaga/paz/actions/workflows/python-publish.yml/badge.svg)](https://github.com/oarriaga/paz/actions/workflows/python-publish.yml)
[![Publish Website](https://github.com/oarriaga/paz/actions/workflows/publish-website.yml/badge.svg?branch=master)](https://github.com/oarriaga/paz/actions/workflows/publish-website.yml)
[![Downloads](https://static.pepy.tech/personalized-badge/pypaz?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/pypaz)

Hierarchical perception library in Python.

## Selected examples:
PAZ is used in the following examples (links to **real-time demos** and training scripts):

| [Probabilistic 2D keypoints](https://github.com/oarriaga/paz/tree/master/examples/probabilistic_keypoint_estimation)| [6D head-pose estimation](https://github.com/oarriaga/paz/tree/master/examples/pose_estimation)  | [Object detection](https://github.com/oarriaga/paz/tree/master/examples/object_detection)|
|---------------------------|--------------------------| ------------------|
| | | |

| [Emotion classifier](https://github.com/oarriaga/paz/tree/master/examples/face_classification) | [2D keypoint estimation](https://github.com/oarriaga/paz/tree/master/examples/keypoint_estimation)   | [Mask-RCNN (in-progress)](https://github.com/oarriaga/paz/tree/mask_rcnn/examples/mask_rcnn)  |
|---------------------------|--------------------------| -----------------------|
|| | |

|[Semantic segmentation](https://github.com/oarriaga/paz/tree/master/examples/semantic_segmentation) | [Hand pose estimation](https://github.com/oarriaga/paz/tree/master/examples/hand_pose_estimation) |  [2D Human pose estimation](https://github.com/oarriaga/paz/tree/master/examples/human_pose_estimation_2D) |
|---------------------------|-----------------------|-----------------|
| |  | |

| [3D keypoint discovery](https://github.com/oarriaga/paz/tree/master/examples/discovery_of_latent_keypoints)     | [Hand closure detection](https://github.com/oarriaga/paz/tree/master/examples/hand_pose_estimation)  | [6D pose estimation](https://github.com/oarriaga/paz/tree/master/examples/pix2pose) |
|---------------------------|-----------------------| --------------------------|
| | |  |

| [Implicit orientation](https://github.com/oarriaga/paz/tree/master/examples/implicit_orientation_learning)  | [Attention (STNs)](https://github.com/oarriaga/paz/tree/master/examples/spatial_transfomer_networks) | [Haar Cascade detector](https://github.com/oarriaga/paz/tree/master/examples/haar_cascade_detectors) |
|---------------------------|-----------------------|-----------------|
||  |  |

| [Eigenfaces](https://github.com/oarriaga/paz/blob/master/examples/eigenfaces) |[Prototypical Networks](https://github.com/oarriaga/paz/tree/master/examples/prototypical_networks) | [3D Human pose estimation](https://github.com/oarriaga/paz/blob/master/examples/human_pose_estimation_3D) |
|---------------------------|-----------------------|-----------------|
||   |  |

|[MAML](https://github.com/oarriaga/paz/blob/master/examples/maml)| | |
|---------------------------|-----------------------|-----------------|
| | |  |

All models can be re-trained with your own data (except for Mask-RCNN, we are working on it [here](https://github.com/oarriaga/paz/tree/mask_rcnn)).

## Table of Contents

* [Examples](#selected-examples)
* [Installation](#installation)
* [Documentation](#documentation)
* [Hierarchical APIs](#hierarchical-apis)
    * [High-level](#high-level) | [Mid-level](#mid-level) | [Low-level](#low-level)
* [Additional functionality](#additional-functionality)
    * [Implemented models](#models)
* [Motivation](#motivation)
* [Citation](#citation)
* [Funding](#funding)

## Installation
PAZ has only **three** dependencies: [Tensorflow2.0](https://www.tensorflow.org/), [OpenCV](https://opencv.org/) and [NumPy](https://numpy.org/).

To install PAZ with pypi run:
```
pip install pypaz --user
```

## Documentation
Full documentation can be found [https://oarriaga.github.io/paz/](https://oarriaga.github.io/paz/).

## Hierarchical APIs
PAZ can be used with three different API levels which are there to be helpful for the user's specific application.

## High-level
Easy out-of-the-box prediction. For example, for detecting objects we can call the following pipeline:

``` python
from paz.applications import SSD512COCO

detect = SSD512COCO()

# apply directly to an image (numpy-array)
inferences = detect(image)
```

There are multiple high-level functions a.k.a. ``pipelines`` already implemented in PAZ [here](https://github.com/oarriaga/paz/tree/master/paz/pipelines). Those functions are build using our mid-level API described now below.

## Mid-level
While the high-level API is useful for quick applications, it might not be flexible enough for your specific purpose. Therefore, in PAZ we can build high-level functions using our a mid-level API.

### Mid-level: Sequential
If your function is sequential you can construct a sequential function using ``SequentialProcessor``. In the example below we create a data-augmentation pipeline:

``` python
from paz.abstract import SequentialProcessor
from paz import processors as pr

augment = SequentialProcessor()
augment.add(pr.RandomContrast())
augment.add(pr.RandomBrightness())
augment.add(pr.RandomSaturation())
augment.add(pr.RandomHue())

# you can now use this now as a normal function
image = augment(image)
```

You can also add **any function** not only those found in ``processors``. For example we can pass a numpy function to our original data-augmentation pipeline:

``` python
augment.add(np.mean)
```
There are multiple functions a.k.a. ``Processors`` already implemented in PAZ [here](https://github.com/oarriaga/paz/tree/master/paz/processors).

Using these processors we can build more complex pipelines e.g. **data augmentation for object detection**: [``pr.AugmentDetection``](https://github.com/oarriaga/paz/blob/master/paz/pipelines/detection.py#L46)

### Mid-level: Explicit
Non-sequential pipelines can be also build by abstracting ``Processor``. In the example below we build a emotion classifier from **scratch** using our high-level and mid-level functions.

``` python
from paz.applications import HaarCascadeFrontalFace, MiniXceptionFER
import paz.processors as pr

class EmotionDetector(pr.Processor):
    def __init__(self):
        super(EmotionDetector, self).__init__()
        self.detect = HaarCascadeFrontalFace(draw=False)
        self.crop = pr.CropBoxes2D()
        self.classify = MiniXceptionFER()
        self.draw = pr.DrawBoxes2D(self.classify.class_names)

    def call(self, image):
        boxes2D = self.detect(image)['boxes2D']
        cropped_images = self.crop(image, boxes2D)
        for cropped_image, box2D in zip(cropped_images, boxes2D):
            box2D.class_name = self.classify(cropped_image)['class_name']
        return self.draw(image, boxes2D)

detect = EmotionDetector()
# you can now apply it to an image (numpy array)
predictions = detect(image)
```

``Processors`` allow us to easily compose, compress and extract away parameters of functions. However, most processors are build using our low-level API (backend) shown next.

## Low-level

Mid-level processors are mostly built from small backend functions found in: [boxes](https://github.com/oarriaga/paz/blob/master/paz/backend/boxes.py), [cameras](https://github.com/oarriaga/paz/blob/master/paz/backend/camera.py), [images](https://github.com/oarriaga/paz/tree/master/paz/backend/image), [keypoints](https://github.com/oarriaga/paz/blob/master/paz/backend/keypoints.py) and [quaternions](https://github.com/oarriaga/paz/blob/master/paz/backend/quaternion.py).

These functions can found in ``paz.backend``:

``` python
from paz.backend import boxes, camera, image, keypoints, quaternion
```
For example, you can use them in your scripts to load or show images:

``` python
from paz.backend.image import load_image, show_image

image = load_image('my_image.png')
show_image(image)
```

## Additional functionality

* PAZ has [built-in messages](https://github.com/oarriaga/paz/blob/master/paz/abstract/messages.py) e.g. ``Pose6D`` for an easier data exchange with other frameworks such as [ROS](https://www.ros.org/).

* There are custom [callbacks](https://github.com/oarriaga/paz/blob/master/paz/optimization/callbacks.py) e.g. MAP evaluation for object detectors while training.

* PAZ comes with [data loaders](https://github.com/oarriaga/paz/tree/master/paz/datasets) for the multiple datasets:
    [OpenImages](https://github.com/oarriaga/paz/blob/master/paz/datasets/open_images.py), [VOC](https://github.com/oarriaga/paz/blob/master/paz/datasets/voc.py), [YCB-Video](https://github.com/oarriaga/paz/blob/master/examples/object_detection/datasets/ycb_video.py), [FAT](https://github.com/oarriaga/paz/blob/master/paz/datasets/fat.py), [FERPlus](https://github.com/oarriaga/paz/blob/master/paz/datasets/ferplus.py), [FER2013](https://github.com/oarriaga/paz/blob/master/paz/datasets/fer.py), [CityScapes](https://github.com/oarriaga/paz/blob/master/paz/datasets/cityscapes.py).

* We have an automatic [batch creation and dispatching wrappers](https://github.com/oarriaga/paz/blob/master/paz/abstract/sequence.py) for an easy connection between you ``pipelines`` and tensorflow generators. Please look at the [tutorials](https://github.com/oarriaga/paz/tree/master/examples/tutorials) for more information.

### Models

The following models are implemented in PAZ and they can be trained with your own data:

| Task (link to implementation)    |Model (link to paper)  |
|---------------------------:|-----------------------|
|[Object detection](https://github.com/oarriaga/paz/blob/master/paz/models/detection/ssd300.py)|[SSD-300](https://arxiv.org/abs/1512.02325)|
|[Object detection](https://github.com/oarriaga/paz/blob/master/paz/models/detection/ssd512.py)|[SSD-512](https://arxiv.org/abs/1512.02325)|
|[Probabilistic keypoint est.](https://github.com/oarriaga/paz/blob/master/examples/probabilistic_keypoint_estimation/model.py) |[Gaussian Mixture CNN](https://www.robots.ox.ac.uk/~vgg/publications/2018/Neumann18a/)   |
|[Detection and Segmentation](https://github.com/oarriaga/paz/tree/mask_rcnn/examples/mask_rcnn)  |[MaskRCNN (in progress)](https://arxiv.org/abs/1703.06870) |
|[Keypoint estimation](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/hrnet.py)|[HRNet](https://arxiv.org/abs/1908.07919)|
|[Semantic segmentation](https://github.com/oarriaga/paz/blob/master/paz/models/segmentation/unet.py)|[U-NET](https://arxiv.org/abs/1505.04597)|
|[6D Pose estimation](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/keypointnet.py)          |[Pix2Pose](https://arxiv.org/abs/1908.07433)          |
|[Implicit orientation](https://github.com/oarriaga/paz/blob/master/examples/implicit_orientation_learning/model.py)        |[AutoEncoder](https://arxiv.org/abs/1902.01275)            |
|[Emotion classification](https://github.com/oarriaga/paz/blob/master/paz/models/classification/xception.py)       |[MiniXception](https://arxiv.org/abs/1710.07557)           |
|[Discovery of Keypoints](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/keypointnet.py)      |[KeypointNet](https://arxiv.org/abs/1807.03146)            |
|[Keypoint estimation](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/keypointnet.py)  |[KeypointNet2D](https://arxiv.org/abs/1807.03146)|
|[Attention](https://github.com/oarriaga/paz/blob/master/examples/spatial_transfomer_networks/STN.py)                   |[Spatial Transformers](https://arxiv.org/abs/1506.02025)   |
|[Object detection](https://github.com/oarriaga/paz/blob/master/paz/models/detection/haar_cascade.py)            |[HaarCascades](https://link.springer.com/article/10.1023/B:VISI.0000013087.49260.fb)  |
|[2D Human pose estimation](https://github.com/oarriaga/paz/blob/master/paz/models/pose_estimation/higher_hrnet.py)            |[HigherHRNet](https://arxiv.org/abs/1908.10357)  |
|[3D Human pose estimation](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/simplebaselines.py) | [Simple Baseline](https://arxiv.org/abs/1705.03098) |
|[Hand pose estimation](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/detnet.py)            |[DetNet](https://vcai.mpi-inf.mpg.de/projects/2020-cvpr-hands/)  |
|[Hand closure classification](https://github.com/oarriaga/paz/blob/master/paz/models/keypoint/iknet.py)            |[IKNet](https://vcai.mpi-inf.mpg.de/projects/2020-cvpr-hands/)  |
|[Hand detection](https://github.com/oarriaga/paz/blob/master/paz/models/detection/ssd512.py)            |[SSD512](https://arxiv.org/abs/1512.02325)|
|[Few-shot classification](https://github.com/oarriaga/paz/blob/master/paz/models/classification/protonet.py)| [Prototypical Networks](https://arxiv.org/abs/1703.05175)|
|[Few-shot classification](https://github.com/oarriaga/paz/blob/master/examples/maml/maml.py)| [Model Agnostic Meta Learning (MAML)](https://arxiv.org/abs/1703.03400)|

## Motivation
Even though there are multiple high-level computer vision libraries in different deep learning frameworks, I felt there was not a consolidated deep learning library for robot-perception in my framework of choice (Keras).

As a final remark, I would like to mention, that I feel that we might tend to forget the great effort and emotional status behind every (open-source) project.
I feel it's easy to blurry a company name with the individuals behind their work, and we forget that there is someone feeling our criticism and our praise.
Therefore, whatever good code you can find here, is all dedicated to the software-engineers and contributors of open-source projects like Pytorch, Tensorflow and Keras.
You put your craft out there for all of us to use and appreciate, and we ought first to give you our thankful consideration.

## Why the name **PAZ**?
* The name PAZ satisfies it's theoretical definition by having it as an acronym for **Perception for Autonomous Systems** where the letter S is replaced for Z in order to indicate that for "System" we mean almost anything i.e. Z being a classical algebraic variable to indicate an unknown element.

## Tests and coverage
Continuous integration is managed trough [github actions](https://github.com/features/actions) using [pytest](https://docs.pytest.org/en/stable/).
You can then check for the tests by running:
```
pytest tests
```
Test coverage can be checked using [coverage](https://coverage.readthedocs.io/en/coverage-5.2.1/).
You can install coverage by calling: `pip install coverage --user`
You can then check for the test coverage by running:
```
coverage run -m pytest tests/
coverage report -m
```

## Citation
If you use PAZ please consider citating it. You can also find our paper here [https://arxiv.org/abs/2010.14541](https://arxiv.org/abs/2010.14541).

```BibTeX
@misc{arriaga2020perception,
      title={Perception for Autonomous Systems (PAZ)},
      author={Octavio Arriaga and Matias Valdenegro-Toro and Mohandass Muthuraja and Sushma Devaramani and Frank Kirchner},
      year={2020},
      eprint={2010.14541},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

## Funding
PAZ is currently developed in the [Robotics Group](https://robotik.dfki-bremen.de/de/ueber-uns/universitaet-bremen-arbeitsgruppe-robotik.html) of the [University of Bremen](https://www.uni-bremen.de/), together with the [Robotics Innovation Center](https://robotik.dfki-bremen.de/en/startpage.html) of the **German Research Center for Artificial Intelligence** (DFKI) in **Bremen**.
PAZ has been funded by the German Federal Ministry for Economic Affairs and Energy and the [German Aerospace Center](https://www.dlr.de/DE/Home/home_node.html) (DLR).
PAZ been used and/or developed in the projects [TransFIT](https://robotik.dfki-bremen.de/en/research/projects/transfit.html) and [KiMMI-SF](https://robotik.dfki-bremen.de/en/research/projects/kimmi-sf/).


|zfit_logo|

*******************************
zfit: scalable pythonic fitting
*******************************

.. image:: https://scikit-hep.org/assets/images/Scikit--HEP-Affiliated-blue.svg
   :target: https://scikit-hep.org

.. image:: https://img.shields.io/badge/DOI-10.1016%2Fj.softx.2020.100508-yellow
   :target: https://www.sciencedirect.com/science/article/pii/S2352711019303851
   :alt: DOI 10.1016/j.softx.2020.100508

.. image:: https://img.shields.io/pypi/pyversions/zfit
   :target: https://pypi.org/project/zfit/
   :alt: PyPI - Python Version

.. image:: https://img.shields.io/pypi/v/zfit.svg
   :target: https://pypi.python.org/pypi/zfit

.. image:: https://img.shields.io/conda/vn/conda-forge/zfit
   :target: https://anaconda.org/conda-forge/zfit
   :alt: conda-forge

.. image:: https://img.shields.io/spack/v/py-zfit
   :target: https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-zfit/package.py

.. image:: https://github.com/zfit/zfit/workflows/build/badge.svg
   :target: https://github.com/zfit/zfit/actions

.. |zfit_logo| image:: docs/images/zfit-logo_hires.png
   :target: https://github.com/zfit/zfit
   :alt: zfit logo

.. |scikit-hep_logo| image:: docs/images/scikit-hep-logo_168x168.png
   :target: https://scikit-hep.org/affiliated
   :alt: scikit-hep logo

zfit is a highly scalable and customizable model manipulation and likelihood fitting library. It uses the same computational backend as
`TensorFlow <https://www.tensorflow.org/>`_ and is optimised for simple and direct manipulation of probability density functions. The project is affiliated with
and well integrated into `Scikit-HEP <https://scikit-hep.org/>`_, the HEP Python ecosystem.

- **Tutorials**: `Interactive introduction and tutorials <https://zfit-tutorials.readthedocs.io/en/latest/>`_
- **Quick start**: `Example scripts <examples>`_
- **Documentation**: See `stable documentation`_ or `latest documentation`_
- **Questions**: see the `FAQ <https://github.com/zfit/zfit/wiki/FAQ>`_,
  `ask on StackOverflow <https://stackoverflow.com/questions/ask?tags=zfit>`_ (with the **zfit** tag) or `contact`_ us directly.
- **Physics, HEP**: `zfit-physics <https://github.com/zfit/zfit-physics>`_ is the place to contribute and find more HEP
  related content
- **Statistical inference**: `hepstats <https://github.com/scikit-hep/hepstats>`_ for limits, CI, sWeights and more

If you use zfit in **research**, please
consider `citing <https://www.sciencedirect.com/science/article/pii/S2352711019303851>`_.

*N.B.*: zfit is currently in *beta stage*, so while most core parts are established,
some may still be missing and bugs may be encountered.
It is, however, mostly ready for production, and is being used in analyses projects.
If you want to use it for your project and you are not sure if all the needed functionality is there,
feel free to `contact`_ us.

Installation
=============

zfit is available on pip and conda-forge. To install it (recommended: use a virtual/conda env!) with all the dependencies (minimizers, uproot, ...), use

.. code-block:: bash

   pip install -U zfit[all]

(the ``-U`` just indicates to upgrade zfit, in case you have it already installed)
or for minimal dependencies

.. code-block:: bash

   pip install zfit

For conda/mamba, use

.. code-block:: bash

   conda install -c conda-forge zfit

How to use
==========

While the zfit library provides a model fitting and sampling framework for a broad list of applications,
we will illustrate its main features with a simple example by fitting a Gaussian distribution with an unbinned
likelihood fit and a parameter uncertainty estimation.

Example in short
----------------
.. code-block:: python

    obs = zfit.Space('x', limits=(-10, 10))

    # create the model
    mu    = zfit.Parameter("mu"   , 2.4, -1, 5)
    sigma = zfit.Parameter("sigma", 1.3,  0, 5)
    gauss = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma)

    # load the data
    data_np = np.random.normal(size=10000)
    data = zfit.Data.from_numpy(obs=obs, array=data_np)

    # build the loss
    nll = zfit.loss.UnbinnedNLL(model=gauss, data=data)

    # minimize
    minimizer = zfit.minimize.Minuit()
    result = minimizer.minimize(nll).update_params()

    # calculate errors
    param_errors = result.hesse()

This follows the zfit workflow

.. image:: docs/images/zfit_workflow_v2.png
    :alt: zfit workflow

Full explanation
----------------

The default space (e.g. normalization range) of a PDF is defined by an *observable space*, which is created using the ``zfit.Space`` class:

.. code-block:: python

    obs = zfit.Space('x', limits=(-10, 10))

To create a simple Gaussian PDF, we define its parameters and their limits using the ``zfit.Parameter`` class.

.. code-block:: python

  # syntax: zfit.Parameter("any_name", value, lower, upper)
    mu    = zfit.Parameter("mu"   , 2.4, -1, 5)
    sigma = zfit.Parameter("sigma", 1.3,  0, 5)
    gauss = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sigma)

For simplicity, we create the dataset to be fitted starting from a numpy array, but zfit allows for the use of other sources such as ROOT files:

.. code-block:: python

    mu_true = 0
    sigma_true = 1
    data_np = np.random.normal(mu_true, sigma_true, size=10000)
    data = zfit.Data.from_numpy(obs=obs, array=data_np)

Fits are performed in three steps:

1. Creation of a loss function, in our case a negative log-likelihood.
2. Instantiation of our minimiser of choice, in the example the ``Minuit``.
3. Minimisation of the loss function.

.. code-block:: python

    # Stage 1: create an unbinned likelihood with the given PDF and dataset
    nll = zfit.loss.UnbinnedNLL(model=gauss, data=data)

    # Stage 2: instantiate a minimiser (in this case a basic minuit)
    minimizer = zfit.minimize.Minuit()

    # Stage 3: minimise the given negative log-likelihood
    result = minimizer.minimize(nll).update_params()

Errors are calculated with a further function call to avoid running potentially expensive operations if not needed:

.. code-block:: python

    param_errors = result.hesse()

Once we've performed the fit and obtained the corresponding uncertainties, we can examine the fit results:

.. code-block:: python

    print("Function minimum:", result.fmin)
    print("Converged:", result.converged)
    print("Full minimizer information:", result)

    # Information on all the parameters in the fit
    params = result.params
    print(params)

    # Printing information on specific parameters, e.g. mu
    print("mu={}".format(params[mu]['value']))

And that's it!
For more details and information of what you can do with zfit, checkout the `latest documentation`_.

Why?
====

The basic idea behind zfit is to offer a Python oriented alternative to the very successful RooFit library
from the `ROOT <https://root.cern.ch/>`_ data analysis package that can integrate with the other packages
that are part if the scientific Python ecosystem.
Contrary to the monolithic approach of ROOT/RooFit, the aim of zfit is to be light and flexible enough t
o integrate with any state-of-art tools and to allow scalability going to larger datasets.

These core ideas are supported by two basic pillars:

- The skeleton and extension of the code is minimalist, simple and finite:
  the zfit library is exclusively designed for the purpose of model fitting and sampling with no attempt to extend its
  functionalities to features such as statistical methods or plotting.

- zfit is designed for optimal parallelisation and scalability by making use of TensorFlow as its backend.
  The use of TensorFlow provides crucial features in the context of model fitting like taking care of the
  parallelisation and analytic derivatives.

Prerequisites
=============

``zfit`` works with Python versions 3.9 and above.
The main dependency is `tensorflow <https://www.tensorflow.org/>`_: zfit follows a close version compatibility with TensorFlow.

For a full list of all dependencies, check the `requirements <requirements.txt>`_.

Installing
==========

zfit is currently available on `PyPI <https://pypi.org/project/zfit/>`_ and `conda-forge <https://anaconda.org/conda-forge/zfit>`_.

If possible, use a conda or virtual environment and do:

.. code-block:: console

    $ pip install zfit

For the newest development version, you can install the version from git with

.. code-block:: console

   $ pip install git+https://github.com/zfit/zfit

Contributing
============

Any idea of how to improve the library? Or interested to write some code?
Contributions are always welcome, please have a look at the `Contributing guide`_.

.. _Contributing guide: CONTRIBUTING.rst

Contact
========

You can contact us directly:
 - via e-mail: zfit@physik.uzh.ch
 - join our `Gitter channel <https://gitter.im/zfit/zfit>`_

Original Authors
================

| Jonas Eschle <jonas.eschle@cern.ch>
| Albert Puig <albert.puig@cern.ch>
| Rafael Silva Coutinho <rsilvaco@cern.ch>

See here for `all authors and contributors`_

..  _all authors and contributors: AUTHORS.rst

Acknowledgements
================

zfit has been developed with support from the University of Zurich and the Swiss National Science Foundation (SNSF) under contracts 168169 and 174182.

The idea of zfit is inspired by the `TensorFlowAnalysis <https://gitlab.cern.ch/poluekt/TensorFlowAnalysis>`_ framework
developed by Anton Poluektov and `TensorProb <https://github.com/tensorprob/tensorprob>`_ by Chris Burr and Igor Babuschkin
using the TensorFlow open source library and more libraries.

.. _`latest documentation`: https://zfit.readthedocs.io/en/latest/
.. _`stable documentation`: https://zfit.readthedocs.io/en/stable/


## HOKUSAI [![CircleCI](https://circleci.com/gh/artsy/hokusai/tree/main.svg?style=svg)](https://circleci.com/gh/artsy/hokusai/tree/main)

Hokusai is a Docker + Kubernetes CLI for application developers.

Hokusai "dockerizes" applications and manages their lifecycle throughout development, testing, and release cycles.

Hokusai wraps calls to [Kubectl](https://kubernetes.io/), [Docker](https://www.docker.com/), [Docker-Compose](https://docs.docker.com/compose/) and [Git](https://git-scm.com/) with a CLI, and defines a CI workflow.

Hokusai currently only supports Kubernetes deployments on AWS, configured to pull from ECS container repositories (ECR), although other providers may be added in the future.

### Why Hokusai?

At [Artsy](http://www.artsy.net), as we began working with Kubernetes, while impressed with its design, capabilities, and flexibility, we were in need of tooling we could deliver to agile development teams that addressed the day-to-day tasks of application development, delivery, introspection and maintenance, while providing a clean and uncomplicated interface.

Transitioning teams to the Docker / Kubernetes ecosystem can be intimidating, and comes with a steep learning curve. We set out to create a Heroku-like CLI that would shepherd the application developer into the ecosystems of Docker and Kubernetes, and while introducing new tooling and concepts, outlining a clear practice for dependency management, local development, testing and CI, image repository structure, deployment and orchestration.

## Installation

### MacOS

We recommend installing via Homebrew:

```
$ brew update
$ brew tap artsy/formulas
$ brew install hokusai
```

If you previously installed Hokusai via an alternate installation method, you may need to force the `link` step.

```
$ brew link --overwrite hokusai
```

If you previously installed Hokusai via Pip, you may want to first uninstall it:

```
$ pip uninstall hokusai
```

### Linux

```
curl -sSL https://raw.githubusercontent.com/artsy/hokusai/main/get-hokusai.sh | sudo bash
```

Note: This method installs Hokusai to `/usr/local/bin/hokusai`.

### Pip

Hokusai can be installed via Pip, on MacOS or Linux. If you do so, please first go through [Pyenv](#Pyenv), [Python](#Python), and [Virtualenv](#Virtualenv) steps.

Python 3.7+ is required.

```
pip install hokusai
```

Note: If Pip fails at upgrading your system Python packages, try:

```
pip install hokusai --ignore-installed
```

### Docker

We also maintain [Hokusai Docker images](https://hub.docker.com/r/artsy/hokusai) for running Hokusai in Docker.

### Github

Release artifacts are available on [Github](https://github.com/artsy/hokusai/releases).

### AWS S3

Release artifacts are also available in AWS S3. You can use this [convenience script](get-hokusai.sh) or Curl to fetch them.

### A note on Python 2.x

Hokusai currently supports Python 3.7+ only. The last version that supported Python 2.x was [v0.5.18](https://github.com/artsy/hokusai/tree/v0.5.18).

## Setup

We assume that your org admin has already set up a Kubernetes cluster and an AWS infrastructure, and that your local environment has Git, Docker, and Docker-Compose installed. Perform the following steps to get Hokusai working:

1. Ensure your org admin has completed the steps mentioned in [Administering Hokusai](./docs/Administering_Hokusai.md).

2. Configure your local environment with [AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#configuring-credentials).

3. Run [hokusai configure command](docs/Command_Reference.md#configuring-hokusai-for-your-organization).

4. Optionally, enable Bash autocompletion:

    ```
    eval "$(_HOKUSAI_COMPLETE=source hokusai)"
    ```

## Getting Started

Once Hokusai is configured, you can start using it on a project. Please see [Getting Started](./docs/Getting_Started.md).

## Command Reference

A full command reference can be found in [Command Reference](./docs/Command_Reference.md).

## Review Apps

Hokusai can be used to simplify the process of spinning up a "review app" instance of your project, based on a feature branch or pull request.

Full details are in the [Review App reference](./docs/Review_Apps.md).

## Hokusai Files

A descripton of all the [files used by Hokusai](docs/hokusai_files.md).

## Developing Hokusai

To work on Hokusai itself, please set up:

### Pyenv

We recommend using [Pyenv](https://github.com/pyenv/pyenv) to install the correct version of Python. For a tutorial of Pyenv, see [this guide](https://realpython.com/intro-to-pyenv/).

When installing on MacOS, please make sure to use brew-installed `openssl` and `readline` libraries, and xcode-installed `zlib` library. And make sure these libraries are correctly linked. Like so:

```
brew install openssl readline zlib

echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.bash_profile
echo 'export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"' >> ~/.bash_profile
echo 'export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"' >> ~/.bash_profile
echo 'export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"' >> ~/.bash_profile
```

### Python

Hokusai is currently tested on Python 3.10 so we recommend using that Python version.

If you use Pyenv to install Python, you should see an output similar to this:

```
$ pyenv install 3.10
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.13.tar.xz...
-> https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tar.xz
Installing Python-3.10.13...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use ncurses from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.13 to /Users/jxu/.pyenv/versions/3.10.13
```

With the desired Python version installed, activate it globally:

```
pyenv global 3.10
```

Note: If you want to create a PyInstaller distribution, Python must be installed with development libraries. Use the environment variable `PYTHON_CONFIGURE_OPTS="--enable-framework"` on Darwin and `PYTHON_CONFIGURE_OPTS="--enable-shared"` on Linux when running `pyenv install`.

### Virtualenv

We recommend using a virtual environment to isolate Hokusai's dependencies from that of other projects on your local environment.

The Pyenv install comes with [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) which can be used to create virtual environments.

### Poetry

Use [Poetry](https://python-poetry.org/) to install Hokusai's dependencies as well as Hokusai itself in [editable mode](https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#editable-installs). Please see [this guide](https://python-poetry.org/docs/basic-usage/) for working with Poetry.

Install Poetry:

```
pip install --upgrade pip
pip install poetry
```

Install dependencies and Hokusai in editable mode:

```
poetry install
```

To update dependencies:

```
poetry lock
```

## Testing

### Install Minikube

[Minikube](https://minikube.sigs.k8s.io/docs/start/) is used for integration tests.

```
brew install minikube
minikube start --kubernetes-version=<version of your Kubernetes clusters, example: v1.2.3>
```

### Run tests

To run unit tests:

```
make test
```

To run integration tests:

```
make integration-local
```

Only specific modules, TestClasses, or even methods:

```
python -m unittest test.unit.test_module.TestClass.test_method
```

Tip: Set `DEBUG=1` environment variable to print boto logging

## Distributing Hokusai

### Beta Release

Merging a branch into `main` automatically creates a beta version useful for testing. A [script](./scripts/update_version_file.sh) that runs in CI automatically generates a Beta version number and writes it into [VERSION](./hokusai/VERSION) file. The version number is based on the combination of the latest canonical version found in [RELEASE_VERSION](./hokusai/RELEASE_VERSION) file and the latest Git commit in `main` branch. [VERSION](./hokusai/VERSION) file in version control has just a dummy number (e.g. 999.999.999).

To install the beta:

#### MacOS

```
$ brew uninstall hokusai
$ brew uninstall hokusai-beta
$ brew update
$ brew tap artsy/formulas
$ brew install hokusai-beta
```

#### Linux

```
curl -sSL https://raw.githubusercontent.com/artsy/hokusai/main/get-hokusai.sh | sudo bash -s beta
```

### Official Release

To create an official release, such as `v1.2.3`, perform the following:

- Create a branch named `prepare-v1.2.3` and make the following changes:
  - Bump canonical version in [RELEASE_VERSION](./hokusai/RELEASE_VERSION) file.
  - Upate [CHANGELOG](./CHANGELOG.md).
  - Open a PR to merge into `main`. Please see [past PRs](https://github.com/artsy/hokusai/pulls?q=is%3Apr+Release+is%3Aclosed+%22prepare+version%22) for example.

- [Open a PR](https://github.com/artsy/hokusai/compare/release...main?expand=1) to merge `main` into `release`. Please see [past PRs](https://github.com/artsy/hokusai/pulls?q=is%3Apr+is%3Aclosed+%22release+version%22) for example.

A CI [script](./scripts/update_version_file.sh) will copy the version in [RELEASE_VERSION](./hokusai/RELEASE_VERSION) file to [VERSION](./hokusai/VERSION) file.

## The Name

The project is named for the great Japanese artist [Katsushika Hokusai](https://www.artsy.net/article/artsy-editorial-7-things-hokusai-creator-great-wave) (1760-1849).

## About Artsy

This project is the work of engineers at [Artsy][footer_website], the world's
leading and largest online art marketplace and platform for discovering art.
One of our core [Engineering Principles][footer_principles] is being [Open
Source by Default][footer_open] which means we strive to share as many details
of our work as possible.

You can learn more about this work from [our blog][footer_blog] and by following
[@ArtsyOpenSource][footer_twitter] or explore our public data by checking out
[our API][footer_api]. If you're interested in a career at Artsy, read through
our [job postings][footer_jobs]!

[footer_website]: https://www.artsy.net/
[footer_principles]: https://github.com/artsy/README/blob/main/culture/engineering-principles.md
[footer_open]: https://github.com/artsy/README/blob/main/culture/engineering-principles.md
[footer_blog]: https://artsy.github.io/
[footer_twitter]: https://twitter.com/ArtsyOpenSource
[footer_api]: https://developers.artsy.net/
[footer_jobs]: https://www.artsy.net/jobs