# Zulip Terminal - [Zulip](https://zulip.com)'s official terminal client

[Recent changes](https://github.com/zulip/zulip-terminal/blob/main/CHANGELOG.md) | [Configuration](#Configuration) | [Hot Keys](https://github.com/zulip/zulip-terminal/blob/main/docs/hotkeys.md) | [FAQs](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md) | [Development](#contributor-guidelines) | [Tutorial](https://github.com/zulip/zulip-terminal/blob/main/docs/getting-started.md)

[![Chat with us!](https://img.shields.io/badge/Zulip-chat_with_us!-brightgreen.svg)](https://github.com/zulip/zulip-terminal/blob/main/README.md#chat-with-fellow-users--developers)
[![PyPI](https://img.shields.io/pypi/v/zulip-term.svg)](https://pypi.python.org/pypi/zulip-term)
[![Python Versions](https://img.shields.io/pypi/pyversions/zulip-term.svg)](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md#what-python-versions-are-supported)
[![Python Implementations](https://img.shields.io/pypi/implementation/zulip-term.svg)](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md#what-python-implementations-are-supported)
[![OS Platforms](https://img.shields.io/static/v1?label=OS&message=Linux%20%7C%20WSL%20%7C%20macOS%20%7C%20Docker&color=blueviolet)](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md#what-operating-systems-are-supported)

[![GitHub Actions - Linting & tests](https://github.com/zulip/zulip-terminal/workflows/Linting%20%26%20tests/badge.svg?branch=main)](https://github.com/zulip/zulip-terminal/actions?query=workflow%3A%22Linting+%26+tests%22+branch%3Amain)
[![Coverage status](https://img.shields.io/codecov/c/github/zulip/zulip-terminal/main.svg)](https://app.codecov.io/gh/zulip/zulip-terminal/branch/main)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v1.json)](https://github.com/charliermarsh/ruff)

![Screenshot](https://user-images.githubusercontent.com/9568999/169362381-3c2b19e2-2176-4ad0-bd41-604946432512.png)

## About

Zulip Terminal is the official terminal client for Zulip, providing a
[text-based user interface (TUI)](https://en.wikipedia.org/wiki/Text-based_user_interface).

Specific aims include:
* Providing a broadly similar user experience to the Zulip web client,
  ultimately supporting all of its features
* Enabling all actions to be achieved through the keyboard (see
  [Hot keys](https://github.com/zulip/zulip-terminal/blob/main/docs/hotkeys.md))
* Exploring alternative user interface designs suited to the display and input
  constraints
* Supporting a wide range of platforms and terminal emulators
* Making best use of available rows/columns to scale from 80x24 upwards (see
  [Small terminal notes](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md#how-small-a-size-of-terminal-is-supported))

Learn how to use Zulip Terminal with our
[Tutorial](https://github.com/zulip/zulip-terminal/blob/main/docs/getting-started.md).

### Feature status

We consider the client to already provide a fairly stable moderately-featureful
everyday-user experience.

The current development focus is on improving aspects of everyday usage which
are more commonly used - to reduce the need for users to temporarily switch to
another client for a particular feature.

Current limitations which we expect to only resolve over the long term include support for:
* All operations performed by users with extra privileges (owners/admins)
* Accessing and updating all settings
* Using a mouse/pointer to achieve all actions
* An internationalized UI

#### Intentional differences

The terminal client currently has a number of intentional differences to the Zulip web client:
- Additional and occasionally *different*
  [Hot keys](https://github.com/zulip/zulip-terminal/blob/main/docs/hotkeys.md)
  to better support keyboard-only navigation; other than directional movement
  these also include:
  - z - zoom in/out, between streams & topics, or all direct messages & specific conversations
  - t - toggle view of topics for a stream in left panel
    (**later adopted for recent topics in web client**)
  - # - narrow to messages in which you're mentioned (@ is already used)
  - f - narrow to messages you've starred (are **f**ollowing)
- Not marking additional messages read when the end of a conversation is
  visible
  ([FAQ entry](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md#when-are-messages-marked-as-having-been-read))
- Emoji and reactions are rendered as text only, for maximum terminal/font
  compatibility
- Footlinks - footnotes for links (URLs) - make messages readable, while
  retaining a list of links to cross-reference
- Content previewable in the web client, such as images, are also stored as
  footlinks

#### Feature queries?

For queries on missing feature support please take a look at the
[Frequently Asked Questions (FAQs)](https://github.com/zulip/zulip-terminal/blob/main/docs/FAQ.md),
our open [Issues](https://github.com/zulip/zulip-terminal/issues/), or
[chat with users & developers](#chat-with-fellow-users--developers) online at
the Zulip Community server!

## Installation

We recommend installing in a dedicated python virtual environment (see below)
or using an automated option such as [pipx](https://pypi.python.org/pypi/pipx)

* **Stable releases** - These are available on PyPI as the package
  [zulip-term](https://pypi.python.org/pypi/zulip-term)

  To install, run a command like: `pip3 install zulip-term`

* **Latest (git) versions** - The latest development version can be installed
  from the git repository `main` branch

  To install, run a command like:
  `pip3 install git+https://github.com/zulip/zulip-terminal.git@main`

We also provide some sample Dockerfiles to build docker images in
[docker/](https://github.com/zulip/zulip-terminal/tree/main/docker).

### Installing into an isolated Python virtual environment

With the python 3.6+ required for running, the following should work on most
systems:
1. `python3 -m venv zt_venv`
   (creates a virtual environment named `zt_venv` in the current directory)
2. `source zt_venv/bin/activate`
   (activates the virtual environment; this assumes a bash-like shell)
3. Run one of the install commands above.

If you open a different terminal window (or log-off/restart your computer),
you'll need to run **step 2** of the above list again before running
`zulip-term`, since that activates that virtual environment.
You can read more about virtual environments in the
[Python 3 library venv documentation](https://docs.python.org/3/library/venv.html).

### Keeping your install up to date

Note that there is no automatic-update system, so please track the update
locations relevant to your installation version:

**Stable releases**

Before upgrading, we recommend you check the
[Changes in recent releases](https://github.com/zulip/zulip-terminal/blob/main/CHANGELOG.md)
so you are aware of any important changes between releases.

- These are now announced in the
[#**announce>terminal releases**](https://chat.zulip.org/#narrow/stream/1-announce/topic/terminal.20releases)
topic on the Zulip Community server (https://chat.zulip.org), which is visible
without an account.

  If you wish to receive emails when updates are announced, you are welcome to
sign up for an account on this server, which will enable you to enable email
notifications for the **#announce** stream
([help article](https://zulip.com/help/using-zulip-via-email),
[notifications settings on chat.zulip.org](https://chat.zulip.org/#settings/notifications)).

- You can also customize your GitHub Watch setting on the project page to include releases.

- PyPI provides a
[RSS Release feed](https://pypi.org/rss/project/zulip-term/releases.xml), and
various other services track this information.

**Latest (git) versions**

Versions installed from the `main` git branch will also not update
automatically - the 'latest' refers to the status at the point of installation.

This also applies to other source or development installs
(eg. https://aur.archlinux.org/packages/python-zulip-term-git/).

Therefore, upgrade your package using the command above, or one pertinent to
your package system (eg. Arch).

While the `main` branch is intended to remain stable, if upgrading between two
arbitrary 'latest' versions, please be aware that **changes are not summarized**,
though our commit log should be very readable.

## Running for the first time

Upon first running `zulip-term` it looks for a `zuliprc` file, by default in
your home directory, which contains the details to log into a Zulip server.

If it doesn't find this file, you have two options:

1. `zulip-term` will prompt you for your server, email and password, and create
   a `zuliprc` file for you in that location

   **NOTE:** If you use Google, Github or another external authentication to
   access your Zulip organization then you likely won't have a password set and
   currently need to create one to use zulip-terminal.
   - If your organization is on Zulip cloud, you can visit
   https://zulip.com/accounts/go?next=/accounts/password/reset to create a new
   password for your account.
   - For self-hosted servers please go to your equivalent of
   `<Zulip server URL>/accounts/password/reset/` to create a new password for
   your account (eg: https://chat.zulip.org/accounts/password/reset/).

2. Each time you run `zulip-term`, you can specify the path to an alternative
   `zuliprc` file using the `-c` or `--config-file` options, eg. `$ zulip-term -c
   /path/to/zuliprc`

   A `.zuliprc` file corresponding to your account on a particular Zulip server
   can be downloaded via Web or Desktop applications connected to that server.
   In recent versions this can be found in your **Personal settings** in the
   **Account & privacy** section, under **API key** as 'Show/change your API key'.

   If this is your only Zulip account, you may want to move and rename this
   file to the default file location above, or rename it to something more
   memorable that you can pass to the `---config-file` option.
   **This `.zuliprc` file gives you all the permissions you have as that user.**

   Similar `.zuliprc files` can be downloaded from the **Bots** section for any
   bots you have set up, though with correspondingly limited permissions.

**NOTE:** If your server uses self-signed certificates or an insecure
connection, you will need to add extra options to the `zuliprc` file manually -
see the documentation for the [Zulip python module](https://pypi.org/project/zulip/).

We suggest running `zulip-term` using the `-e` or `--explore` option (in
explore mode) when you are trying Zulip Terminal for the first time, where we
intentionally do not mark messages as read. Try following along with our
[Tutorial](https://github.com/zulip/zulip-terminal/blob/main/docs/getting-started.md)
to get the hang of things.

## Configuration

The `zuliprc` file contains two sections:
- an `[api]` section with information required to connect to your Zulip server
- a `[zterm]` section with configuration specific to `zulip-term`

A file with only the first section can be auto-generated in some cases by
`zulip-term`, or you can download one from your account on your server (see
above). Parts of the second section can be added and adjusted in stages when
you wish to customize the behavior of `zulip-term`.

The example below, with dummy `[api]` section contents, represents a working
configuration file with all the default compatible `[zterm]` values uncommented
and with accompanying notes:
```
[api]
email=example@example.com
key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
site=https://example.zulipchat.com

[zterm]
## Theme: available themes can be found by running `zulip-term --list-themes`, or in docs/FAQ.md
theme=zt_dark

## Autohide: set to 'autohide' to hide the left & right panels except when they're focused
autohide=no_autohide

## Exit Confirmation: set to 'disabled' to exit directly with no warning popup first
exit_confirmation=enabled

## Footlinks: set to 'disabled' to hide footlinks; 'enabled' will show the first 3 per message
## For more flexibility, comment-out this value, and un-comment maximum-footlinks below
footlinks=enabled
## Maximum-footlinks: set to any value 0 or greater, to limit footlinks shown per message
# maximum-footlinks=3

## Notify: set to 'enabled' to display notifications (see elsewhere for configuration notes)
notify=disabled

## Color-depth: set to one of 1 (for monochrome), 16, 256, or 24bit
color-depth=256
```

> **NOTE:** Most of these configuration settings may be specified on the
command line when `zulip-term` is started; `zulip-term -h` or `zulip-term --help`
will give the full list of options.

### Notifications

Note that notifications are not currently supported on WSL; see
[#767](https://github.com/zulip/zulip-terminal/issues/767).

#### Linux

The following command installs `notify-send` on Debian based systems, similar
commands can be found for other linux systems as well.
```
sudo apt-get install libnotify-bin
```

#### OSX

No additional package is required to enable notifications in OS X.
However to have a notification sound, set the following variable (based on your
type of shell).
The sound value (here Ping) can be any one of the `.aiff` files found at
`/System/Library/Sounds` or `~/Library/Sounds`.

*Bash*
```
echo 'export ZT_NOTIFICATION_SOUND=Ping' >> ~/.bash_profile
source ~/.bash_profile
```
*ZSH*
```
echo 'export ZT_NOTIFICATION_SOUND=Ping' >> ~/.zshenv
source ~/.zshenv
```

### Copy to clipboard

Zulip Terminal allows users to copy certain texts to the clipboard via a Python
module, [`Pyperclip`](https://pypi.org/project/pyperclip/).
This module makes use of various system packages which may or may not come with
the OS.
The "Copy to clipboard" feature is currently only available for copying Stream
email, from the [Stream information popup](docs/hotkeys.md#stream-list-actions).

#### Linux

On Linux, this module makes use of `xclip` or `xsel` commands, which should
already come with the OS.
If none of these commands are installed on your system, then install any ONE
using:
```
sudo apt-get install xclip [Recommended]
```
OR
```
sudo apt-get install xsel
```

#### OSX and WSL

No additional package is required to enable copying to clipboard.

## Chat with fellow users & developers!

While Zulip Terminal is designed to work with any Zulip server, the main
contributors are present on the Zulip Community server at
https://chat.zulip.org, with most conversation in the
[**#zulip-terminal** stream](https://chat.zulip.org/#narrow/stream/206-zulip-terminal).

You are welcome to view conversations in that stream using the link above, or
sign up for an account and chat with us - whether you are a user or developer!

We aim to keep the Zulip community friendly, welcoming and productive, so if
participating, please respect our
[Community norms](https://zulip.com/development-community/#community-norms).

### Notes more relevant to the **#zulip-terminal** stream

These are a subset of the **Community norms** linked above, which are more
relevant to users of Zulip Terminal: those more likely to be in a text
environment, limited in character rows/columns, and present in this one smaller
stream.

* **Prefer text in [code blocks](https://zulip.com/help/code-blocks), instead of screenshots**

  Zulip Terminal supports downloading images, but there is no guarantee that
  users will be able to view them.

  *Try Meta+m to see example content formatting, including code blocks*

* **Prefer [silent mentions](https://zulip.com/help/mention-a-user-or-group#silently-mention-a-user)
  over regular mentions - or avoid mentions entirely**

  With Zulip's topics, the intended recipient can often already be clear.
  Experienced members will be present as their time permits - responding
  to messages when they return - and others may be able to assist before then.

  (Save [regular mentions](https://zulip.com/help/mention-a-user-or-group#mention-a-user-or-group_1)
  for those who you do not expect to be present on a regular basis)

  *Try Ctrl+f/b to cycle through autocompletion in message content, after typing `@_` to specify a silent mention*

* **Prefer trimming [quote and reply](https://zulip.com/help/quote-and-reply)
  text to only the relevant parts of longer messages - or avoid quoting entirely**

  Zulip's topics often make it clear which message you're replying to. Long
  messages can be more difficult to read with limited rows and columns of text,
  but this is worsened if quoting an entire long message with extra content.

  *Try > to quote a selected message, deleting text as normal when composing a message*

* **Prefer a [quick emoji reaction](https://zulip.com/help/emoji-reactions)
to show agreement instead of simple short messages**

  Reactions take up less space, including in Zulip Terminal, particularly
  when multiple users wish to respond with the same sentiment.

  *Try + to toggle thumbs-up (+1) on a message, or use : to search for other reactions*

## Contributor Guidelines

Zulip Terminal is being built by the awesome [Zulip](https://zulip.com/team) community.

To be a part of it and to contribute to the code, feel free to work on any
[issue](https://github.com/zulip/zulip-terminal/issues) or propose your idea on
[#zulip-terminal](https://chat.zulip.org/#narrow/stream/206-zulip-terminal).

For commit structure and style, please review the [Commit Style](#commit-style)
section below.

If you are new to `git` (or not!), you may benefit from the
[Zulip git guide](http://zulip.readthedocs.io/en/latest/git/index.html).
When contributing, it's important to note that we use a **rebase-oriented
workflow**.

A simple
[tutorial](https://github.com/zulip/zulip-terminal/blob/main/docs/developer-feature-tutorial.md)
is available for implementing the `typing` indicator.
Follow it to understand how to implement a new feature for zulip-terminal.

You can of course browse the source on GitHub & in the source tree you
download, and check the
[source file overview](https://github.com/zulip/zulip-terminal/blob/main/docs/developer-file-overview.md)
for ideas of how files are currently arranged.

### Urwid

Zulip Terminal uses [urwid](http://urwid.org/) to render the UI components in
terminal.
Urwid is an awesome library through which you can render a decent terminal UI
just using python.
[Urwid's Tutorial](http://urwid.org/tutorial/index.html) is a great place to
start for new contributors.

### Getting Zulip Terminal code and connecting it to upstream

First, fork the `zulip/zulip-terminal` repository on GitHub
([see how](https://docs.github.com/en/get-started/quickstart/fork-a-repo))
and then clone your forked repository locally, replacing **YOUR_USERNAME** with
your GitHub username:

```
$ git clone --config pull.rebase git@github.com:YOUR_USERNAME/zulip-terminal.git
```

This should create a new directory for the repository in the current directory,
so enter the repository directory with `cd zulip-terminal` and configure and
fetch the upstream remote repository for your cloned fork of Zulip Terminal:

```
$ git remote add -f upstream https://github.com/zulip/zulip-terminal.git
```

For detailed explanation on the commands used for cloning and setting upstream,
refer to Step 1 of the
[Get Zulip Code](https://zulip.readthedocs.io/en/latest/git/cloning.html)
section of Zulip's Git guide.

### Setting up a development environment

Various options are available; we are exploring the benefits of each and would
appreciate feedback on which you use or feel works best.

Note that the tools used in each case are typically the same, but are called in
different ways.

The following commands should be run in the repository directory, created by a
process similar to that in the previous section.

#### Pipenv

1. Install pipenv
   (see the [recommended installation notes](https://pipenv.readthedocs.io/en/latest/installation);
   pipenv can be installed in a virtual environment, if you wish)
```
$ pip3 install --user pipenv
```
2. Initialize the pipenv virtual environment for zulip-term (using the default
   python 3; use eg. `--python 3.6` to be more specific)

```
$ pipenv --three
```

3. Install zulip-term, with the development requirements

```
$ pipenv install --dev
$ pipenv run pip3 install -e '.[dev]'
```

4. Connect the gitlint commit-message hook

```
$ pipenv run gitlint install-hook
```

#### Pip

1. Manually create & activate a virtual environment; any method should work,
   such as that used in the above simple installation

    1. `python3 -m venv zt_venv` (creates a venv named `zt_venv` in the current directory)
    2. `source zt_venv/bin/activate` (activates the venv; this assumes a bash-like shell)

2. Install zulip-term, with the development requirements
```
$ pip3 install -e '.[dev]'
```

3. Connect the gitlint commit-message hook
```
$ gitlint install-hook
```

#### make/pip

This is the newest and simplest approach, if you have `make` installed:

1. `make` (sets up an installed virtual environment in `zt_venv` in the current directory)
2. `source zt_venv/bin/activate` (activates the venv; this assumes a bash-like shell)
3. `gitlint install-hook` (connect the gitlint commit-message hook)

### Development tasks

Once you have a development environment set up, you might find the following useful, depending upon your type of environment:

| Task | Make & Pip | Pipenv |
|-|-|-|
| Run normally | `zulip-term` | `pipenv run zulip-term` |
| Run in debug mode | `zulip-term -d` | `pipenv run zulip-term -d` |
| Run with profiling | `zulip-term --profile` | `pipenv run zulip-term --profile` |
| Run all linters | `./tools/lint-all` | `pipenv run ./tools/lint-all` |
| Run all tests | `pytest` | `pipenv run pytest` |
| Build test coverage report | `pytest --cov-report html:cov_html --cov=./` | `pipenv run pytest --cov-report html:cov_html --cov=./` |

If using make with pip, running `make` will ensure the development environment
is up to date with the specified dependencies, useful after fetching from git
and rebasing.

### Editing the source

Pick your favorite text editor or development environment!

The source includes an `.editorconfig` file which enables many editors to
automatically configure themselves to produce files which meet the minimum
requirements for the project. See https://editorconfig.org for editor support;
note that some may require plugins if you wish to use this feature.

### Running linters and automated tests

The linters and automated tests (pytest) are automatically run in CI (GitHub
Actions) when you submit a pull request (PR), or push changes to an existing
pull request.

However, running these checks on your computer can speed up your development by
avoiding the need to repeatedly push your code to GitHub.
Commands to achieve this are listed in the table of development tasks above
(individual linters may also be run via scripts in `tools/`).

In addition, if using a `make`-based system:
- `make lint` and `make test` run all of each group of tasks
- `make check` runs all checks, which is useful before pushing a PR (or an update)
- `tools/check-branch` will run `make check` on each commit in your branch

> **NOTE: It is highly unlikely that a pull request will be merged, until *all*
linters and tests are passing, including on a per-commit basis.**

#### The linters and tests aren't passing on my branch/PR - what do I do?

Correcting some linting errors requires manual intervention, such as from
`mypy` for type-checking.

For tips on testing, please review the section further below regarding pytest.

However, other linting errors may be fixed automatically, as detailed below -
**this can save a lot of time manually adjusting your code to pass the
linters!**

> If you have troubles understanding why the linters or pytest are failing,
> please do push your code to a branch/PR and we can discuss the problems in
> the PR or on chat.zulip.org.

#### Automatically updating hotkeys & file docstrings, vs related documentation

If you update these, note that you do not need to update the text in both
places manually to pass linting.

The source of truth is in the source code, so simply update the python file and
run the relevant tool. Currently we have:
* `tools/lint-hotkeys --fix` to regenerate docs/hotkeys.md from config/keys.py
* `tools/lint-docstring --fix` to regenerate docs/developer-file-overview.md from file docstrings

(these tools are also used for the linting process, to ensure that these files
are synchronized)

#### Automatically formatting code

The project uses `black` and `isort` for code-style and import sorting respectively.

These tools can be run as linters locally, but can also *automatically* format
your code for you.

If you're using a `make`-based setup, running `make fix` will run both (and a
few other tools) and reformat the current state of your code - so you'll want
to commit first just in case, then `--amend` that commit if you're happy with
the changes.

You can also use the tools individually on a file or directory, eg.
`black zulipterminal` or `isort tests/model/test_model.py`

### Structuring Commits - speeding up reviews, merging & development

As you work locally, investigating changes to make, it's common to make a
series of small commits to store your progress.
Often this can include commits that fix linting or testing issues in the
previous commit(s).
These are **developmental-style commits** - and almost everyone is likely to
write commits in this style to some degree.

Developmental-style commits store the changes just fine for you right now.
However, when sharing your code, commit messages are a great place to instead
communicate to others what you're changing and why. Tidying the structure can
also make it easier and quicker for a reader to understand changes, and that you
respect their time. One example is that very large single commits can take a
lot of time to review, compared to if they are split up. Another is if you fix
the tests/linting in a commit: which commit (or commits!) does this fix, and if
it's in the same branch/PR, why isn't the original commit just fixed instead?

Therefore, when creating a Pull Request (PR), please consider that your code is
**more likely to be merged, more quickly, if it is easier to read, understand
and review** - and a big part of that is how you structure your changes into
commits, and describe those changes in commit messages.

To be productive and make it easier for your PRs to be reviewed and updated, we
follow an approach taken at
[Zulip](https://zulip.readthedocs.io/en/latest/contributing/commit-discipline.html)
and elsewhere, aiming for PRs to consist of a series of **minimal coherent**
commits:
* **Minimal**:
  Look at each commit. Does it make different kinds of changes to lots of
  files? Are you treating the title of a commit as a list of changes?  Consider
  how to break down a large change into a sequence of smaller changes that you
  can individually easily describe and that can flow after one another.
* **Coherent**:
  Each commit should individually pass all linting and existing automated
  tests, and if you add new behavior, then add or extend the tests to cover it.

Note that keeping to these principles can give other benefits, both before,
during and after reviewing a PR, including:
* Minimal commits can always later be *squashed* (combined) together - splitting commits is more challenging!
* Coherent commits mean nothing should be broken between and after commits
  * If a new commit in your branch breaks the linting/tests, it's likely that commit at fault!
  * This improves the utility of `git bisect` in your branch or on `main`
* These commits are easier to *reorder* in a branch
  * Commits at the start of a branch (or can be moved there), may be merged early to simplify a branch
* Understanding what changes were made and why is easier when reading a git log comprising these commits

We now enforce a limited aspect of the *coherent* nature of commits in a PR in
a job as part of our Continuous Integration (CI), *Ensure isolated PR commits*,
which essentially runs `make check` on each commit in your branch.  You can
replicate this locally before pushing to GitHub using `tools/check-branch`.

While small or proof-of-concept PRs are initially fine to push as they are,
they will likely only be reviewed based on the overall changes. Generally if
individual commits look like they have a developmental style then reviewers are
likely to give less specific feedback, and minimal coherent commits are certain
to be requested before merging.

**Restructuring commands** - Most restructuring relies upon *interactive
rebasing* (eg. `git rebase -i upstream/main`), but consider searching online
for specific actions, as well as searching or asking in **#git help** or
**#learning** on chat.zulip.org.

**Self review** - Another useful approach is to review your own commits both
locally (see [Zulip
suggestions](https://zulip.readthedocs.io/en/latest/git/setup.html#get-a-graphical-client))
and after you push to GitHub. This allows you to inspect and fix anything that
looks out of place, which someone is likely to pick up in their review, helping
your submissions look more polished, as well as again indicating that you
respect reviewers' time.

#### Commit Message Style

We aim to follow a standard commit style to keep the `git log` consistent and
easy to read.

Much like working with code, we suggest you refer to recent commits in the git
log, for examples of the style we're actively using.

Our overall style for commit messages broadly follows the general guidelines
given for
[Zulip Commit messages](http://zulip.readthedocs.io/en/latest/contributing/commit-discipline.html#commit-messages),
so we recommend reading that first.

Our commit titles (summaries) have slight variations from the general Zulip style, with each:
* starting with one or more **areas** - in lower case, followed by a colon and space
  * generally each commit has at least one area of each modified file - without extensions, separated by a `/`
  * generally *don't* include test files in this list (instead `Tests updated` or `Tests added` in the commit text)
  * other common areas include types like `refactor:`, `bugfix:` and `requirements:` (see below)
* ending with a **concise description** starting with a capital and ending with a full-stop (period)
* having a maximum overall length of 72 (fitting the github web interface without abbreviation)

Some example commit titles: (ideally more descriptive in practice!)
* `file3/file1/file2: Improve behavior of something.`
  - a general commit updating files `file1.txt`, `file2.py` and `file3.md`
* `refactor: file1/file2: Extract some common function.`
  - a pure refactor which doesn't change the functional behavior, involving `file1.py` and `file2.py`
* `bugfix: file1: Avoid some noticeable bug.`
  - a small commit to fix a bug in `file1.py`
* `tests: file1: Improve test for something.`
  - only improve tests for `file1`, likely in `test_file1.py`
* `requirements: Upgrade some-dependency from ==9.2 to ==9.3.`
  - upgrade a dependency from version ==9.2 to version ==9.3, in the central dependencies file (*not* some file requirements.*)

To aid in satisfying some of these rules you can use `GitLint`, as described in
the following section.

**However**, please check your commits manually versus these style rules, since
GitLint cannot check everything - including language or grammar!

#### GitLint

The `gitlint` tool is installed by default in the development environment, and
can help ensure that your commits meet the expected standard.

The tool can check specific commits manually, eg. `gitlint` for the latest
commit, or `gitlint --commits main..` for commits leading from `main`.
However, we highly recommend running `gitlint install-hook` to install the
`gitlint` commit-message hook
(or `pipenv run gitlint install-hook` with pipenv setups).

If the hook is installed as described above, then after completing the text for
a commit, it will be checked by gitlint against the style we have set up, and
will offer advice if there are any issues it notices.
If gitlint finds any, it will ask if you wish to commit with the message as it
is (`y` for 'yes'), stop the commit process (`n` for 'no'), or edit the commit
message (`e` for 'edit').

While the content still depends upon your writing skills, this ensures a more
consistent formatting structure between commits, including by different
authors.

### Tips for working with tests (pytest)

Tests for zulip-terminal are written using [pytest](https://pytest.org/).
You can read the tests in the `/tests` folder to learn about writing tests for
a new class/function.
If you are new to pytest, reading its documentation is definitely recommended.

We currently have thousands of tests which get checked upon running `pytest`.
While it is dependent on your system capability, this should typically take
less than one minute to run.
However, during debugging you may still wish to limit the scope of your tests,
to improve the turnaround time:

* If lots of tests are failing in a very verbose way, you might try the `-x`
  option (eg. `pytest -x`) to stop tests after the first failure; due to
parametrization of tests and test fixtures, many apparent errors/failures can
be resolved with just one fix! (try eg. `pytest --maxfail 3` for a less-strict
version of this)

* To avoid running all the successful tests each time, along with the failures,
  you can run with `--lf` (eg. `pytest --lf`), short for `--last-failed`
(similar useful options may be `--failed-first` and `--new-first`, which may
work well with `-x`)

* Since pytest 3.10 there is `--sw` (`--stepwise`), which works through known
  failures in the same way as `--lf` and `-x` can be used, which can be
combined with `--stepwise-skip` to control which test is the current focus

* If you know the names of tests which are failing and/or in a specific
  location, you might limit tests to a particular location (eg. `pytest
tests/model`) or use a selected keyword (eg. `pytest -k __handle`)

When only a subset of tests are running it becomes more practical and useful to
use the `-v` option (`--verbose`); instead of showing a `.` (or `F`, `E`, `x`,
etc) for each test result, it gives the name (with parameters) of each test
being run (eg. `pytest -v -k __handle`).
This option also shows more detail in tests and can be given multiple times
(eg. `pytest -vv`).

For additional help with pytest options see `pytest -h`, or check out the [full
pytest documentation](https://docs.pytest.org/en/latest/).

### Debugging Tips

#### Output using `print`

The stdout (standard output) for zulip-terminal is redirected to `./debug.log`
if debugging is enabled at run-time using `-d` or `--debug`.

This means that if you want to check the value of a variable, or perhaps
indicate reaching a certain point in the code, you can simply use `print()`,
eg.
```python3
print(f"Just about to do something with {variable}")
```
and when running with a debugging option, the string will be printed to
`./debug.log`.

With a bash-like terminal, you can run something like `tail -f debug.log` in
another terminal, to see the output from `print` as it happens.

#### Interactive debugging using pudb & telnet

If you want to debug zulip-terminal while it is running, or in a specific
state, you can insert
```python3
from pudb.remote import set_trace
set_trace()
```
in the part of the code you want to debug.
This will start a telnet connection for you. You can find the IP address and
port of the telnet connection in `./debug.log`. Then simply run
```
$ telnet 127.0.0.1 6899
```
in another terminal, where `127.0.0.1` is the IP address and `6899` is port you
find in `./debug.log`.

#### There's no effect in Zulip Terminal after making local changes!

This likely means that you have installed both normal and development versions
of zulip-terminal.

To ensure you run the development version:
* If using pipenv, call `pipenv run zulip-term` from the cloned/downloaded
  `zulip-terminal` directory;

* If using pip (pip3), ensure you have activated the correct virtual
  environment (venv); depending on how your shell is configured, the name of
the venv may appear in the command prompt.
Note that not including the `-e` in the pip3 command will also cause this
problem.


# Metagenome-Atlas

[![Anaconda-Server Badge](https://anaconda.org/bioconda/metagenome-atlas/badges/latest_release_relative_date.svg)](https://anaconda.org/bioconda/metagenome-atlas)
[![Bioconda](https://img.shields.io/conda/dn/bioconda/metagenome-atlas.svg?label=Bioconda )](https://anaconda.org/bioconda/metagenome-atlas)
[![Documentation Status](https://readthedocs.org/projects/metagenome-atlas/badge/?version=latest)](https://metagenome-atlas.readthedocs.io/en/latest/?badge=latest)
![Mastodon Follow](https://img.shields.io/mastodon/follow/109273833677404282?domain=https%3A%2F%2Fmstdn.science&style=social)
<!--[![follow on twitter](https://img.shields.io/twitter/follow/SilasKieser.svg?style=social&label=Follow)](https://twitter.com/search?f=tweets&q=%40SilasKieser%20%23metagenomeAtlas&src=typd) -->

Metagenome-atlas is a easy-to-use metagenomic pipeline based on snakemake. It handles all steps from QC, Assembly, Binning, to Annotation.

![scheme of workflow](resources/images/atlas_list.png?raw=true)

You can start using atlas with three commands:
```
    mamba install -y -c bioconda -c conda-forge metagenome-atlas={latest_version}
    atlas init --db-dir databases path/to/fastq/files
    atlas run all
```
where `{latest_version}` should be replaced by [![Version](https://anaconda.org/bioconda/metagenome-atlas/badges/version.svg)](https://anaconda.org/bioconda/metagenome-atlas)

# Webpage

[metagenome-atlas.github.io](https://metagenome-atlas.github.io/)

# Documentation

https://metagenome-atlas.readthedocs.io/

[Tutorial](https://github.com/metagenome-atlas/Tutorial)

# Citation

> ATLAS: a Snakemake workflow for assembly, annotation, and genomic binning of metagenome sequence data.
> Kieser, S., Brown, J., Zdobnov, E. M., Trajkovski, M. & McCue, L. A.
> BMC Bioinformatics 21, 257 (2020).
> doi: [10.1186/s12859-020-03585-4](https://doi.org/10.1186/s12859-020-03585-4)

# Developpment/Extensions

Here are some ideas I work or want to work on when I have time. If you want to contribute or have some ideas let me know via a feature request issue.

- Optimized MAG recovery (e.g. [Spacegraphcats](https://github.com/spacegraphcats/spacegraphcats))
- Integration of viruses/plasmid that live for now as [extensions](https://github.com/metagenome-atlas/virome_atlas)
- Add statistics and visualisations as in [atlas_analyze](https://github.com/metagenome-atlas/atlas_analyze)
- Implementation of most rules as snakemake wrapper
- Cloud execution
- Update to new Snakemake version and use cool reports.


.. image:: https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/_static/img/pyhf-logo.svg
   :alt: pyhf logo
   :width: 320
   :align: center

pure-python fitting/limit-setting/interval estimation HistFactory-style
=======================================================================

|GitHub Project| |DOI| |JOSS DOI| |Scikit-HEP| |NSF Award Number IRIS-HEP v1| |NSF Award Number IRIS-HEP v2| |NumFOCUS Affiliated Project|

|Docs from latest| |Docs from main| |Jupyter Book tutorial| |Binder|

|PyPI version| |Conda-forge version| |Supported Python versions| |Docker Hub pyhf| |Docker Hub pyhf CUDA|

|Code Coverage| |CodeFactor| |pre-commit.ci Status| |Code style: black|

|GitHub Actions Status: CI| |GitHub Actions Status: Docs| |GitHub Actions Status: Publish|
|GitHub Actions Status: Docker|

The HistFactory p.d.f. template
[`CERN-OPEN-2012-016 <https://cds.cern.ch/record/1456844>`__] is per-se
independent of its implementation in ROOT and sometimes, it’s useful to
be able to run statistical analysis outside of ROOT, RooFit, RooStats
framework.

This repo is a pure-python implementation of that statistical model for
multi-bin histogram-based analysis and its interval estimation is based
on the asymptotic formulas of “Asymptotic formulae for likelihood-based
tests of new physics”
[`arXiv:1007.1727 <https://arxiv.org/abs/1007.1727>`__]. The aim is also
to support modern computational graph libraries such as PyTorch and
TensorFlow in order to make use of features such as autodifferentiation
and GPU acceleration.

..
  Comment: JupyterLite segment goes here in docs

User Guide
----------

For an in depth walkthrough of usage of the latest release of ``pyhf`` visit the |pyhf tutorial|_.

.. |pyhf tutorial| replace:: ``pyhf`` tutorial
.. _pyhf tutorial: https://pyhf.github.io/pyhf-tutorial/

Hello World
-----------

This is how you use the ``pyhf`` Python API to build a statistical model and run basic inference:

.. code:: pycon

   >>> import pyhf
   >>> pyhf.set_backend("numpy")
   >>> model = pyhf.simplemodels.uncorrelated_background(
   ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
   ... )
   >>> data = [51, 48] + model.config.auxdata
   >>> test_mu = 1.0
   >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(
   ...     test_mu, data, model, test_stat="qtilde", return_expected=True
   ... )
   >>> print(f"Observed: {CLs_obs:.8f}, Expected: {CLs_exp:.8f}")
   Observed: 0.05251497, Expected: 0.06445321

Alternatively the statistical model and observational data can be read from its serialized JSON representation (see next section).

.. code:: pycon

   >>> import pyhf
   >>> import requests
   >>> pyhf.set_backend("numpy")
   >>> url = "https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/examples/json/2-bin_1-channel.json"
   >>> wspace = pyhf.Workspace(requests.get(url).json())
   >>> model = wspace.model()
   >>> data = wspace.data(model)
   >>> test_mu = 1.0
   >>> CLs_obs, CLs_exp = pyhf.infer.hypotest(
   ...     test_mu, data, model, test_stat="qtilde", return_expected=True
   ... )
   >>> print(f"Observed: {CLs_obs:.8f}, Expected: {CLs_exp:.8f}")
   Observed: 0.35998409, Expected: 0.35998409

Finally, you can also use the command line interface that ``pyhf`` provides

.. code:: bash

   $ cat << EOF  | tee likelihood.json | pyhf cls
   {
       "channels": [
           { "name": "singlechannel",
             "samples": [
               { "name": "signal",
                 "data": [12.0, 11.0],
                 "modifiers": [ { "name": "mu", "type": "normfactor", "data": null} ]
               },
               { "name": "background",
                 "data": [50.0, 52.0],
                 "modifiers": [ {"name": "uncorr_bkguncrt", "type": "shapesys", "data": [3.0, 7.0]} ]
               }
             ]
           }
       ],
       "observations": [
           { "name": "singlechannel", "data": [51.0, 48.0] }
       ],
       "measurements": [
           { "name": "Measurement", "config": {"poi": "mu", "parameters": []} }
       ],
       "version": "1.0.0"
   }
   EOF

which should produce the following JSON output:

.. code:: json

   {
      "CLs_exp": [
         0.0026062609501074576,
         0.01382005356161206,
         0.06445320535890459,
         0.23525643861460702,
         0.573036205919389
      ],
      "CLs_obs": 0.05251497423736956
   }

What does it support
--------------------

Implemented variations:
  - ☑ HistoSys
  - ☑ OverallSys
  - ☑ ShapeSys
  - ☑ NormFactor
  - ☑ Multiple Channels
  - ☑ Import from XML + ROOT via `uproot <https://github.com/scikit-hep/uproot4>`__
  - ☑ ShapeFactor
  - ☑ StatError
  - ☑ Lumi Uncertainty
  - ☑ Non-asymptotic calculators

Computational Backends:
  - ☑ NumPy
  - ☑ PyTorch
  - ☑ TensorFlow
  - ☑ JAX

Optimizers:
  - ☑ SciPy (``scipy.optimize``)
  - ☑ MINUIT (``iminuit``)

All backends can be used in combination with all optimizers.
Custom user backends and optimizers can be used as well.

Todo
----

-  ☐ StatConfig

results obtained from this package are validated against output computed
from HistFactory workspaces

A one bin example
-----------------

.. code:: python

   import pyhf
   import numpy as np
   import matplotlib.pyplot as plt
   from pyhf.contrib.viz import brazil

   pyhf.set_backend("numpy")
   model = pyhf.simplemodels.uncorrelated_background(
       signal=[10.0], bkg=[50.0], bkg_uncertainty=[7.0]
   )
   data = [55.0] + model.config.auxdata

   poi_vals = np.linspace(0, 5, 41)
   results = [
       pyhf.infer.hypotest(
           test_poi, data, model, test_stat="qtilde", return_expected_set=True
       )
       for test_poi in poi_vals
   ]

   fig, ax = plt.subplots()
   fig.set_size_inches(7, 5)
   brazil.plot_results(poi_vals, results, ax=ax)
   fig.show()

**pyhf**

.. image:: https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/_static/img/README_1bin_example.png
   :alt: manual
   :width: 500
   :align: center

**ROOT**

.. image:: https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/_static/img/hfh_1bin_55_50_7.png
   :alt: manual
   :width: 500
   :align: center

A two bin example
-----------------

.. code:: python

   import pyhf
   import numpy as np
   import matplotlib.pyplot as plt
   from pyhf.contrib.viz import brazil

   pyhf.set_backend("numpy")
   model = pyhf.simplemodels.uncorrelated_background(
       signal=[30.0, 45.0], bkg=[100.0, 150.0], bkg_uncertainty=[15.0, 20.0]
   )
   data = [100.0, 145.0] + model.config.auxdata

   poi_vals = np.linspace(0, 5, 41)
   results = [
       pyhf.infer.hypotest(
           test_poi, data, model, test_stat="qtilde", return_expected_set=True
       )
       for test_poi in poi_vals
   ]

   fig, ax = plt.subplots()
   fig.set_size_inches(7, 5)
   brazil.plot_results(poi_vals, results, ax=ax)
   fig.show()

**pyhf**

.. image:: https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/_static/img/README_2bin_example.png
   :alt: manual
   :width: 500
   :align: center

**ROOT**

.. image:: https://raw.githubusercontent.com/scikit-hep/pyhf/main/docs/_static/img/hfh_2_bin_100.0_145.0_100.0_150.0_15.0_20.0_30.0_45.0.png
   :alt: manual
   :width: 500
   :align: center

Installation
------------

To install ``pyhf`` from PyPI with the NumPy backend run

.. code:: bash

   python -m pip install pyhf

and to install ``pyhf`` with all additional backends run

.. code:: bash

   python -m pip install pyhf[backends]

or a subset of the options.

To uninstall run

.. code:: bash

   python -m pip uninstall pyhf

Documentation
-------------

For model specification, API reference, examples, and answers to FAQs visit the |pyhf documentation|_.

.. |pyhf documentation| replace:: ``pyhf`` documentation
.. _pyhf documentation: https://pyhf.readthedocs.io/

Questions
---------

If you have a question about the use of ``pyhf`` not covered in `the
documentation <https://pyhf.readthedocs.io/>`__, please ask a question
on the `GitHub Discussions <https://github.com/scikit-hep/pyhf/discussions>`__.

If you believe you have found a bug in ``pyhf``, please report it in the
`GitHub
Issues <https://github.com/scikit-hep/pyhf/issues/new?template=Bug-Report.md&labels=bug&title=Bug+Report+:+Title+Here>`__.
If you're interested in getting updates from the ``pyhf`` dev team and release
announcements you can join the |pyhf-announcements mailing list|_.

.. |pyhf-announcements mailing list| replace:: ``pyhf-announcements`` mailing list
.. _pyhf-announcements mailing list: https://groups.google.com/group/pyhf-announcements/subscribe

Citation
--------

As noted in `Use and Citations <https://scikit-hep.org/pyhf/citations.html>`__,
the preferred BibTeX entry for citation of ``pyhf`` includes both the
`Zenodo <https://zenodo.org/>`__ archive and the
`JOSS <https://joss.theoj.org/>`__ paper:

.. code:: bibtex

   @software{pyhf,
     author = {Lukas Heinrich and Matthew Feickert and Giordon Stark},
     title = "{pyhf: v0.7.6}",
     version = {0.7.6},
     doi = {10.5281/zenodo.1169739},
     url = {https://doi.org/10.5281/zenodo.1169739},
     note = {https://github.com/scikit-hep/pyhf/releases/tag/v0.7.6}
   }

   @article{pyhf_joss,
     doi = {10.21105/joss.02823},
     url = {https://doi.org/10.21105/joss.02823},
     year = {2021},
     publisher = {The Open Journal},
     volume = {6},
     number = {58},
     pages = {2823},
     author = {Lukas Heinrich and Matthew Feickert and Giordon Stark and Kyle Cranmer},
     title = {pyhf: pure-Python implementation of HistFactory statistical models},
     journal = {Journal of Open Source Software}
   }

Authors
-------

``pyhf`` is openly developed by Lukas Heinrich, Matthew Feickert, and Giordon Stark.

Please check the `contribution statistics for a list of
contributors <https://github.com/scikit-hep/pyhf/graphs/contributors>`__.

Milestones
----------

- 2022-09-12: 2000 GitHub issues and pull requests. (See PR `#2000 <https://github.com/scikit-hep/pyhf/pull/2000>`__)
- 2021-12-09: 1000 commits to the project. (See PR `#1710 <https://github.com/scikit-hep/pyhf/pull/1710>`__)
- 2020-07-28: 1000 GitHub issues and pull requests. (See PR `#1000 <https://github.com/scikit-hep/pyhf/pull/1000>`__)

Acknowledgements
----------------

Matthew Feickert has received support to work on ``pyhf`` provided by NSF
cooperative agreements `OAC-1836650 <https://nsf.gov/awardsearch/showAward?AWD_ID=1836650>`__
and `PHY-2323298 <https://nsf.gov/awardsearch/showAward?AWD_ID=2323298>`__ (IRIS-HEP)
and grant `OAC-1450377 <https://www.nsf.gov/awardsearch/showAward?AWD_ID=1450377>`__ (DIANA/HEP).

``pyhf`` is a `NumFOCUS Affiliated Project <https://numfocus.org/sponsored-projects/affiliated-projects>`__.

.. |GitHub Project| image:: https://img.shields.io/badge/GitHub--blue?style=social&logo=GitHub
   :target: https://github.com/scikit-hep/pyhf
.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1169739.svg
   :target: https://doi.org/10.5281/zenodo.1169739
.. |JOSS DOI| image:: https://joss.theoj.org/papers/10.21105/joss.02823/status.svg
   :target: https://doi.org/10.21105/joss.02823
.. |Scikit-HEP| image:: https://scikit-hep.org/assets/images/Scikit--HEP-Project-blue.svg
   :target: https://scikit-hep.org/
.. |NSF Award Number IRIS-HEP v1| image:: https://img.shields.io/badge/NSF-1836650-blue.svg
   :target: https://nsf.gov/awardsearch/showAward?AWD_ID=1836650
.. |NSF Award Number IRIS-HEP v2| image:: https://img.shields.io/badge/NSF-2323298-blue.svg
   :target: https://nsf.gov/awardsearch/showAward?AWD_ID=2323298
.. |NumFOCUS Affiliated Project| image:: https://img.shields.io/badge/NumFOCUS-Affiliated%20Project-orange.svg?style=flat&colorA=E1523D&colorB=007D8A
   :target: https://numfocus.org/sponsored-projects/affiliated-projects
.. |Docs from latest| image:: https://img.shields.io/badge/docs-v0.7.6-blue.svg
   :target: https://pyhf.readthedocs.io/
.. |Docs from main| image:: https://img.shields.io/badge/docs-main-blue.svg
   :target: https://scikit-hep.github.io/pyhf
.. |Jupyter Book tutorial| image:: https://jupyterbook.org/_images/badge.svg
   :target: https://pyhf.github.io/pyhf-tutorial/
.. |Binder| image:: https://mybinder.org/badge_logo.svg
   :target: https://mybinder.org/v2/gh/scikit-hep/pyhf/main?labpath=docs%2Fexamples%2Fnotebooks%2Fbinderexample%2FStatisticalAnalysis.ipynb

.. |PyPI version| image:: https://badge.fury.io/py/pyhf.svg
   :target: https://badge.fury.io/py/pyhf
.. |Conda-forge version| image:: https://img.shields.io/conda/vn/conda-forge/pyhf.svg
   :target: https://prefix.dev/channels/conda-forge/packages/pyhf
.. |Supported Python versions| image:: https://img.shields.io/pypi/pyversions/pyhf.svg
   :target: https://pypi.org/project/pyhf/
.. |Docker Hub pyhf| image:: https://img.shields.io/badge/pyhf-v0.7.6-blue?logo=Docker
   :target: https://hub.docker.com/r/pyhf/pyhf/tags
.. |Docker Hub pyhf CUDA| image:: https://img.shields.io/badge/pyhf-CUDA-blue?logo=Docker
   :target: https://hub.docker.com/r/pyhf/cuda/tags

.. |Code Coverage| image:: https://codecov.io/gh/scikit-hep/pyhf/graph/badge.svg?branch=main
   :target: https://codecov.io/gh/scikit-hep/pyhf?branch=main
.. |CodeFactor| image:: https://www.codefactor.io/repository/github/scikit-hep/pyhf/badge
   :target: https://www.codefactor.io/repository/github/scikit-hep/pyhf
.. |pre-commit.ci Status| image:: https://results.pre-commit.ci/badge/github/scikit-hep/pyhf/main.svg
  :target: https://results.pre-commit.ci/latest/github/scikit-hep/pyhf/main
  :alt: pre-commit.ci status
.. |Code style: black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black

.. |GitHub Actions Status: CI| image:: https://github.com/scikit-hep/pyhf/actions/workflows/ci.yml/badge.svg
   :target: https://github.com/scikit-hep/pyhf/actions/workflows/ci.yml?query=branch%3Amain
.. |GitHub Actions Status: Docs| image:: https://github.com/scikit-hep/pyhf/actions/workflows/docs.yml/badge.svg
   :target: https://github.com/scikit-hep/pyhf/actions/workflows/docs.yml?query=branch%3Amain
.. |GitHub Actions Status: Publish| image:: https://github.com/scikit-hep/pyhf/actions/workflows/publish-package.yml/badge.svg
   :target: https://github.com/scikit-hep/pyhf/actions/workflows/publish-package.yml?query=branch%3Amain
.. |GitHub Actions Status: Docker| image:: https://github.com/scikit-hep/pyhf/actions/workflows/docker.yml/badge.svg
   :target: https://github.com/scikit-hep/pyhf/actions/workflows/docker.yml?query=branch%3Amain


# XKNX - An asynchronous KNX library written in Python

![PyPI - Python Version](https://img.shields.io/pypi/pyversions/xknx?logo=python)
[![codecov](https://codecov.io/gh/XKNX/xknx/branch/main/graph/badge.svg?token=irWbIygS84)](https://codecov.io/gh/XKNX/xknx)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=f8b424)](https://github.com/pre-commit/pre-commit)
[![HA integration usage](https://img.shields.io/badge/dynamic/json?color=41BDF5&logo=home-assistant&label=integration%20usage&suffix=%20installs&cacheSeconds=15600&url=https://analytics.home-assistant.io/data.json&query=current.integrations.knx)](https://www.home-assistant.io/integrations/knx/)
[![Discord](https://img.shields.io/discord/338619021215924227?color=7289da&label=Discord&logo=discord&logoColor=7289da)](https://discord.gg/bkZe9m4zvw)

## Documentation

See documentation at: [https://xknx.io/](https://xknx.io/)

## Help

We need your help for testing and improving XKNX. For questions, feature requests, bug reports either open an [issue](https://github.com/XKNX/xknx/issues), join the [XKNX chat on Discord](https://discord.gg/EuAQDXU) or write an [email](mailto:xknx@xknx.io).

## Development

You will need at least Python 3.10 in order to use XKNX.

Setting up your local environment:

1. Install requirements: `pip install -r requirements/testing.txt`
2. Install pre-commit hook: `pre-commit install`

## Testing

To run all tests, linters, formatters and type checker call `tox`

Running only unit tests is possible with `pytest`
Running specific unit tests can be invoked by: `pytest -vv test/management_tests/procedures_test.py -k test_nm_individual_address_serial_number_write_fail`

## Home-Assistant

XKNX is the underlying library for the KNX integration in [Home Assistant](https://home-assistant.io/).

## Example

```python
"""Example for switching a light on and off."""
import asyncio

from xknx import XKNX
from xknx.devices import Light

async def main():
    """Connect to KNX/IP bus, switch on light, wait 2 seconds and switch it off again."""
    async with XKNX() as xknx:
        light = Light(
            xknx,
            name='TestLight',
            group_address_switch='1/0/9',
        )
        xknx.devices.async_add(light)

        await light.set_on()
        await asyncio.sleep(2)
        await light.set_off()

asyncio.run(main())
```

## Attributions

Many thanks to [Weinzierl Engineering GmbH](https://weinzierl.de) and [MDT technologies GmbH](https://www.mdt.de) for providing us each an IP Secure Router to support testing and development of xknx.


# ZHA Device Handlers For Home Assistant

![CI](https://github.com/zigpy/zha-device-handlers/workflows/CI/badge.svg?branch=dev)
[![Coverage Status](https://codecov.io/gh/zigpy/zha-device-handlers/branch/dev/graph/badge.svg)](https://app.codecov.io/gh/zigpy/zha-device-handlers/tree/dev)

ZHA Device Handlers are custom quirks implementations for [Zigpy](https://github.com/zigpy/zigpy), the library that provides the [Zigbee](http://www.zigbee.org) support for the [ZHA](https://www.home-assistant.io/components/zha/) component in [Home Assistant](https://www.home-assistant.io).

ZHA device handlers bridge the functionality gap created when manufacturers deviate from the ZCL specification, handling deviations and exceptions by parsing custom messages to and from Zigbee devices. Zigbee devices that deviate from or do not fully conform to the standard specifications set by the Zigbee Alliance may require the development of custom ZHA Device Handlers (ZHA custom quirks handler implementation) to for all their functions to work properly with the ZHA component in Home Assistant.

Custom quirks implementations for zigpy implemented as ZHA Device Handlers are a similar concept to that of [Hub-connected Device Handlers for the SmartThings Classics platform](https://stdavedemo.readthedocs.io/en/latest/device-type-developers-guide/) as well that of [Zigbee-Herdsman Converters (formerly Zigbee-Shepherd Converters) as used by Zigbee2mqtt](https://www.zigbee2mqtt.io/advanced/support-new-devices/01_support_new_devices.html), meaning they are virtual representation of a physical device that expose additional functionality that is not provided out-of-the-box by the existing integration between these platforms. See [Device Specifics](#Device-Specifics) for details.

# How to contribute

## Primer

ZHA device handlers and its provided Quirks allow Zigpy, ZHA and Home Assistant to work with non-standard Zigbee devices. If you are reading this you may have a device that isn't working as expected. This can be the case for a number of reasons but in this guide we will cover the cases where functionality is provided by a device in a non specification compliant manner by the device manufacturer.

## What are these specifications

Reference official Zigbee specification documentation from Connectivity Standards Alliance (a.k.a. "CSA-IOT", formerly "Zigbee Alliance"):

- Zigbee Cluster Library Specification
  - [Zigbee Cluster Library Specification R8 (Revision 8)](https://zigbeealliance.org/wp-content/uploads/2021/10/07-5123-08-Zigbee-Cluster-Library.pdf)
  - [Zigbee Cluster Library Specification R7 (Revision 7)](https://zigbeealliance.org/wp-content/uploads/2021/10/07-5123-07-ZigbeeClusterLibrary_Revision_7-1.pdf)
  - [Zigbee Cluster Library Specification R6 (Revision 6)](https://zigbeealliance.org/wp-content/uploads/2019/12/07-5123-06-zigbee-cluster-library-specification.pdf)
- Zigbee Protocol Specification (also known as "Zigbee Pro" specifications)
  - [Zigbee Protocol Specification 2023 (also known as "Zigbee PRO 2023" / Zigbee R23)](https://csa-iot.org/wp-content/uploads/2023/04/05-3474-23-csg-zigbee-specification-compressed.pdf)
  - [Zigbee Protocol Specification 2017 (also known as "Zigbee PRO 2017" / Zigbee R22)](https://csa-iot.org/wp-content/uploads/2022/01/docs-05-3474-22-0csg-zigbee-specification-1.pdf)
  - [Zigbee Protocol Specification 2015 (also known as "Zigbee PRO 2015" / Zigbee R21)](https://zigbeealliance.org/wp-content/uploads/2019/11/docs-05-3474-21-0csg-zigbee-specification.pdf)
- Zigbee Device Specifications
  - [Zigbee Base Device Behavior Specification (v1.0)](https://zigbeealliance.org/wp-content/uploads/2019/12/docs-13-0402-13-00zi-Base-Device-Behavior-Specification-2-1.pdf)
  - [Zigbee Lighting & Occupancy Device Specification (v1.0)](https://zigbeealliance.org/wp-content/uploads/2019/11/docs-15-0014-05-0plo-Lighting-OccupancyDevice-Specification-V1.0.pdf)
- Zigbee Green Power (ZGP "GreenPower" Profile) specifications
  - [Zigbee PRO Green Power feature specification Basic functionality set (v1.1.1)](https://csa-iot.org/wp-content/uploads/2022/01/docs-14-0563-18-batt-Green-Power-Basic-specification-v1.1.1.pdf)
  - [Zigbee PRO Green Power feature specification 1.0a (Revision 26)](https://zigbeealliance.org/wp-content/uploads/2019/11/docs-09-5499-26-batt-zigbee-green-power-specification.pdf)
- Zigbee Smart Energy (ZSE / Zigbee SE "Smart Energy" Profile) specifications
  - Zigbee Smart Energy Standard 1.4
  - [ZigBee Smart Energy Standard (v1.2a)](https://zigbeealliance.org/wp-content/uploads/2019/11/docs-07-5356-19-0zse-zigbee-smart-energy-profile-specification.pdf)

Additionally, see these third-party and manufacturer specific documents:

- [Tuya - Zigbee Connection Standard (Tuya Smart Documentation)](https://github.com/Koenkk/zigbee-herdsman/blob/master/docs/Zigbee%20Connection%20Standard_Tuya%20Smart_Documentation.pdf)
  - [Zigbee2MQTT guide on understanding the custom 'manuSpecificTuya' cluster that TuYa devices uses](https://www.zigbee2mqtt.io/advanced/support-new-devices/02_support_new_tuya_devices.html)
- [Samsung SmartThings -Device Handlers](https://stdavedemo.readthedocs.io/en/latest/device-type-developers-guide/)
  - [Samsung SmartThings - Zigbee Primer](https://stdavedemo.readthedocs.io/en/latest/device-type-developers-guide/zigbee-primer.html)
  - [Samsung SmartThings - Building ZigBee Device Handlers](https://stdavedemo.readthedocs.io/en/latest/device-type-developers-guide/building-zigbee-device-handlers.html)

## What is a device in human terms

A device is a physical object that you want to join to a Zigbee network: a light bulb, a switch, a sensor etc. The host application, in this case Zigpy, needs to understand how to interact with the device so there are standards that define how the application and devices can communicate. The device's functionality is described by several **descriptors** while the device itself contains **endpoints** and **endpoints** contain **clusters**. There are two types of clusters an endpoint contains:

- **in_clusters** - are "Server" clusters in ZCL terms. These clusters control the device, e.g. a smart plug or light bulb would have an `on_off` server cluster. **in_clusters** are also the ones which also send attribute reports and/or you can read an attribute from an **in_cluster**.
- **out_clusters** - are "Client" clusters. These clusters control some other device, as "Client" cluster sends commands to "Server" cluster. For example an On/Off remote would have an `on_off` client cluster and will generate cluster commands and send those to some other device.
  Zigpy needs to understand all these elements in order to correctly work with the device.

### Endpoints

Endpoints are essentially groupings of functionality. For example, a typical Zigbee light bulb will have a single endpoint for the light. A multi-gang wall switch may have an endpoint for each individual switch, so they can all be controlled separately. Each endpoint has several functions represented by clusters.

### Clusters

Clusters are objects that contain the information (attributes and commands) for individual functions. There is the ability to turn the switch on and off, maybe there is energy monitoring, maybe there is the ability to add each switch to an individual group or a scene, etc. Each of these functions belong to a cluster.

### Descriptors

For the purposes of Zigpy and Quirks we will focus on two descriptors: **Node Descriptor** and **Simple Descriptor**.

#### Node Descriptor

A node descriptor explains some basic device attributes to Zigpy. The manufacturer code and the power type are the ones that we generally care about. In most cases you won't have to worry about this, but it is good to know why it is there in case you come across it while looking at an existing quirk. Here is an example:
`<Optional byte1=2 byte2=64 mac_capability_flags=128 manufacturer_code=4174 maximum_buffer_size=82 maximum_incoming_transfer_size=82 server_mask=0 maximum_outgoing_transfer_size=82 descriptor_capability_field=0>`

#### Simple Descriptor

A simple descriptor is a description of a Zigbee device endpoint and is responsible for explaining the endpoint's functionality. It contains a profile id, the device type, and collections of clusters. The profile id tells the application what set of Zigbee rules to use. The most common profile will be 260 (0x0104) for the Home Automation profile. The device type tells the application what logical type of device this is ex: on off light, color light, etc. The clusters explain to the application what types of functionality exist on the endpoint. Here is an example:
`<SimpleDescriptor endpoint=1 profile=260 device_type=1026 device_version=0 input_clusters=[0, 1, 3, 32, 1026, 1280, 2821] output_clusters=[25]>`

## What the heck is a quirk

In human terms you can think of a quirk like Google Translate. I know it's a weird comparison but let's dig in a bit. You may only speak one language but there is an interesting article written in another language that you really want to read. Google Translate takes the original article and displays it in a format (language) that you understand. A quirk is a file that translates device functionality from the format that the manufacturer chose to implement it in to a format that Zigpy and in turn ZHA understand. The main purpose of a quirk is to serve as a translator. A quirk comprises several parts:

- Signature - To identify and apply the correct quirk
- Replacement - To allow Zigpy and ZHA to correctly work with the device
- device_automation_triggers - To let the Home Assistant Device Automation engine and users interact with the device

### Signature

The signature on a quirk identifies the device as the manufacturer implemented it. You can think of it as a fingerprint or the dna of the device. The signature is what we use to identify the device. If any part of the signature doesn't match what the device returns during discovery the quirk will not match and as a result it will not be applied. The signature is made up of several parts:

- `models_info`
- `endpoints`

Models info tells the application which devices should use this particular quirk. Endpoints are the simple descriptors that we spoke about earlier exactly as they are on the device. `endpoints` is a dict where the key is the id of the endpoint and the value is an object with the following properties: `profile_id`, `device_type`, `input_clusters` and `output_clusters`. Creating the signature element is generally just a job of transcribing what the device gives us. Here is an example:

```python
signature = {
    MODELS_INFO: [(LUMI, "lumi.plug.maus01")],
    ENDPOINTS: {
        # <SimpleDescriptor endpoint=1 profile=260 device_type=81
        # device_version=1
        # input_clusters=[0, 4, 3, 6, 16, 5, 10, 1, 2, 2820]
        # output_clusters=[25, 10]>
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
            INPUT_CLUSTERS: [
                Basic.cluster_id,
                PowerConfiguration.cluster_id,
                DeviceTemperature.cluster_id,
                Groups.cluster_id,
                Identify.cluster_id,
                OnOff.cluster_id,
                Scenes.cluster_id,
                BinaryOutput.cluster_id,
                Time.cluster_id,
                ElectricalMeasurement.cluster_id,
            ],
            OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
        },
    },
}
```

### Replacement

The replacement on a quirk is what we want the device to be. Remember, we said that quirks were like Google Translate... You can think of the replacement like the output from Google Translate. The replacement dict is what will actually be used by Zigpy and ZHA to interact with the device. The structure of `replacement` is the same as signature with 2 key differences: `models_info` is generally omitted and there is an extra element `skip_configuration` that instructs the application to skip configuration if necessary. Some manufacturers have not implemented the specifications correctly and the devices come pre-configured and therefore the configuration calls fail (non Zigbee 3.0 Xiaomi devices for instance). Usually, you should not add `skip_configuration`.

Here is an example:

```python
replacement = {
    SKIP_CONFIGURATION: True,
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
            INPUT_CLUSTERS: [
                BasicCluster,
                PowerConfiguration.cluster_id,
                DeviceTemperature.cluster_id,
                Groups.cluster_id,
                Identify.cluster_id,
                OnOff.cluster_id,
                Scenes.cluster_id,
                BinaryOutput.cluster_id,
                Time.cluster_id,
                ElectricalMeasurementCluster,
            ],
            OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
        },
    },
}
```

### device_automation_triggers

Device automation triggers are essentially representations of the events that the devices fire in HA. They allow users to use actions in the UI instead of using the raw events.

# Building a quirk

Now that we got that out of the way we can focus on the task at hand: make our devices work the way they should with Zigpy and ZHA. Because the device doesn't work correctly out of the box we have to write a quirk for it. First lets look at what the quirk looks like when complete:

```python
class Plug(XiaomiCustomDevice):
    """lumi.plug.maus01 plug."""

    def __init__(self, *args, **kwargs):
        """Init."""
        self.voltage_bus = Bus()
        self.consumption_bus = Bus()
        self.power_bus = Bus()
        super().__init__(*args, **kwargs)

    signature = {
        MODELS_INFO: [(LUMI, "lumi.plug.maus01")],
        ENDPOINTS: {
            # <SimpleDescriptor endpoint=1 profile=260 device_type=81
            # device_version=1
            # input_clusters=[0, 4, 3, 6, 16, 5, 10, 1, 2, 2820]
            # output_clusters=[25, 10]>
            1: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
                INPUT_CLUSTERS: [
                    Basic.cluster_id,
                    PowerConfiguration.cluster_id,
                    DeviceTemperature.cluster_id,
                    Groups.cluster_id,
                    Identify.cluster_id,
                    OnOff.cluster_id,
                    Scenes.cluster_id,
                    BinaryOutput.cluster_id,
                    Time.cluster_id,
                    ElectricalMeasurement.cluster_id,
                ],
                OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
            },
            # <SimpleDescriptor endpoint=2 profile=260 device_type=9
            # device_version=1
            # input_clusters=[12]
            # output_clusters=[12, 4]>
            2: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.MAIN_POWER_OUTLET,
                INPUT_CLUSTERS: [AnalogInput.cluster_id],
                OUTPUT_CLUSTERS: [AnalogInput.cluster_id, Groups.cluster_id],
            },
            # <SimpleDescriptor endpoint=3 profile=260 device_type=83
            # device_version=1
            # input_clusters=[12]
            # output_clusters=[12]>
            3: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.METER_INTERFACE,
                INPUT_CLUSTERS: [AnalogInput.cluster_id],
                OUTPUT_CLUSTERS: [AnalogInput.cluster_id],
            },
            # <SimpleDescriptor endpoint=100 profile=260 device_type=263
            # device_version=2
            # input_clusters=[15]
            # output_clusters=[15, 4]>
            100: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.OCCUPANCY_SENSOR,
                INPUT_CLUSTERS: [BinaryInput.cluster_id],
                OUTPUT_CLUSTERS: [BinaryInput.cluster_id, Groups.cluster_id],
            },
        },
    }
    replacement = {
        SKIP_CONFIGURATION: True,
        ENDPOINTS: {
            1: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
                INPUT_CLUSTERS: [
                    BasicCluster,
                    PowerConfiguration.cluster_id,
                    DeviceTemperature.cluster_id,
                    Groups.cluster_id,
                    Identify.cluster_id,
                    OnOff.cluster_id,
                    Scenes.cluster_id,
                    BinaryOutput.cluster_id,
                    Time.cluster_id,
                    ElectricalMeasurementCluster,
                ],
                OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
            },
            2: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.MAIN_POWER_OUTLET,
                INPUT_CLUSTERS: [AnalogInputCluster],
                OUTPUT_CLUSTERS: [AnalogInput.cluster_id, Groups.cluster_id],
            },
            3: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.METER_INTERFACE,
                INPUT_CLUSTERS: [AnalogInput.cluster_id],
                OUTPUT_CLUSTERS: [AnalogInput.cluster_id],
            },
            100: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.OCCUPANCY_SENSOR,
                INPUT_CLUSTERS: [BinaryInput.cluster_id],
                OUTPUT_CLUSTERS: [BinaryInput.cluster_id, Groups.cluster_id],
            },
        },
    }
```

This quirk is for the US version of the Xiaomi plug. Xiaomi is notorious for not following the Zigbee specifications and most of their non Zigbee 3.0 devices need a quirk to function correctly. In this case we are correcting the `ElectricalMeasurement` cluster readings. Xiaomi decided to report the values for this cluster on the `AnalogInput` cluster instead. To fix this we will create a custom cluster to replace the `AnalogInput` and `ElectricalMeasurement` clusters. We will take the values that are reported on the `AnalogInput` cluster and publish them to the `ElectricalMeasurement` cluster. Doing this allows the device to work as if Xiaomi had implemented this in the first place. This is the act of translating that was mentioned in the Google Translate analogy above.

First things first. All device definitions in quirks must extend `CustomDevice` or a derivative of it and all clusters that you define must extend `CustomCluster` or a derivative of it. If you want to send messages between `CustomCluster` definitions as we do here you need to create channels for the communication to flow through. We do this by adding instances of `Bus` on our `CustomDevice` implementation. `Bus` is a utility class used specifically for this purpose and adding it to the device implementation ensures that all clusters that you define will have access to the `Bus` so that they can communicate with each other.

```python
class Plug(XiaomiCustomDevice):
    """lumi.plug.maus01 plug."""

    def __init__(self, *args, **kwargs):
        """Init."""
        self.voltage_bus = Bus()
        self.consumption_bus = Bus()
        self.power_bus = Bus()
        super().__init__(*args, **kwargs)
```

You can see that we have extended `XiaomiCustomDevice` which is a derivative of `CustomDevice` shared by Xiaomi devices. You can also see that we have added some instances of `Bus` so that we can pass messages between `CustomCluster` definitions. To be clear, this is not always necessary. Quirks can be used to change formats of data on an existing cluster, to add manufacturer specific attributes or commands to clusters etc. In these instances you just need to create a derivative of `CustomCluster` and add your logic. This is more of an advanced example to illustrate what is possible.

Here are the custom cluster definitions:

```python
class AnalogInputCluster(CustomCluster, AnalogInput):
    """Analog input cluster, only used to relay power consumption information to ElectricalMeasurementCluster."""

    cluster_id = AnalogInput.cluster_id

    def __init__(self, *args, **kwargs):
        """Init."""
        self._current_state = {}
        super().__init__(*args, **kwargs)

    def _update_attribute(self, attrid, value):
        super()._update_attribute(attrid, value)
        if value is not None and value >= 0:
            self.endpoint.device.power_bus.listener_event(POWER_REPORTED, value)

class ElectricalMeasurementCluster(LocalDataCluster, ElectricalMeasurement):
    """Electrical measurement cluster to receive reports that are sent to the basic cluster."""

    cluster_id = ElectricalMeasurement.cluster_id
    POWER_ID = 0x050B
    VOLTAGE_ID = 0x0500
    CONSUMPTION_ID = 0x0304

    def __init__(self, *args, **kwargs):
        """Init."""
        super().__init__(*args, **kwargs)
        self.endpoint.device.voltage_bus.add_listener(self)
        self.endpoint.device.consumption_bus.add_listener(self)
        self.endpoint.device.power_bus.add_listener(self)

    def power_reported(self, value):
        """Power reported."""
        self._update_attribute(self.POWER_ID, value)

    def voltage_reported(self, value):
        """Voltage reported."""
        self._update_attribute(self.VOLTAGE_ID, value)

    def consumption_reported(self, value):
        """Consumption reported."""
        self._update_attribute(self.CONSUMPTION_ID, value)
```

In the `AnalogInput` cluster we override the `_update_attribute` method so that we can access the data that the cluster receives when the device sends a report and we send the data via an event on a bus to the `ElectricalMeasurement` cluster. This is the line that does the heavy lifting:

`self.endpoint.device.power_bus.listener_event(POWER_REPORTED, value)`

Then in the `ElectricalMeasurement` cluster we need to subscribe to these events and handle them. This is how we subscribe to our custom events:

`self.endpoint.device.power_bus.add_listener(self)`

and this method (the method name must match the event name that you publish EXACTLY):

```python
def power_reported(self, value):
    """Power reported."""
    self._update_attribute(self.POWER_ID, value)
```

receives the event and handles updating the attribute on the correct zigbee cluster. As you can see there really isn't much here that needs to be done to accomplish our goal.

Once we have created our `CustomCluster` implementations we have to tell the `CustomDevice` implementation to use them. We do this in the `replacement` dict in the quirk definition. Start by copying the `signature` dict and remove the `models_info` from it. Then we replace the cluster ids that we want to override with the names of our `CustomCluster` implementations that we have created. The result looks like this:

```python
replacement = {
    SKIP_CONFIGURATION: True,
    ENDPOINTS: {
        1: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
            INPUT_CLUSTERS: [
                BasicCluster,
                PowerConfiguration.cluster_id,
                DeviceTemperature.cluster_id,
                Groups.cluster_id,
                Identify.cluster_id,
                OnOff.cluster_id,
                Scenes.cluster_id,
                BinaryOutput.cluster_id,
                Time.cluster_id,
                ElectricalMeasurementCluster,
            ],
            OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
        },
        2: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.MAIN_POWER_OUTLET,
            INPUT_CLUSTERS: [AnalogInputCluster],
            OUTPUT_CLUSTERS: [AnalogInput.cluster_id, Groups.cluster_id],
        },
        3: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.METER_INTERFACE,
            INPUT_CLUSTERS: [AnalogInput.cluster_id],
            OUTPUT_CLUSTERS: [AnalogInput.cluster_id],
        },
        100: {
            PROFILE_ID: zha.PROFILE_ID,
            DEVICE_TYPE: zha.DeviceType.OCCUPANCY_SENSOR,
            INPUT_CLUSTERS: [BinaryInput.cluster_id],
            OUTPUT_CLUSTERS: [BinaryInput.cluster_id, Groups.cluster_id],
        },
```

You can see that we have replaced `ElectricalMeasurement.cluster_id` from endpoint 1 in the `signature` dict with the name of our cluster that we created: `ElectricalMeasurementCluster` and on endpoint 2 we replaced `AnalogInput.cluster_id` with the implementation we created for that: `AnalogInputCluster`. This instructs Zigpy to use these `CustomCluster` derivatives instead of the normal cluster definitions for these clusters and this is why this part of the quirk is called `replacement`.

Now lets put this all together. If you examine the device definition above you will see that we have defined our custom device, we defined the `signature` dict where we transcribed the `SimpleDescriptor` output we obtained when the device joined the network and we defined the `replacement` dict where we swapped the cluster ids for the culsters that we wanted to replace with the `CustomCluster` implementations that we created.

# Contribution Guidelines

- All code is formatted with black. The check format script that runs in CI will ensure that code meets this requirement and that it is correctly formatted with black. Instructions for installing black in many editors can be found here: <https://github.com/psf/black#editor-integration>

- Capture the SimpleDescriptor log entries for each endpoint on the device. These can be found in the HA logs after joining a device and they look like this: `<SimpleDescriptor endpoint=1 profile=260 device_type=1026 device_version=0 input_clusters=[0, 1, 3, 32, 1026, 1280, 2821] output_clusters=[25]>`. This information can also be obtained from the zigbee.db if you want to take the time to query the tables and reconstitute the log entry. I find it easier to just remove and rejoin the device. ZHA entity ids are stable for the most part so it _shouldn't_ disrupt anything you have configured. These need to match what the device reports EXACTLY or zigpy will not match them when a device joins and the handler will not be used for the device. You can also obtain this information from the device screen in HA for the device. The `Zigbee Device Signature` button will launch a dialog that contains all of the information necessary to create quirks.

- All custom device definitions must extend `CustomDevice` or a derivative of it

- All custom cluster definitions must extend `CustomCluster` or a derivative of it

- Use constants for all attribute values referencing the appropriate labels from Zigpy / HA as necessary

- Use an existing handler as a guide. signature and replacement dicts are required. Include the SimpleDescriptor entry for each endpoint in the signature dict above the definition of the endpoint in this format:

  ```yaml
  #  <SimpleDescriptor endpoint=1 profile=260 device_type=1026
  #  device_version=0
  #  input_clusters=[0, 1, 3, 32, 1026, 1280, 2821]
  #  output_clusters=[25]>
  ```

# How `device_automation_triggers` work

Device automation triggers are essentially representations of the events that the devices fire in HA. They allow users to use actions in the UI instead of using the raw events. Ex: For the Hue remote - the on button fires this event:

`<Event zha_event[L]: unique_id=00:17:88:01:04:e7:f9:37:1:0x0006, device_ieee=00:17:88:01:04:e7:f9:37, endpoint_id=1, cluster_id=6, command=on, args=[]>`

and the action defined for this is:

`(SHORT_PRESS, TURN_ON): {COMMAND: COMMAND_ON}`

The first part `(SHORT_PRESS, TURN_ON)` corresponds to the txt the user will see in the UI:

<img width="620" alt="image" src="https://user-images.githubusercontent.com/1335687/73609115-76480b80-4598-11ea-97eb-8d8343e2355b.png">

The second part is the event data. You only need to supply enough of the event data to uniquely match the event which in this case is just the command for this event fired by this device: `{COMMAND: COMMAND_ON}`

If you look at another example for the same device:

`(SHORT_PRESS, DIM_UP): {COMMAND: COMMAND_STEP, CLUSTER_ID: 8, ENDPOINT_ID: 1, PARAMS: {'step_mode': 0},}`

You can see a pattern that illustrates how to match a more complex event. In this case the step command is used for the dim up and dim down buttons so we need to match more of the event data to uniquely match the event.

## Setting up the development environment

Open a terminal at the root of the project and run the setup script: `script/setup` This script will install all necessary dependencies and it will install the precommit hook.

The tests use the [pytest](https://docs.pytest.org/en/latest/) framework.

### Getting started

To get set up, you need install the test dependencies:

```bash
pip install -r requirements_test_all.txt
```

### Running the tests

See the [pytest documentation](https://docs.pytest.org/en/latest/) for details about how to run
the tests. For example, to run all the `test_tuya.py` tests:

```bash
$ pytest --disable-warnings tests/test_tuya.py
Test session starts (platform: linux, Python 3.9.2, pytest 6.2.5, pytest-sugar 0.9.4)

collecting ...
 tests/test_tuya.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                                100% ██████████

Results (3.58s):
      41 passed
```

### Writing tests

To add a new test, start by adding a new function to one of the existing test files. You
can follow the instructions in the [Getting started](https://docs.pytest.org/en/latest/getting-started.html)
section of the pytest documentation.

### Using fixtures to set things up

In order to write a test, you will need to access an instance of a quirk to run the tests against. Pytest
provides a useful feature called Fixtures that allow you to write and use the setup code necessary in one
place, similar to how we use libraries to provide common functions to other code.

You can read more about fixtures [here](https://docs.pytest.org/en/latest/how-to/fixtures.html#how-to-fixtures).

You can find the common fixtures in files named `conftest.py`. Pytest will list them for you as follows:

```bash
$ pytest --fixtures
[...]
--- fixtures defined from tests.conftest ---
MockAppController
    App controller mock.

ieee_mock
    Return a static ieee.

zigpy_device_mock
    Zigpy device mock.

zigpy_device_from_quirk
    Create zigpy device from Quirks signature.

[...]
--- fixtures defined from tests.test_tuya_clusters ---
TuyaCluster
    Mock of the new Tuya manufacturer cluster.
```

Some fixtures such as `app_controller_mock` will provide an object instance that you can
use directly. Others, such as `zigpy_device_mock` will return a function, which you can
call to create a customised object during your own setup.

### Testing the quirk signature matching

The fixture `assert_signature_matches_quirk` provides a function that can be
used to check that a particular device signature matches the corresponding quirk.
By capturing the signature and adding a few lines to the test file, this means that
you can verify that your device will be matched against the quirk without needing to
go through the paring process directly.

You need to capture the device signature and save it. If you have previously started the
pairing process in Home assistant, you can find the signature under 'Zigbee Device Signature'
on the device page.

Now you can create a test that checks the signature as follows:

```python
def test_ts0121_signature(assert_signature_matches_quirk):
    signature = {
        "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",
        "endpoints": {
            "1": {
            "profile_id": 260,
            "device_type": "0x0051",
            "in_clusters": [
                "0x0000",
                "0x0004",
                "0x0005",
                "0x0006",
                "0x0702",
                "0x0b04"
            ],
            "out_clusters": [
                "0x000a",
                "0x0019"
            ]
            }
        },
        "manufacturer": "_TZ3000_g5xawfcq",
        "model": "TS0121",
        "class": "zhaquirks.tuya.ts0121_plug.Plug"
    }
    assert_signature_matches_quirk(zhaquirks.tuya.ts0121_plug.Plug, signature)
```

# Thanks

- Special thanks to damarco for the majority of the device tracker code
- Special thanks to Yoda-x for the Xioami attribute parsing code
- Special thanks to damarco and Adminiuga for allowing me to bounce ideas off of them and for listening to me ramble

# Related projects

## Zigpy

**[zigpy](https://github.com/zigpy/zigpy)** is a **[Zigbee protocol stack](https://en.wikipedia.org/wiki/Zigbee)** integration project to implement the **[Zigbee Home Automation](https://www.zigbee.org/)** standard as a Python 3 library. Zigbee Home Automation integration with zigpy allows you to connect one of many off-the-shelf Zigbee adapters using one of the available Zigbee radio library modules compatible with zigpy to control Zigbee based devices. There is currently support for controlling Zigbee device types such as binary sensors (e.g., motion and door sensors), sensors (e.g., temperature sensors), light bulbs, switches, locks, fans, covers (blinds, marquees, and more). A working implementation of zigpy exists in **[Home Assistant](https://www.home-assistant.io)** (Python based open source home automation software) as part of its **[ZHA component](https://www.home-assistant.io/components/zha/)**

## ZHA Network Card

[zha-network-card](https://github.com/dmulcahey/zha-network-card) is a custom Lovelace card that displays ZHA network and device information in Home Assistant