[![checkov](https://raw.githubusercontent.com/bridgecrewio/checkov/main/docs/web/images/checkov_blue_logo.png)](#)

[![Maintained by Prisma Cloud](https://img.shields.io/badge/maintained_by-Prisma_Cloud-blue)](https://prismacloud.io/?utm_source=github&utm_medium=organic_oss&utm_campaign=checkov)
[![build status](https://github.com/bridgecrewio/checkov/workflows/build/badge.svg)](https://github.com/bridgecrewio/checkov/actions?query=workflow%3Abuild)
[![security status](https://github.com/bridgecrewio/checkov/workflows/security/badge.svg)](https://github.com/bridgecrewio/checkov/actions?query=event%3Apush+branch%3Amaster+workflow%3Asecurity)
[![code_coverage](https://raw.githubusercontent.com/bridgecrewio/checkov/main/coverage.svg?sanitize=true)](https://github.com/bridgecrewio/checkov/actions?query=workflow%3Acoverage)
[![docs](https://img.shields.io/badge/docs-passing-brightgreen)](https://www.checkov.io/1.Welcome/What%20is%20Checkov.html?utm_source=github&utm_medium=organic_oss&utm_campaign=checkov)
[![PyPI](https://img.shields.io/pypi/v/checkov)](https://pypi.org/project/checkov/)
[![Python Version](https://img.shields.io/pypi/pyversions/checkov)](#)
[![Terraform Version](https://img.shields.io/badge/tf-%3E%3D0.12.0-blue.svg)](#)
[![Downloads](https://static.pepy.tech/badge/checkov)](https://pepy.tech/project/checkov)
[![Docker Pulls](https://img.shields.io/docker/pulls/bridgecrew/checkov.svg)](https://hub.docker.com/r/bridgecrew/checkov)
[![slack-community](https://img.shields.io/badge/Slack-4A154B?style=plastic&logo=slack&logoColor=white)](https://codifiedsecurity.slack.com/)

**Checkov** is a static code analysis tool for infrastructure as code (IaC) and also a software composition analysis (SCA) tool for images and open source packages.

It scans cloud infrastructure provisioned using [Terraform](https://terraform.io/), [Terraform plan](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Terraform%20Plan%20Scanning.md), [Cloudformation](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Cloudformation.md), [AWS SAM](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/AWS%20SAM.md), [Kubernetes](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Kubernetes.md), [Helm charts](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Helm.md), [Kustomize](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Kustomize.md), [Dockerfile](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Dockerfile.md),  [Serverless](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Serverless%20Framework.md), [Bicep](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Bicep.md), [OpenAPI](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/OpenAPI.md) or [ARM Templates](https://github.com/bridgecrewio/checkov/blob/main/docs/7.Scan%20Examples/Azure%20ARM%20templates.md) and detects security and compliance misconfigurations using graph-based scanning.

It performs [Software Composition Analysis (SCA) scanning](docs/7.Scan%20Examples/Sca.md) which is a scan of open source packages and images for Common Vulnerabilities and Exposures (CVEs).

Checkov also powers [**Prisma Cloud Application Security**](https://www.prismacloud.io/prisma/cloud/cloud-code-security/?utm_source=github&utm_medium=organic_oss&utm_campaign=checkov), the developer-first platform that codifies and streamlines cloud security throughout the development lifecycle. Prisma Cloud identifies, fixes, and prevents misconfigurations in cloud resources and infrastructure-as-code files.

## **Table of contents**

- [Features](#features)
- [Screenshots](#screenshots)
- [Getting Started](#getting-started)
- [Disclaimer](#disclaimer)
- [Support](#support)
- [Migration - v2 to v3](https://github.com/bridgecrewio/checkov/blob/main/docs/1.Welcome/Migration.md)

 ## Features

 * [Over 1000 built-in policies](https://github.com/bridgecrewio/checkov/blob/main/docs/5.Policy%20Index/all.md) cover security and compliance best practices for AWS, Azure and Google Cloud.
 * Scans Terraform, Terraform Plan, Terraform JSON, CloudFormation, AWS SAM, Kubernetes, Helm, Kustomize, Dockerfile, Serverless framework, Ansible, Bicep and ARM template files.
 * Scans Argo Workflows, Azure Pipelines, BitBucket Pipelines, Circle CI Pipelines, GitHub Actions and GitLab CI workflow files
 * Supports Context-awareness policies based on in-memory graph-based scanning.
 * Supports Python format for attribute policies and YAML format for both attribute and composite policies.
 * Detects [AWS credentials](https://github.com/bridgecrewio/checkov/blob/main/docs/2.Basics/Scanning%20Credentials%20and%20Secrets.md) in EC2 Userdata, Lambda environment variables and Terraform providers.
 * [Identifies secrets](https://www.prismacloud.io/prisma/cloud/secrets-security) using regular expressions, keywords, and entropy based detection.
 * Evaluates [Terraform Provider](https://registry.terraform.io/browse/providers) settings to regulate the creation, management, and updates of IaaS, PaaS or SaaS managed through Terraform.
 * Policies support evaluation of [variables](https://github.com/bridgecrewio/checkov/blob/main/docs/2.Basics/Handling%20Variables.md) to their optional default value.
 * Supports in-line [suppression](https://github.com/bridgecrewio/checkov/blob/main/docs/2.Basics/Suppressing%20and%20Skipping%20Policies.md) of accepted risks or false-positives to reduce recurring scan failures. Also supports global skip from using CLI.
 * [Output](https://github.com/bridgecrewio/checkov/blob/main/docs/2.Basics/Reviewing%20Scan%20Results.md) currently available as CLI, [CycloneDX](https://cyclonedx.org), JSON, JUnit XML, CSV, SARIF and github markdown and link to remediation [guides](https://docs.prismacloud.io/en/enterprise-edition/policy-reference/).

## Screenshots

Scan results in CLI

![scan-screenshot](https://raw.githubusercontent.com/bridgecrewio/checkov/main/docs/checkov-recording.gif)

Scheduled scan result in Jenkins

![jenikins-screenshot](https://raw.githubusercontent.com/bridgecrewio/checkov/main/docs/checkov-jenkins.png)

## Getting started

### Requirements
 * Python >= 3.8 (Data classes are available for Python 3.8+)
 * Terraform >= 0.12

### Installation

To install pip follow the official [docs](https://pip.pypa.io/en/stable/cli/pip_install/)

```sh
pip3 install checkov
```

or with [Homebrew](https://formulae.brew.sh/formula/checkov) (macOS or Linux)

```sh
brew install checkov
```

### Enabling bash autocomplete
```sh
source <(register-python-argcomplete checkov)
```
### Upgrade

if you installed checkov with pip3
```sh
pip3 install -U checkov
```

or with Homebrew

```sh
brew upgrade checkov
```

### Configure an input folder or file

```sh
checkov --directory /user/path/to/iac/code
```

Or a specific file or files

```sh
checkov --file /user/tf/example.tf
```
Or
```sh
checkov -f /user/cloudformation/example1.yml -f /user/cloudformation/example2.yml
```

Or a terraform plan file in json format
```sh
terraform init
terraform plan -out tf.plan
terraform show -json tf.plan  > tf.json
checkov -f tf.json
```

Note: `terraform show` output file `tf.json` will be a single line.
For that reason all findings will be reported line number 0 by Checkov

```sh
check: CKV_AWS_21: "Ensure all data stored in the S3 bucket have versioning enabled"
	FAILED for resource: aws_s3_bucket.customer
	File: /tf/tf.json:0-0
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/s3-policies/s3-16-enable-versioning
  ```

If you have installed `jq` you can convert json file into multiple lines with the following command:
```sh
terraform show -json tf.plan | jq '.' > tf.json
```
Scan result would be much user friendly.
```sh
checkov -f tf.json
Check: CKV_AWS_21: "Ensure all data stored in the S3 bucket have versioning enabled"
	FAILED for resource: aws_s3_bucket.customer
	File: /tf/tf1.json:224-268
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/s3-policies/s3-16-enable-versioning

		225 |               "values": {
		226 |                 "acceleration_status": "",
		227 |                 "acl": "private",
		228 |                 "arn": "arn:aws:s3:::mybucket",

```

Alternatively, specify the repo root of the hcl files used to generate the plan file, using the `--repo-root-for-plan-enrichment` flag, to enrich the output with the appropriate file path, line numbers, and codeblock of the resource(s). An added benefit is that check suppressions will be handled accordingly.
```sh
checkov -f tf.json --repo-root-for-plan-enrichment /user/path/to/iac/code
```

### Scan result sample (CLI)

```sh
Passed Checks: 1, Failed Checks: 1, Suppressed Checks: 0
Check: "Ensure all data stored in the S3 bucket is securely encrypted at rest"
/main.tf:
	 Passed for resource: aws_s3_bucket.template_bucket
Check: "Ensure all data stored in the S3 bucket is securely encrypted at rest"
/../regionStack/main.tf:
	 Failed for resource: aws_s3_bucket.sls_deployment_bucket_name
```

Start using Checkov by reading the [Getting Started](https://github.com/bridgecrewio/checkov/blob/main/docs/1.Welcome/Quick%20Start.md) page.

### Using Docker

```sh
docker pull bridgecrew/checkov
docker run --tty --rm --volume /user/tf:/tf --workdir /tf bridgecrew/checkov --directory /tf
```
Note: if you are using Python 3.6(Default version in Ubuntu 18.04) checkov will not work, and it will fail with `ModuleNotFoundError: No module named 'dataclasses'`  error message. In this case, you can use the docker version instead.

Note that there are certain cases where redirecting `docker run --tty` output to a file - for example, if you want to save the Checkov JUnit output to a file - will cause extra control characters to be printed. This can break file parsing. If you encounter this, remove the `--tty` flag.

The `--workdir /tf` flag is optional to change the working directory to the mounted volume. If you are using the SARIF output `-o sarif` this will output the results.sarif file to the mounted volume (`/user/tf` in the example above). If you do not include that flag, the working directory will be "/".

### Running or skipping checks

By using command line flags, you can specify to run only named checks (allow list) or run all checks except
those listed (deny list). If you are using the platform integration via API key, you can also specify a severity threshold to skip and / or include.
Moreover, as json files can't contain comments, one can pass regex pattern to skip json file secret scan.

See the docs for more detailed information about how these flags work together.

## Examples

Allow only the two specified checks to run:
```sh
checkov --directory . --check CKV_AWS_20,CKV_AWS_57
```

Run all checks except the one specified:
```sh
checkov -d . --skip-check CKV_AWS_20
```

Run all checks except checks with specified patterns:
```sh
checkov -d . --skip-check CKV_AWS*
```

Run all checks that are MEDIUM severity or higher (requires API key):
```sh
checkov -d . --check MEDIUM --bc-api-key ...
```

Run all checks that are MEDIUM severity or higher, as well as check CKV_123 (assume this is a LOW severity check):
```sh
checkov -d . --check MEDIUM,CKV_123 --bc-api-key ...
```

Skip all checks that are MEDIUM severity or lower:
```sh
checkov -d . --skip-check MEDIUM --bc-api-key ...
```

Skip all checks that are MEDIUM severity or lower, as well as check CKV_789 (assume this is a high severity check):
```sh
checkov -d . --skip-check MEDIUM,CKV_789 --bc-api-key ...
```

Run all checks that are MEDIUM severity or higher, but skip check CKV_123 (assume this is a medium or higher severity check):
```sh
checkov -d . --check MEDIUM --skip-check CKV_123 --bc-api-key ...
```

Run check CKV_789, but skip it if it is a medium severity (the --check logic is always applied before --skip-check)
```sh
checkov -d . --skip-check MEDIUM --check CKV_789 --bc-api-key ...
```

For Kubernetes workloads, you can also use allow/deny namespaces.  For example, do not report any results for the
kube-system namespace:
```sh
checkov -d . --skip-check kube-system
```

Run a scan of a container image. First pull or build the image then refer to it by the hash, ID, or name:tag:
```sh
checkov --framework sca_image --docker-image sha256:1234example --dockerfile-path /Users/path/to/Dockerfile --bc-api-key ...

checkov --docker-image <image-name>:tag --dockerfile-path /User/path/to/Dockerfile --bc-api-key ...
```

You can use --image flag also to scan container image instead of --docker-image for shortener:
```sh
checkov --image <image-name>:tag --dockerfile-path /User/path/to/Dockerfile --bc-api-key ...
```

Run an SCA scan of packages in a repo:
```sh
checkov -d . --framework sca_package --bc-api-key ... --repo-id <repo_id(arbitrary)>
```

Run a scan of a directory with environment variables removing buffering, adding debug level logs:
```sh
PYTHONUNBUFFERED=1 LOG_LEVEL=DEBUG checkov -d .
```
OR enable the environment variables for multiple runs
```sh
export PYTHONUNBUFFERED=1 LOG_LEVEL=DEBUG
checkov -d .
```

Run secrets scanning on all files in MyDirectory. Skip CKV_SECRET_6 check on json files that their suffix is DontScan
```sh
checkov -d /MyDirectory --framework secrets --bc-api-key ... --skip-check CKV_SECRET_6:.*DontScan.json$
```

Run secrets scanning on all files in MyDirectory. Skip CKV_SECRET_6 check on json files that contains "skip_test" in path
```sh
checkov -d /MyDirectory --framework secrets --bc-api-key ... --skip-check CKV_SECRET_6:.*skip_test.*json$
```

One can mask values from scanning results by supplying a configuration file (using --config-file flag) with mask entry.
The masking can apply on resource & value (or multiple values, seperated with a comma).
Examples:
```sh
mask:
- aws_instance:user_data
- azurerm_key_vault_secret:admin_password,user_passwords
```
In the example above, the following values will be masked:
- user_data for aws_instance resource
- both admin_password &user_passwords for azurerm_key_vault_secret

### Suppressing/Ignoring a check

Like any static-analysis tool it is limited by its analysis scope.
For example, if a resource is managed manually, or using subsequent configuration management tooling,
suppression can be inserted as a simple code annotation.

#### Suppression comment format

To skip a check on a given Terraform definition block or CloudFormation resource, apply the following comment pattern inside it's scope:

`checkov:skip=<check_id>:<suppression_comment>`

* `<check_id>` is one of the [available check scanners](docs/5.Policy Index/all.md)
* `<suppression_comment>` is an optional suppression reason to be included in the output

#### Example

The following comment skips the `CKV_AWS_20` check on the resource identified by `foo-bucket`, where the scan checks if an AWS S3 bucket is private.
In the example, the bucket is configured with public read access; Adding the suppress comment would skip the appropriate check instead of the check to fail.

```hcl-terraform
resource "aws_s3_bucket" "foo-bucket" {
  region        = var.region
    #checkov:skip=CKV_AWS_20:The bucket is a public static content host
  bucket        = local.bucket_name
  force_destroy = true
  acl           = "public-read"
}
```

The output would now contain a ``SKIPPED`` check result entry:

```bash
...
...
Check: "S3 Bucket has an ACL defined which allows public access."
	SKIPPED for resource: aws_s3_bucket.foo-bucket
	Suppress comment: The bucket is a public static content host
	File: /example_skip_acl.tf:1-25

...
```
To skip multiple checks, add each as a new line.

```
  #checkov:skip=CKV2_AWS_6
  #checkov:skip=CKV_AWS_20:The bucket is a public static content host
```

To suppress checks in Kubernetes manifests, annotations are used with the following format:
`checkov.io/skip#: <check_id>=<suppression_comment>`

For example:

```bash
apiVersion: v1
kind: Pod
metadata:
  name: mypod
  annotations:
    checkov.io/skip1: CKV_K8S_20=I don't care about Privilege Escalation :-O
    checkov.io/skip2: CKV_K8S_14
    checkov.io/skip3: CKV_K8S_11=I have not set CPU limits as I want BestEffort QoS
spec:
  containers:
...
```

#### Logging

For detailed logging to stdout set up the environment variable `LOG_LEVEL` to `DEBUG`.

Default is `LOG_LEVEL=WARNING`.

#### Skipping directories
To skip files or directories, use the argument `--skip-path`, which can be specified multiple times. This argument accepts regular expressions for paths relative to the current working directory. You can use it to skip entire directories and / or specific files.

By default, all directories named `node_modules`, `.terraform`, and `.serverless` will be skipped, in addition to any files or directories beginning with `.`.
To cancel skipping directories beginning with `.` override `CKV_IGNORE_HIDDEN_DIRECTORIES` environment variable `export CKV_IGNORE_HIDDEN_DIRECTORIES=false`

You can override the default set of directories to skip by setting the environment variable `CKV_IGNORED_DIRECTORIES`.
 Note that if you want to preserve this list and add to it, you must include these values. For example, `CKV_IGNORED_DIRECTORIES=mynewdir` will skip only that directory, but not the others mentioned above. This variable is legacy functionality; we recommend using the `--skip-file` flag.

#### Console Output

The console output is in colour by default, to switch to a monochrome output, set the environment variable:
`ANSI_COLORS_DISABLED`

#### VS Code Extension

If you want to use Checkov within VS Code, give a try to the vscode extension available at [VS Code](https://marketplace.visualstudio.com/items?itemName=Bridgecrew.checkov)

### Configuration using a config file

Checkov can be configured using a YAML configuration file. By default, checkov looks for a `.checkov.yaml` or `.checkov.yml` file in the following places in order of precedence:
* Directory against which checkov is run. (`--directory`)
* Current working directory where checkov is called.
* User's home directory.

**Attention**: it is a best practice for checkov configuration file to be loaded from a trusted source composed by a verified identity, so that scanned files, check ids and loaded custom checks are as desired.

Users can also pass in the path to a config file via the command line. In this case, the other config files will be ignored. For example:
```sh
checkov --config-file path/to/config.yaml
```
Users can also create a config file using the `--create-config` command, which takes the current command line args and writes them out to a given path. For example:
```sh
checkov --compact --directory test-dir --docker-image sample-image --dockerfile-path Dockerfile --download-external-modules True --external-checks-dir sample-dir --quiet --repo-id prisma-cloud/sample-repo --skip-check CKV_DOCKER_3,CKV_DOCKER_2 --skip-framework dockerfile secrets --soft-fail --branch develop --check CKV_DOCKER_1 --create-config /Users/sample/config.yml
```
Will create a `config.yaml` file which looks like this:
```yaml
branch: develop
check:
  - CKV_DOCKER_1
compact: true
directory:
  - test-dir
docker-image: sample-image
dockerfile-path: Dockerfile
download-external-modules: true
evaluate-variables: true
external-checks-dir:
  - sample-dir
external-modules-download-path: .external_modules
framework:
  - all
output: cli
quiet: true
repo-id: prisma-cloud/sample-repo
skip-check:
  - CKV_DOCKER_3
  - CKV_DOCKER_2
skip-framework:
  - dockerfile
  - secrets
soft-fail: true
```

Users can also use the `--show-config` flag to view all the args and settings and where they came from i.e. commandline, config file, environment variable or default. For example:
```sh
checkov --show-config
```
Will display:
```sh
Command Line Args:   --show-config
Environment Variables:
  BC_API_KEY:        your-api-key
Config File (/Users/sample/.checkov.yml):
  soft-fail:         False
  branch:            master
  skip-check:        ['CKV_DOCKER_3', 'CKV_DOCKER_2']
Defaults:
  --output:          cli
  --framework:       ['all']
  --download-external-modules:False
  --external-modules-download-path:.external_modules
  --evaluate-variables:True
```

## Contributing

Contribution is welcomed!

Start by reviewing the [contribution guidelines](https://github.com/bridgecrewio/checkov/blob/main/CONTRIBUTING.md). After that, take a look at a [good first issue](https://github.com/bridgecrewio/checkov/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).

You can even start this with one-click dev in your browser through Gitpod at the following link:

[![Open in Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#https://github.com/bridgecrewio/checkov)

Looking to contribute new checks? Learn how to write a new check (AKA policy) [here](https://github.com/bridgecrewio/checkov/blob/main/docs/6.Contribution/Contribution%20Overview.md).

## Disclaimer
`checkov` does not save, publish or share with anyone any identifiable customer information.
No identifiable customer information is used to query Prisma Cloud's publicly accessible guides.
`checkov` uses Prisma Cloud's API to enrich the results with links to remediation guides.
To skip this API call use the flag `--skip-download`.

## Support

[Prisma Cloud](https://www.prismacloud.io/?utm_source=github&utm_medium=organic_oss&utm_campaign=checkov) builds and maintains Checkov to make policy-as-code simple and accessible.

Start with our [Documentation](https://www.checkov.io/1.Welcome/Quick%20Start.html) for quick tutorials and examples.

## Python Version Support
We follow the official support cycle of Python, and we use automated tests for all supported versions of Python. This means we currently support Python 3.8 - 3.11, inclusive. Note that Python 3.7 reached EOL on June 2023 and Python 3.8 will reach EOL in October 2024. If you run into any issues with any non-EOL Python version, please open an Issue.


# TF-Agents: A reliable, scalable and easy to use TensorFlow library for Contextual Bandits and Reinforcement Learning.

[![PyPI tf-agents](https://badge.fury.io/py/tf-agents.svg)](https://badge.fury.io/py/tf-agents)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/tf-agents)

[TF-Agents](https://github.com/tensorflow/agents) makes implementing, deploying,
and testing new Bandits and RL algorithms easier. It provides well tested and
modular components that can be modified and extended. It enables fast code
iteration, with good test integration and benchmarking.

To get started, we recommend checking out one of our Colab tutorials. If you
need an intro to RL (or a quick recap),
[start here](docs/tutorials/0_intro_rl.ipynb). Otherwise, check out our
[DQN tutorial](docs/tutorials/1_dqn_tutorial.ipynb) to get an agent up and
running in the Cartpole environment. API documentation for the current stable
release is on
[tensorflow.org](https://www.tensorflow.org/agents/api_docs/python/tf_agents).

TF-Agents is under active development and interfaces may change at any time.
Feedback and comments are welcome.

## Table of contents

Agents

Tutorials

Multi-Armed Bandits

Examples

Installation

Contributing

Releases

Principles

Contributors

Citation

Disclaimer

## Agents

In TF-Agents, the core elements of RL algorithms are implemented as `Agents`. An
agent encompasses two main responsibilities: defining a Policy to interact with
the Environment, and how to learn/train that Policy from collected experience.

Currently the following algorithms are available under TF-Agents:

*   [DQN: __Human level control through deep reinforcement learning__ Mnih et
    al., 2015](https://deepmind.com/research/dqn/)
*   [DDQN: __Deep Reinforcement Learning with Double Q-learning__ Hasselt et
    al., 2015](https://arxiv.org/abs/1509.06461)
*   [DDPG: __Continuous control with deep reinforcement learning__ Lillicrap et
    al., 2015](https://arxiv.org/abs/1509.02971)
*   [TD3: __Addressing Function Approximation Error in Actor-Critic Methods__
    Fujimoto et al., 2018](https://arxiv.org/abs/1802.09477)
*   [REINFORCE: __Simple Statistical Gradient-Following Algorithms for
    Connectionist Reinforcement Learning__ Williams,
    1992](https://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)
*   [PPO: __Proximal Policy Optimization Algorithms__ Schulman et al., 2017](https://arxiv.org/abs/1707.06347)
*   [SAC: __Soft Actor Critic__ Haarnoja et al., 2018](https://arxiv.org/abs/1812.05905)

## Tutorials

See [`docs/tutorials/`](docs/tutorials) for tutorials on the major components
provided.

## Multi-Armed Bandits

The TF-Agents library contains a comprehensive Multi-Armed Bandits suite,
including Bandits environments and agents. RL agents can also be used on Bandit
environments. There is a tutorial in
[`bandits_tutorial.ipynb`](https://github.com/tensorflow/agents/tree/master/docs/tutorials/bandits_tutorial.ipynb).
and ready-to-run examples in
[`tf_agents/bandits/agents/examples/v2`](https://github.com/tensorflow/agents/tree/master/tf_agents/bandits/agents/examples/v2).

## Examples

End-to-end examples training agents can be found under each agent directory.
e.g.:

*   DQN:
    [`tf_agents/agents/dqn/examples/v2/train_eval.py`](https://github.com/tensorflow/agents/tree/master/tf_agents/agents/dqn/examples/v2/train_eval.py)

## Installation

TF-Agents publishes nightly and stable builds. For a list of releases read the
Releases section. The commands below cover installing
TF-Agents stable and nightly from [pypi.org](https://pypi.org) as well as from a
GitHub clone.

> :warning: If using Reverb (replay buffer), which is very common,
TF-Agents will only work with Linux.

> Note: Python 3.11 requires pygame 2.1.3+.

### Stable

Run the commands below to install the most recent stable release. API
documentation for the release is on
[tensorflow.org](https://www.tensorflow.org/agents/api_docs/python/tf_agents).

```shell
$ pip install --user tf-agents[reverb]

# Use keras-2
$ export TF_USE_LEGACY_KERAS=1
# Use this tag get the matching examples and colabs.
$ git clone https://github.com/tensorflow/agents.git
$ cd agents
$ git checkout v0.18.0
```

If you want to install TF-Agents with versions of Tensorflow or
[Reverb](https://github.com/deepmind/reverb) that are flagged as not compatible
by the pip dependency check, use the following pattern below at your own risk.

```shell
$ pip install --user tensorflow
$ pip install --user tf-keras
$ pip install --user dm-reverb
$ pip install --user tf-agents
```

If you want to use TF-Agents with TensorFlow 1.15 or 2.0, install version 0.3.0:

```shell
# Newer versions of tensorflow-probability require newer versions of TensorFlow.
$ pip install tensorflow-probability==0.8.0
$ pip install tf-agents==0.3.0
```

### Nightly

Nightly builds include newer features, but may be less stable than the versioned
releases. The nightly build is pushed as `tf-agents-nightly`. We suggest
installing nightly versions of TensorFlow (`tf-nightly`) and TensorFlow
Probability (`tfp-nightly`) as those are the versions TF-Agents nightly are
tested against.

To install the nightly build version, run the following:

```shell
# Use keras-2
$ export TF_USE_LEGACY_KERAS=1

# `--force-reinstall helps guarantee the right versions.
$ pip install --user --force-reinstall tf-nightly
$ pip install --user --force-reinstall tf-keras-nightly
$ pip install --user --force-reinstall tfp-nightly
$ pip install --user --force-reinstall dm-reverb-nightly

# Installing with the `--upgrade` flag ensures you'll get the latest version.
$ pip install --user --upgrade tf-agents-nightly
```

### From GitHub

After cloning the repository, the dependencies can be installed by running `pip
install -e .[tests]`. TensorFlow needs to be installed independently: `pip
install --user tf-nightly`.

## Contributing

We're eager to collaborate with you! See [`CONTRIBUTING.md`](CONTRIBUTING.md)
for a guide on how to contribute. This project adheres to TensorFlow's
[code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to
uphold this code.

## Releases

TF Agents has stable and nightly releases. The nightly releases are often fine
but can have issues due to upstream libraries being in flux. The table below
lists the version(s) of TensorFlow that align with each TF Agents' release.
Release versions of interest:

  * 0.19.0 supports tensorflow-2.15.0.
  * 0.18.0 dropped Python 3.8 support.
  * 0.16.0 is the first version to support Python 3.11.
  * 0.15.0 is the last release compatible with Python 3.7.
  * If using numpy < 1.19, then use TF-Agents 0.15.0 or earlier.
  * 0.9.0 is the last release compatible with Python 3.6.
  * 0.3.0 is the last release compatible with Python 2.x.

Release | Branch / Tag                                               | TensorFlow Version | dm-reverb Version
------- | ---------------------------------------------------------- | ------------------ | -----------
Nightly | [master](https://github.com/tensorflow/agents)             | tf-nightly         | dm-reverb-nightly
0.19.0  | [v0.19.0](https://github.com/tensorflow/agents/tree/v0.19.0) | 2.15.0           | 0.14.0
0.18.0  | [v0.18.0](https://github.com/tensorflow/agents/tree/v0.18.0) | 2.14.0           | 0.13.0
0.17.0  | [v0.17.0](https://github.com/tensorflow/agents/tree/v0.17.0) | 2.13.0           | 0.12.0
0.16.0  | [v0.16.0](https://github.com/tensorflow/agents/tree/v0.16.0) | 2.12.0           | 0.11.0
0.15.0  | [v0.15.0](https://github.com/tensorflow/agents/tree/v0.15.0) | 2.11.0           | 0.10.0
0.14.0  | [v0.14.0](https://github.com/tensorflow/agents/tree/v0.14.0) | 2.10.0           | 0.9.0
0.13.0  | [v0.13.0](https://github.com/tensorflow/agents/tree/v0.13.0) | 2.9.0            | 0.8.0
0.12.0  | [v0.12.0](https://github.com/tensorflow/agents/tree/v0.12.0) | 2.8.0            | 0.7.0
0.11.0  | [v0.11.0](https://github.com/tensorflow/agents/tree/v0.11.0) | 2.7.0            | 0.6.0
0.10.0  | [v0.10.0](https://github.com/tensorflow/agents/tree/v0.10.0) | 2.6.0            |
0.9.0   | [v0.9.0](https://github.com/tensorflow/agents/tree/v0.9.0) | 2.6.0              |
0.8.0   | [v0.8.0](https://github.com/tensorflow/agents/tree/v0.8.0) | 2.5.0              |
0.7.1   | [v0.7.1](https://github.com/tensorflow/agents/tree/v0.7.1) | 2.4.0              |
0.6.0   | [v0.6.0](https://github.com/tensorflow/agents/tree/v0.6.0) | 2.3.0              |
0.5.0   | [v0.5.0](https://github.com/tensorflow/agents/tree/v0.5.0) | 2.2.0              |
0.4.0   | [v0.4.0](https://github.com/tensorflow/agents/tree/v0.4.0) | 2.1.0              |
0.3.0   | [v0.3.0](https://github.com/tensorflow/agents/tree/v0.3.0) | 1.15.0 and 2.0.0.  |

## Principles

This project adheres to [Google's AI principles](PRINCIPLES.md). By
participating, using or contributing to this project you are expected to adhere
to these principles.

## Contributors

We would like to recognize the following individuals for their code
contributions, discussions, and other work to make the TF-Agents library.

* James Davidson
* Ethan Holly
* Toby Boyd
* Summer Yue
* Robert Ormandi
* Kuang-Huei Lee
* Alexa Greenberg
* Amir Yazdanbakhsh
* Yao Lu
* Gaurav Jain
* Christof Angermueller
* Mark Daoust
* Adam Wood

## Citation

If you use this code, please cite it as:

```
@misc{TFAgents,
  title = {{TF-Agents}: A library for Reinforcement Learning in TensorFlow},
  author = {Sergio Guadarrama and Anoop Korattikara and Oscar Ramirez and
     Pablo Castro and Ethan Holly and Sam Fishman and Ke Wang and
     Ekaterina Gonina and Neal Wu and Efi Kokiopoulou and Luciano Sbaiz and
     Jamie Smith and Gábor Bartók and Jesse Berent and Chris Harris and
     Vincent Vanhoucke and Eugene Brevdo},
  howpublished = {\url{https://github.com/tensorflow/agents}},
  url = "https://github.com/tensorflow/agents",
  year = 2018,
  note = "[Online; accessed 25-June-2019]"
}
```

## Disclaimer

This is not an official Google product.


[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=LibrePhotos_ownphotos&metric=alert_status)](https://sonarcloud.io/dashboard?id=LibrePhotos_ownphotos) [![Discord](https://img.shields.io/discord/784619049208250388?style=plastic)][discord] [![Website](https://img.shields.io/website?down_color=lightgrey&down_message=offline&style=plastic&up_color=blue&up_message=online&url=https%3A%2F%2Flibrephotos.com)](https://librephotos.com/)
[![Read the docs](https://img.shields.io/static/v1?label=Read&message=the%20docs&color=blue&style=plastic)](https://docs.librephotos.com/) [![GitHub contributors](https://img.shields.io/github/contributors/librephotos/librephotos?style=plastic)](https://github.com/LibrePhotos/librephotos/graphs/contributors)

# LibrePhotos

![](https://github.com/LibrePhotos/librephotos/blob/dev/screenshots/mockups_main_fhd.png?raw=true)
Mockup designed by rawpixel.com / Freepik

- **Stable** demo is available here:https://demo1.librephotos.com/ . User is ```demo```, password is ```demo1234``` (with sample images).
- Latest **development** demo is available here: https://demo2.librephotos.com/ (same user/password)
- You can watch development videos on [Niaz Faridani-Rad's channel](https://www.youtube.com/channel/UCZJ2pk2BPKxwbuCV9LWDR0w)
- You can join our [Discord][discord].

## Installation

Step-by-step installation instructions are available in our [documentation](https://docs.librephotos.com/docs/installation/standard-install)

## How to help out
- ⭐ **Star** this repository if you like this project!
- 🚀 **Developing**: Get started in less than 30 minutes by following [this guide](https://docs.librephotos.com/docs/development/dev-install).
- 🗒 **Documentation**: Improving the documentation is as simple as submitting a pull request [here](https://github.com/LibrePhotos/librephotos.docs)
- 🧪 **Testing**: If you want to help find bugs, use the ```dev``` tag and update it regularly. If you find a bug, open an issue.
- 🧑🤝🧑 **Outreach**: Talk about this project with other people and help them to get started too!
- 🌐 **Translations**: Make LibrePhotos accessible to more people with [weblate](https://hosted.weblate.org/engage/librephotos/).
- 💸 [**Donate**](https://github.com/sponsors/derneuere) to the developers of LibrePhotos

## Features

  - Support for all types of photos including raw photos
  - Support for videos
  - Timeline view
  - Scans pictures on the file system
  - Multiuser support
  - Generate albums based on events like "Thursday in Berlin"
  - Face recognition / Face classification
  - Reverse geocoding
  - Object / Scene detection
  - Semantic image search
  - Search by metadata

## What does it use?

- **Image Conversion:** [ImageMagick](https://github.com/ImageMagick/ImageMagick)
- **Video Conversion:** [FFmpeg](https://github.com/FFmpeg/FFmpeg)
- **Exif Support:** [ExifTool](https://github.com/exiftool/exiftool)
- **Face detection:** [face_recognition](https://github.com/ageitgey/face_recognition)
- **Face classification/clusterization:** [scikit-learn](https://scikit-learn.org/) and [hdbscan](https://github.com/scikit-learn-contrib/hdbscan)
- **Image captioning:** [im2txt](https://github.com/HughKu/Im2txt),
- **Scene classification** [places365](http://places.csail.mit.edu/)
- **Reverse geocoding:** [Mapbox](https://www.mapbox.com/): You need to have an API key. First 50,000 geocode lookups are free every month.

[discord]: https://discord.gg/xwRvtSDGWb


[![Returns logo](https://raw.githubusercontent.com/dry-python/brand/master/logo/returns_white-outline.png)](https://github.com/dry-python/returns)

-----

[![Build Status](https://github.com/dry-python/returns/workflows/test/badge.svg?branch=master&event=push)](https://github.com/dry-python/returns/actions?query=workflow%3Atest)
[![codecov](https://codecov.io/gh/dry-python/returns/branch/master/graph/badge.svg)](https://codecov.io/gh/dry-python/returns)
[![Documentation Status](https://readthedocs.org/projects/returns/badge/?version=latest)](https://returns.readthedocs.io/en/latest/?badge=latest)
[![Python Version](https://img.shields.io/pypi/pyversions/returns.svg)](https://pypi.org/project/returns/)
[![conda](https://img.shields.io/conda/v/conda-forge/returns?label=conda)](https://anaconda.org/conda-forge/returns)
[![wemake-python-styleguide](https://img.shields.io/badge/style-wemake-000000.svg)](https://github.com/wemake-services/wemake-python-styleguide)
[![Telegram chat](https://img.shields.io/badge/chat-join-blue?logo=telegram)](https://t.me/drypython)

-----

Make your functions return something meaningful, typed, and safe!

## Features

- Brings functional programming to Python land
- Provides a bunch of primitives to write declarative business logic
- Enforces better architecture
- Fully typed with annotations and checked with `mypy`, [PEP561 compatible](https://www.python.org/dev/peps/pep-0561/)
- Adds emulated Higher Kinded Types support
- Provides type-safe interfaces to create your own data-types with enforced laws
- Has a bunch of helpers for better composition
- Pythonic and pleasant to write and to read 🐍
- Support functions and coroutines, framework agnostic
- Easy to start: has lots of docs, tests, and tutorials

[Quickstart](https://returns.readthedocs.io/en/latest/pages/quickstart.html) right now!

## Installation

```bash
pip install returns
```

You can also install `returns` with the latest supported `mypy` version:

```bash
pip install returns[compatible-mypy]
```

You would also need to configure our [`mypy` plugin](https://returns.readthedocs.io/en/latest/pages/contrib/mypy_plugins.html):

```ini
# In setup.cfg or mypy.ini:
[mypy]
plugins =
  returns.contrib.mypy.returns_plugin
```

or:

```toml
[tool.mypy]
plugins = ["returns.contrib.mypy.returns_plugin"]
```

We also recommend to use the same `mypy` settings [we use](https://github.com/wemake-services/wemake-python-styleguide/blob/master/styles/mypy.toml).

Make sure you know how to get started, [check out our docs](https://returns.readthedocs.io/en/latest/)!
[Try our demo](https://repl.it/@sobolevn/returns#ex.py).

## Contents

- [Maybe container](#maybe-container) that allows you to write `None`-free code
- [RequiresContext container](#requirescontext-container) that allows you to use typed functional dependency injection
- [Result container](#result-container) that let's you to get rid of exceptions
- [IO container](#io-container) and [IOResult](#troublesome-io) that marks all impure operations and structures them
- [Future container](#future-container) and [FutureResult](#async-code-without-exceptions) to work with `async` code
- [Write your own container!](https://returns.readthedocs.io/en/latest/pages/create-your-own-container.html) You would still have all the features for your own types (including full existing code reuse and type-safety)
- Use [`do-notation`](https://returns.readthedocs.io/en/latest/pages/do-notation.html) to make your code easier!

## Maybe container

`None` is called the [worst mistake in the history of Computer Science](https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare/).

So, what can we do to check for `None` in our programs?
You can use builtin [Optional](https://mypy.readthedocs.io/en/stable/kinds_of_types.html#optional-types-and-the-none-type) type
and write a lot of `if some is not None:` conditions.
But, **having `null` checks here and there makes your code unreadable**.

```python
user: Optional[User]
discount_program: Optional['DiscountProgram'] = None

if user is not None:
     balance = user.get_balance()
     if balance is not None:
         credit = balance.credit_amount()
         if credit is not None and credit > 0:
             discount_program = choose_discount(credit)
```

Or you can use
[Maybe](https://returns.readthedocs.io/en/latest/pages/maybe.html) container!
It consists of `Some` and `Nothing` types,
representing existing state and empty (instead of `None`) state respectively.

```python
from typing import Optional
from returns.maybe import Maybe, maybe

@maybe  # decorator to convert existing Optional[int] to Maybe[int]
def bad_function() -> Optional[int]:
    ...

maybe_number: Maybe[float] = bad_function().bind_optional(
    lambda number: number / 2,
)
# => Maybe will return Some[float] only if there's a non-None value
#    Otherwise, will return Nothing
```

You can be sure that `.bind_optional()` method won't be called for `Nothing`.
Forget about `None`-related errors forever!

We can also bind a `Optional`-returning function over a container.
To achieve this, we are going to use `.bind_optional` method.

And here's how your initial refactored code will look:

```python
user: Optional[User]

# Type hint here is optional, it only helps the reader here:
discount_program: Maybe['DiscountProgram'] = Maybe.from_optional(
    user,
).bind_optional(  # This won't be called if `user is None`
    lambda real_user: real_user.get_balance(),
).bind_optional(  # This won't be called if `real_user.get_balance()` is None
    lambda balance: balance.credit_amount(),
).bind_optional(  # And so on!
    lambda credit: choose_discount(credit) if credit > 0 else None,
)
```

Much better, isn't it?

## RequiresContext container

Many developers do use some kind of dependency injection in Python.
And usually it is based on the idea
that there's some kind of a container and assembly process.

Functional approach is much simpler!

Imagine that you have a `django` based game, where you award users with points for each guessed letter in a word (unguessed letters are marked as `'.'`):

```python
from django.http import HttpRequest, HttpResponse
from words_app.logic import calculate_points

def view(request: HttpRequest) -> HttpResponse:
    user_word: str = request.POST['word']  # just an example
    points = calculate_points(user_word)
    ...  # later you show the result to user somehow

# Somewhere in your `words_app/logic.py`:

def calculate_points(word: str) -> int:
    guessed_letters_count = len([letter for letter in word if letter != '.'])
    return _award_points_for_letters(guessed_letters_count)

def _award_points_for_letters(guessed: int) -> int:
    return 0 if guessed < 5 else guessed  # minimum 6 points possible!
```

Awesome! It works, users are happy, your logic is pure and awesome.
But, later you decide to make the game more fun:
let's make the minimal accountable letters threshold
configurable for an extra challenge.

You can just do it directly:

```python
def _award_points_for_letters(guessed: int, threshold: int) -> int:
    return 0 if guessed < threshold else guessed
```

The problem is that `_award_points_for_letters` is deeply nested.
And then you have to pass `threshold` through the whole callstack,
including `calculate_points` and all other functions that might be on the way.
All of them will have to accept `threshold` as a parameter!
This is not useful at all!
Large code bases will struggle a lot from this change.

Ok, you can directly use `django.settings` (or similar)
in your `_award_points_for_letters` function.
And **ruin your pure logic with framework specific details**. That's ugly!

Or you can use `RequiresContext` container. Let's see how our code changes:

```python
from django.conf import settings
from django.http import HttpRequest, HttpResponse
from words_app.logic import calculate_points

def view(request: HttpRequest) -> HttpResponse:
    user_word: str = request.POST['word']  # just an example
    points = calculate_points(user_words)(settings)  # passing the dependencies
    ...  # later you show the result to user somehow

# Somewhere in your `words_app/logic.py`:

from typing import Protocol
from returns.context import RequiresContext

class _Deps(Protocol):  # we rely on abstractions, not direct values or types
    WORD_THRESHOLD: int

def calculate_points(word: str) -> RequiresContext[int, _Deps]:
    guessed_letters_count = len([letter for letter in word if letter != '.'])
    return _award_points_for_letters(guessed_letters_count)

def _award_points_for_letters(guessed: int) -> RequiresContext[int, _Deps]:
    return RequiresContext(
        lambda deps: 0 if guessed < deps.WORD_THRESHOLD else guessed,
    )
```

And now you can pass your dependencies in a really direct and explicit way.
And have the type-safety to check what you pass to cover your back.
Check out [RequiresContext](https://returns.readthedocs.io/en/latest/pages/context.html) docs for more. There you will learn how to make `'.'` also configurable.

We also have [RequiresContextResult](https://returns.readthedocs.io/en/latest/pages/context.html#requirescontextresult-container)
for context-related operations that might fail. And also [RequiresContextIOResult](https://returns.readthedocs.io/en/latest/pages/context.html#requirescontextioresult-container) and [RequiresContextFutureResult](https://returns.readthedocs.io/en/latest/pages/context.html#requirescontextfutureresult-container).

## Result container

Please, make sure that you are also aware of
[Railway Oriented Programming](https://fsharpforfunandprofit.com/rop/).

### Straight-forward approach

Consider this code that you can find in **any** `python` project.

```python
import requests

def fetch_user_profile(user_id: int) -> 'UserProfile':
    """Fetches UserProfile dict from foreign API."""
    response = requests.get('/api/users/{0}'.format(user_id))
    response.raise_for_status()
    return response.json()
```

Seems legit, does it not?
It also seems like a pretty straightforward code to test.
All you need is to mock `requests.get` to return the structure you need.

But, there are hidden problems in this tiny code sample
that are almost impossible to spot at the first glance.

### Hidden problems

Let's have a look at the exact same code,
but with the all hidden problems explained.

```python
import requests

def fetch_user_profile(user_id: int) -> 'UserProfile':
    """Fetches UserProfile dict from foreign API."""
    response = requests.get('/api/users/{0}'.format(user_id))

    # What if we try to find user that does not exist?
    # Or network will go down? Or the server will return 500?
    # In this case the next line will fail with an exception.
    # We need to handle all possible errors in this function
    # and do not return corrupt data to consumers.
    response.raise_for_status()

    # What if we have received invalid JSON?
    # Next line will raise an exception!
    return response.json()
```

Now, all (probably all?) problems are clear.
How can we be sure that this function will be safe
to use inside our complex business logic?

We really cannot be sure!
We will have to create **lots** of `try` and `except` cases
just to catch the expected exceptions. Our code will become complex and unreadable with all this mess!

Or we can go with the top level `except Exception:` case
to catch literally everything.
And this way we would end up with catching unwanted ones.
This approach can hide serious problems from us for a long time.

### Pipe example

```python
import requests
from returns.result import Result, safe
from returns.pipeline import flow
from returns.pointfree import bind

def fetch_user_profile(user_id: int) -> Result['UserProfile', Exception]:
    """Fetches `UserProfile` TypedDict from foreign API."""
    return flow(
        user_id,
        _make_request,
        bind(_parse_json),
    )

@safe
def _make_request(user_id: int) -> requests.Response:
    # TODO: we are not yet done with this example, read more about `IO`:
    response = requests.get('/api/users/{0}'.format(user_id))
    response.raise_for_status()
    return response

@safe
def _parse_json(response: requests.Response) -> 'UserProfile':
    return response.json()
```

Now we have a clean and a safe and declarative way
to express our business needs:

- We start from making a request, that might fail at any moment,
- Then parsing the response if the request was successful,
- And then return the result.

Now, instead of returning regular values
we return values wrapped inside a special container
thanks to the
[@safe](https://returns.readthedocs.io/en/latest/pages/result.html#safe)
decorator. It will return [Success[YourType] or Failure[Exception]](https://returns.readthedocs.io/en/latest/pages/result.html).
And will never throw exception at us!

We also use [flow](https://returns.readthedocs.io/en/latest/pages/pipeline.html#flow)
and [bind](https://returns.readthedocs.io/en/latest/pages/pointfree.html#bind)
functions for handy and declarative composition.

This way we can be sure that our code won't break in
random places due to some implicit exception.
Now we control all parts and are prepared for the explicit errors.

We are not yet done with this example,
let's continue to improve it in the next chapter.

## IO container

Let's look at our example from another angle.
All its functions look like regular ones:
it is impossible to tell whether they are [pure](https://en.wikipedia.org/wiki/Pure_function)
or impure from the first sight.

It leads to a very important consequence:
*we start to mix pure and impure code together*.
We should not do that!

When these two concepts are mixed
we suffer really bad when testing or reusing it.
Almost everything should be pure by default.
And we should explicitly mark impure parts of the program.

That's why we have created `IO` container
to mark impure functions that never fail.

These impure functions use `random`, current datetime, environment, or console:

```python
import random
import datetime as dt

from returns.io import IO

def get_random_number() -> IO[int]:  # or use `@impure` decorator
    return IO(random.randint(1, 10))  # isn't pure, because random

now: Callable[[], IO[dt.datetime]] = impure(dt.datetime.now)

@impure
def return_and_show_next_number(previous: int) -> int:
    next_number = previous + 1
    print(next_number)  # isn't pure, because does IO
    return next_number
```

Now we can clearly see which functions are pure and which ones are impure.
This helps us a lot in building large applications, unit testing you code,
and composing business logic together.

### Troublesome IO

As it was already said, we use `IO` when we handle functions that do not fail.

What if our function can fail and is impure?
Like `requests.get()` we had earlier in our example.

Then we have to use a special `IOResult` type instead of a regular `Result`.
Let's find the difference:

- Our `_parse_json` function always returns
  the same result (hopefully) for the same input:
  you can either parse valid `json` or fail on invalid one.
  That's why we return pure `Result`, there's no `IO` inside
- Our `_make_request` function is impure and can fail.
  Try to send two similar requests with and without internet connection.
  The result will be different for the same input.
  That's why we must use `IOResult` here: it can fail and has `IO`

So, in order to fulfill our requirement and separate pure code from impure one,
we have to refactor our example.

### Explicit IO

Let's make our [IO](https://returns.readthedocs.io/en/latest/pages/io.html)
explicit!

```python
import requests
from returns.io import IOResult, impure_safe
from returns.result import safe
from returns.pipeline import flow
from returns.pointfree import bind_result

def fetch_user_profile(user_id: int) -> IOResult['UserProfile', Exception]:
    """Fetches `UserProfile` TypedDict from foreign API."""
    return flow(
        user_id,
        _make_request,
        # before: def (Response) -> UserProfile
        # after safe: def (Response) -> ResultE[UserProfile]
        # after bind_result: def (IOResultE[Response]) -> IOResultE[UserProfile]
        bind_result(_parse_json),
    )

@impure_safe
def _make_request(user_id: int) -> requests.Response:
    response = requests.get('/api/users/{0}'.format(user_id))
    response.raise_for_status()
    return response

@safe
def _parse_json(response: requests.Response) -> 'UserProfile':
    return response.json()
```

And later we can use [unsafe_perform_io](https://returns.readthedocs.io/en/latest/pages/io.html#unsafe-perform-io)
somewhere at the top level of our program to get the pure (or "real") value.

As a result of this refactoring session, we know everything about our code:

- Which parts can fail,
- Which parts are impure,
- How to compose them in a smart, readable, and typesafe manner.

## Future container

There are several issues with `async` code in Python:

1. You cannot call `async` function from a sync one
2. Any unexpectedly thrown exception can ruin your whole event loop
3. Ugly composition with lots of `await` statements

`Future` and `FutureResult` containers solve these issues!

### Mixing sync and async code

The main feature of [Future](https://returns.readthedocs.io/en/latest/pages/future.html)
is that it allows to run async code
while maintaining sync context. Let's see an example.

Let's say we have two functions,
the `first` one returns a number and the `second` one increments it:

```python
async def first() -> int:
    return 1

def second():  # How can we call `first()` from here?
    return first() + 1  # Boom! Don't do this. We illustrate a problem here.
```

If we try to just run `first()`, we will just create an unawaited coroutine.
It won't return the value we want.

But, if we would try to run `await first()`,
then we would need to change `second` to be `async`.
And sometimes it is not possible for various reasons.

However, with `Future` we can "pretend" to call async code from sync code:

```python
from returns.future import Future

def second() -> Future[int]:
    return Future(first()).map(lambda num: num + 1)
```

Without touching our `first` async function
or making `second` async we have achieved our goal.
Now, our async value is incremented inside a sync function.

However, `Future` still requires to be executed inside a proper eventloop:

```python
import anyio  # or asyncio, or any other lib

# We can then pass our `Future` to any library: asyncio, trio, curio.
# And use any event loop: regular, uvloop, even a custom one, etc
assert anyio.run(second().awaitable) == 2
```

As you can see `Future` allows you
to work with async functions from a sync context.
And to mix these two realms together.
Use raw `Future` for operations that cannot fail or raise exceptions.
Pretty much the same logic we had with our `IO` container.

### Async code without exceptions

We have already covered how [`Result`](#result-container) works
for both pure and impure code.
The main idea is: we don't raise exceptions, we return them.
It is **especially** critical in async code,
because a single exception can ruin
all our coroutines running in a single eventloop.

We have a handy combination of `Future` and `Result` containers: `FutureResult`.
Again, this is exactly like `IOResult`, but for impure async code.
Use it when your `Future` might have problems:
like HTTP requests or filesystem operations.

You can easily turn any wild throwing coroutine into a calm `FutureResult`:

```python
import anyio
from returns.future import future_safe
from returns.io import IOFailure

@future_safe
async def raising():
    raise ValueError('Not so fast!')

ioresult = anyio.run(raising.awaitable)  # all `Future`s return IO containers
assert ioresult == IOFailure(ValueError('Not so fast!'))  # True
```

Using `FutureResult` will keep your code safe from exceptions.
You can always `await` or execute inside an eventloop any `FutureResult`
to get sync `IOResult` instance to work with it in a sync manner.

### Better async composition

Previously, you had to do quite a lot of `await`ing while writing `async` code:

```python
async def fetch_user(user_id: int) -> 'User':
    ...

async def get_user_permissions(user: 'User') -> 'Permissions':
    ...

async def ensure_allowed(permissions: 'Permissions') -> bool:
    ...

async def main(user_id: int) -> bool:
    # Also, don't forget to handle all possible errors with `try / except`!
    user = await fetch_user(user_id)  # We will await each time we use a coro!
    permissions = await get_user_permissions(user)
    return await ensure_allowed(permissions)
```

Some people are ok with it, but some people don't like this imperative style.
The problem is that there was no choice.

But now, you can do the same thing in functional style!
With the help of `Future` and `FutureResult` containers:

```python
import anyio
from returns.future import FutureResultE, future_safe
from returns.io import IOSuccess, IOFailure

@future_safe
async def fetch_user(user_id: int) -> 'User':
    ...

@future_safe
async def get_user_permissions(user: 'User') -> 'Permissions':
    ...

@future_safe
async def ensure_allowed(permissions: 'Permissions') -> bool:
    ...

def main(user_id: int) -> FutureResultE[bool]:
    # We can now turn `main` into a sync function, it does not `await` at all.
    # We also don't care about exceptions anymore, they are already handled.
    return fetch_user(user_id).bind(get_user_permissions).bind(ensure_allowed)

correct_user_id: int  # has required permissions
banned_user_id: int  # does not have required permissions
wrong_user_id: int  # does not exist

# We can have correct business results:
assert anyio.run(main(correct_user_id).awaitable) == IOSuccess(True)
assert anyio.run(main(banned_user_id).awaitable) == IOSuccess(False)

# Or we can have errors along the way:
assert anyio.run(main(wrong_user_id).awaitable) == IOFailure(
    UserDoesNotExistError(...),
)
```

Or even something really fancy:

```python
from returns.pointfree import bind
from returns.pipeline import flow

def main(user_id: int) -> FutureResultE[bool]:
    return flow(
        fetch_user(user_id),
        bind(get_user_permissions),
        bind(ensure_allowed),
    )
```

Later we can also refactor our logical functions to be sync
and to return `FutureResult`.

Lovely, isn't it?

## More!

Want more?
[Go to the docs!](https://returns.readthedocs.io)
Or read these articles:

- [Python exceptions considered an anti-pattern](https://sobolevn.me/2019/02/python-exceptions-considered-an-antipattern)
- [Enforcing Single Responsibility Principle in Python](https://sobolevn.me/2019/03/enforcing-srp)
- [Typed functional Dependency Injection in Python](https://sobolevn.me/2020/02/typed-functional-dependency-injection)
- [How Async Should Have Been](https://sobolevn.me/2020/06/how-async-should-have-been)
- [Higher Kinded Types in Python](https://sobolevn.me/2020/10/higher-kinded-types-in-python)
- [Make Tests a Part of Your App](https://sobolevn.me/2021/02/make-tests-a-part-of-your-app)

Do you have an article to submit? Feel free to open a pull request!


# wemake-python-styleguide

[![wemake.services](https://img.shields.io/badge/%20-wemake.services-green.svg?label=%20&logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAMAAAAoLQ9TAAAABGdBTUEAALGPC%2FxhBQAAAAFzUkdCAK7OHOkAAAAbUExURQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP%2F%2F%2F5TvxDIAAAAIdFJOUwAjRA8xXANAL%2Bv0SAAAADNJREFUGNNjYCAIOJjRBdBFWMkVQeGzcHAwksJnAPPZGOGAASzPzAEHEGVsLExQwE7YswCb7AFZSF3bbAAAAABJRU5ErkJggg%3D%3D)](https://wemake-services.github.io)
[![Supporters](https://img.shields.io/opencollective/all/wemake-python-styleguide.svg?color=gold&label=supporters)](https://opencollective.com/wemake-python-styleguide)
[![Build Status](https://github.com/wemake-services/wemake-python-styleguide/workflows/test/badge.svg?branch=master&event=push)](https://github.com/wemake-services/wemake-python-styleguide/actions?query=workflow%3Atest)
[![codecov](https://codecov.io/gh/wemake-services/wemake-python-styleguide/branch/master/graph/badge.svg)](https://codecov.io/gh/wemake-services/wemake-python-styleguide)
[![Python Version](https://img.shields.io/pypi/pyversions/wemake-python-styleguide.svg)](https://pypi.org/project/wemake-python-styleguide/)
[![wemake-python-styleguide](https://img.shields.io/badge/style-wemake-000000.svg)](https://github.com/wemake-services/wemake-python-styleguide)

---

Welcome to the strictest and most opinionated Python linter ever.

`wemake-python-styleguide` is actually a [flake8](http://flake8.pycqa.org/en/latest/)
plugin with [some other plugins](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/violations/index.html#external-plugins) as dependencies.

## Quickstart

```bash
pip install wemake-python-styleguide
```

You will also need to create a `setup.cfg` file with the [configuration](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/configuration.html).

[Try it online](https://wps.orsinium.dev)!

We highly recommend to also use:

- [ondivi](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/integrations/ondivi.html) for easy integration into a **legacy** codebase
- [nitpick](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/integrations/nitpick.html) for sharing and validating configuration across multiple projects

## Running

```bash
flake8 your_module.py
```

This app is still just good old `flake8`!
And it won't change your existing workflow.

See ["Usage" section](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/setup.html)
in the docs for examples and integrations.

We also support [GitHub Actions](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/integrations/github-actions.html) as first class-citizens.
[Try it out](https://github.com/marketplace/actions/wemake-python-styleguide)!

## Strict is the new cool

Strict linting offers the following benefits to developers and companies:
1. Ensures consistency - no matter who works on it, the end product will always be the same dependable code
2. Helps avoid potential bugs - strict rules make sure that you don't make common mistakes
3. Efficient code reviews - each piece of code has a similar familiar style and syntax. If it passes all the checks, there's little left to review!
4. Fewer code revisions - strict linting ensures that you don't have to re-write the codebase again and again
5. Reduce code redundancy - Sometimes we write complex code as we are thinking in a certain way about a problem. The linter offers suggestions that can help simplify the code and eliminate redundant statements

## What we are about

The ultimate goal of this project is
to make all people write **exactly** the same `Python` code.

|                            | flake8 | pylint | black | mypy | ruff | wemake-python-styleguide |
|----------------------------|--------|--------|-------|------|------|--------------------------|
| Formats code?              |   ❌   |   ❌   |   ✅  |  ❌  |  ✅  |            ❌            |
| Finds style issues?        |   🤔   |   ✅   |   🤔  |  ❌  |  ✅  |            ✅            |
| Finds bugs?                |   🤔   |   ✅   |   ❌  |  ✅  |  ✅  |            ✅            |
| Finds complex code?        |   ❌   |   🤔   |   ❌  |  ❌  |  ✅  |            ✅            |
| Has a lot of strict rules? |   ❌   |   🤔   |   ❌  |  ❌  |  ✅  |            ✅            |
| Has a lot of plugins?      |   ✅   |   ❌   |   ❌  |  🤔  |  ❌  |            ✅            |

We have several primary objectives:

1. Significantly reduce the complexity of your code and make it more maintainable
2. Enforce "There should be one -- and preferably only one -- obvious way to do it" rule to coding and naming styles
3. Protect developers from possible errors and enforce best practices

You can find all error codes and plugins [in the docs](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/violations/index.html).

## What we are not

We are *not* planning to do the following things:

0. Assume or check types, use `mypy` together with our linter
1. [Reformat code](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/integrations/auto-formatters.html), since we believe that developers should do that
2. Check for `SyntaxError` or logical bugs, write tests instead
3. Appeal to everyone. But, you can [switch off](https://wemake-python-styleguide.rtfd.io/en/latest/pages/usage/setup.html#ignoring-violations) any rules that you don't like

## Supporting us :tada:

We in [wemake.services](https://wemake.services) make
all our tools open-source by default, so the community can benefit from them.
If you use our tools and they make your life easier and brings business value,
you can return us a favor by supporting the work we do.

[![Gold Tier](https://opencollective.com/wemake-python-styleguide/tiers/gold-sponsor.svg?width=890)](https://opencollective.com/wemake-python-styleguide)

[![Silver Tier](https://opencollective.com/wemake-python-styleguide/tiers/silver-sponsor.svg?width=890&avatarHeight=45&button=0)](https://opencollective.com/wemake-python-styleguide)

[![Bronze Tier](https://opencollective.com/wemake-python-styleguide/tiers/bronze-sponsor.svg?width=890&avatarHeight=35&button=0)](https://opencollective.com/wemake-python-styleguide)

## Show your style :sunglasses:

If you use our linter - it means that your code is awesome.
You can be proud of it!
And you should share your accomplishment with others
by including a badge in your `README` file. It looks like this:

[![wemake-python-styleguide](https://img.shields.io/badge/style-wemake-000000.svg)](https://github.com/wemake-services/wemake-python-styleguide)

### Markdown

```md
[![wemake-python-styleguide](https://img.shields.io/badge/style-wemake-000000.svg)](https://github.com/wemake-services/wemake-python-styleguide)
```

### Restructured text

```rst
.. image:: https://img.shields.io/badge/style-wemake-000000.svg
   :target: https://github.com/wemake-services/wemake-python-styleguide
```

## Contributing

We **warmly welcome** all contributions!

[![List of contributors](https://opencollective.com/wemake-python-styleguide/contributors.svg?width=890&button=0)](https://github.com/wemake-services/wemake-python-styleguide/graphs/contributors)

See ["Contributing"](https://wemake-python-styleguide.rtfd.io/en/latest/pages/api/index.html#contributing) section in the documentation if you want to contribute.

You can start with [issues that need some help](https://github.com/wemake-services/wemake-python-styleguide/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
right now.


#######
aiogram
#######

.. image:: https://img.shields.io/pypi/l/aiogram.svg?style=flat-square
    :target: https://opensource.org/licenses/MIT
    :alt: MIT License

.. image:: https://img.shields.io/pypi/status/aiogram.svg?style=flat-square
    :target: https://pypi.python.org/pypi/aiogram
    :alt: PyPi status

.. image:: https://img.shields.io/pypi/v/aiogram.svg?style=flat-square
    :target: https://pypi.python.org/pypi/aiogram
    :alt: PyPi Package Version

.. image:: https://img.shields.io/pypi/dm/aiogram.svg?style=flat-square
    :target: https://pypi.python.org/pypi/aiogram
    :alt: Downloads

.. image:: https://img.shields.io/pypi/pyversions/aiogram.svg?style=flat-square
    :target: https://pypi.python.org/pypi/aiogram
    :alt: Supported python versions

.. image:: https://img.shields.io/badge/dynamic/json?color=blue&logo=telegram&label=Telegram%20Bot%20API&query=%24.api.version&url=https%3A%2F%2Fraw.githubusercontent.com%2Faiogram%2Faiogram%2Fdev-3.x%2F.butcher%2Fschema%2Fschema.json&style=flat-square
    :target: https://core.telegram.org/bots/api
    :alt: Telegram Bot API

.. image:: https://img.shields.io/github/actions/workflow/status/aiogram/aiogram/tests.yml?branch=dev-3.x&style=flat-square
    :target: https://github.com/aiogram/aiogram/actions
    :alt: Tests

.. image:: https://img.shields.io/codecov/c/github/aiogram/aiogram?style=flat-square
    :target: https://app.codecov.io/gh/aiogram/aiogram
    :alt: Codecov

**aiogram** is a modern and fully asynchronous framework for
`Telegram Bot API <https://core.telegram.org/bots/api>`_ written in Python 3.8+ using
`asyncio <https://docs.python.org/3/library/asyncio.html>`_ and
`aiohttp <https://github.com/aio-libs/aiohttp>`_.

Make your bots faster and more powerful!

Documentation:
 - 🇺🇸 `English <https://docs.aiogram.dev/en/dev-3.x/>`_
 - 🇺🇦 `Ukrainian <https://docs.aiogram.dev/uk_UA/dev-3.x/>`_

Features
========

- Asynchronous (`asyncio docs <https://docs.python.org/3/library/asyncio.html>`_, :pep:`492`)
- Has type hints (:pep:`484`) and can be used with `mypy <http://mypy-lang.org/>`_
- Supports `PyPy <https://www.pypy.org/>`_
- Supports `Telegram Bot API 7.5 <https://core.telegram.org/bots/api>`_ and gets fast updates to the latest versions of the Bot API
- Telegram Bot API integration code was `autogenerated <https://github.com/aiogram/tg-codegen>`_ and can be easily re-generated when API gets updated
- Updates router (Blueprints)
- Has Finite State Machine
- Uses powerful `magic filters <https://docs.aiogram.dev/en/latest/dispatcher/filters/magic_filters.html#magic-filters>`_
- Middlewares (incoming updates and API calls)
- Provides `Replies into Webhook <https://core.telegram.org/bots/faq#how-can-i-make-requests-in-response-to-updates>`_
- Integrated I18n/L10n support with GNU Gettext (or Fluent)

.. warning::

    It is strongly advised that you have prior experience working
    with `asyncio <https://docs.python.org/3/library/asyncio.html>`_
    before beginning to use **aiogram**.

    If you have any questions, you can visit our community chats on Telegram:

    - 🇺🇸 `@aiogram <https://t.me/aiogram>`_
    - 🇺🇦 `@aiogramua <https://t.me/aiogramua>`_
    - 🇺🇿 `@aiogram_uz <https://t.me/aiogram_uz>`_
    - 🇰🇿 `@aiogram_kz <https://t.me/aiogram_kz>`_
    - 🇷🇺 `@aiogram_ru <https://t.me/aiogram_ru>`_
    - 🇮🇷 `@aiogram_fa <https://t.me/aiogram_fa>`_
    - 🇮🇹 `@aiogram_it <https://t.me/aiogram_it>`_
    - 🇧🇷 `@aiogram_br <https://t.me/aiogram_br>`_