![CI Tests](https://github.com/faucetsdn/daq/workflows/DAQ%20test%20suite/badge.svg?branch=master)

# DAQ: Device Automated Qualification for IoT Devices.

**DEPRECATED DEPRECATED DEPRECATED DEPRECATED DEPRECATED**

DAQ is a framework designed to test and operate IoT devices in an enterprise IoT environment.
Nominally about device testing and qualification, Device Automated Qualification (DAQ), provides
a means to _automate_ many capabilities, resulting in a more manageable, robust, and
secure platform.

Join the [DEPRECATED daq-users@googlegroups.com](https://groups.google.com/forum/#!forum/daq-users) email
list for ongoing discussion about using DAQ for enterprise IoT devices.--

There are several main categories of capabilities that DAQ addresses:
* [_Device automated qualification and testing_](docs/qualification.md):
Testing the behavior of a device against established security and network standards.
More details about the goals and objectives behind this can be found in the IEEE Computer article
[Taming the IoT: Operationalized Testing to Secure Connected Devices](https://www.computer.org/csdl/mags/co/2018/06/mco2018060090-abs.html).
* [_Network security orchestration_](docs/orchestration.md): Use standard
[SDN capabilities](https://queue.acm.org/detail.cfm?id=2560327), such as the
[FAUCET OpenFlow controller](https://faucet.nz/), to orchestrate "microsegmentation" on
the network for improved security.
* _Device Management Tools_: A suite of tools, consoles, and dashboards that help operate
a robust ecosystem of IoT devices. (Details forthcoming.)

## System Requirements

Most aspects of DAQ assume a baseline setup consisting of:
* Linux install: DAQ has been tested against both `Ubuntu 18.04.2 LTS bionic` and
`Debian GNU/Linux 9.4 stretch`, YMMV with other platforms.
* Dedicated network adapters: At the very minimum one dedicated ethernet adapter is
required. This could either be a separate built-in NIC, or a USB-Ethernet dongle.
* (Optional) OpenFlow-compatible hardware switch, described in the
[Network Topologies](docs/topologies.md) overview.

## Quick Start
Please reference [here](docs/quickstart.md)

## Folder Structure

The top-level DAQ folders correspond to the following structure:
* `bin/`: System setup and management commands.
* _`build/`_: Dynamically created directory for build logs.
* `cmd/`: Primary commands for running DAQ testing.
* `config/`: System and module configuration files.
* `daq/`: Python source for DAQ runtime.
* `docker/`: Docker build files for DAQ components and tests.
* `docs/`: Documentation.
* `etc/`: Version files and external switch setup files.
* _`faucet/`_: Dynamically downloaded version of SDN controller.
* _`forch/`_: Dynamically downloaded version of SDN orchestrator.
* `firebase/`: Hosted pages and functions for web dashboard.
* `functions/`: Additional Cloud Functions for data processing.
* _`inst/`_: Install directory for specific runtime contents.
* **`local/`**: Local setup and config information.
* _`mininet/`_: Local version of the mininet virtual network host framework.
* `mudacl/`: Utilities for managing and testing MUD network files.
* `mud_files/`: Examples and prototype device MUD files.
* _`out/`_: Misc transitory output files.
* `proto/`: Protobuf related files.
* _`protoc-gen-doc/`_: Dynamically downloaded Protobuf documentation generator.
* `pubber/`: Sample code for generating cloud-ingest traffic.
* `resources/`: Templates for reports, module configs, test-site definitions.
* `schemas/`: Device/cloud data exchange schemas.
* `subset/`: Subsets of device tests (e.g. penetration tests).
* `testing/`: Scripts for system continuous integration testing.
* `topology/`: Network topology setups.
* `validator/`: Tools for validating data exchange schemas.
* _`venv/`_: Dynamically downloaded python virtual environment files.

Items in _italics_ can generally be deleted without any loss of functionality (dynamically
created at install/runtime). The **local** subdirectory contains local setup information that
is not part of the source distribution.


[![Build Status](https://github.com/RTXteam/rtx/actions/workflows/pytest.yml/badge.svg)](https://github.com/RTXteam/RTX/actions/workflows/pytest.yml)
[![DOI](https://zenodo.org/badge/111240202.svg)](https://zenodo.org/badge/latestdoi/111240202)

# Table of contents

- [About the Translator project, Team Expander Agent, and ARAX](#about-the-translator-project-team-expander-agent-and-arax)
- [ARAX analyzes knowledge graphs to answer biomedical questions](#arax-analyzes-knowledge-graphs-to-answer-biomedical-questions)
- [How does ARAX work?](#how-does-arax-work)
- [The Reasoners Standard Application Programming Interface](#the-reasoners-standard-application-programming-interface)
- [What knowledge providers does ARAX use?](#what-knowledge-providers-does-arax-use)
  - [RTX-KG2](#rtx-kg2)
  - [Columbia Open Health Data (COHD)](#columbia-open-health-data-cohd)
  - [PubMed](#pubmed)
  - [Identifier mapping](#identifier-mapping)
- [Team Expander Agent: who we are](#team-expander-agent-who-we-are)
  - [Principal investigators](#principal-investigators)
  - [Team members](#team-members)
- [What is RTX? How does it differ from ARAX?](#what-is-rtx-how-does-it-differ-from-arax)
- [Organization of the ARAX/RTX software repository](#organization-of-the-araxrtx-software-repository)
  - [subdirectory `code`](#subdirectory-code)
    - [subdirectory `code/ARAX`](#subdirectory-codearax)
    - [subdirectory `code/ARAX/Examples`](#subdirectory-codearaxexamples)
    - [subdirectory `code/UI/OpenAPI`](#subdirectory-codeuiopenapi)
    - [subdirectory `code/UI/Feedback`](#subdirectory-codeuifeedback)
    - [subdirectory `code/UI/interactive`](#subdirectory-codeuiinteractive)
    - [subdirectory `code/kg2`](#subdirectory-codekg2)
    - [subdirectory `code/kg2/mediKanren`](#subdirectory-codekg2medikanren)
    - [subdirectory `code/reasoningtool/kg-construction`](#subdirectory-codereasoningtoolkg-construction)
    - [subdirectory `code/reasoningtool/SemMedDB`](#subdirectory-codereasoningtoolsemmeddb)
    - [subdirectory `code/reasoningtool/QuestionAnswering`](#subdirectory-codereasoningtoolquestionanswering)
    - [subdirectory `code/reasoningtool/MLDrugRepurposing`](#subdirectory-codereasoningtoolmldrugrepurposing)
    - [subdirectory `code/autocomplete`](#subdirectory-codeautocomplete)
  - [subdirectory `data`](#subdirectory-data)
  - [Key repository branches](#key-repository-branches)
- [License](#license)
- [Disclaimer](#disclaimer)
- [Credits](#credits)
- [Installation and dependencies](#installation-and-dependencies)
- [Contact us](#contact-us)
- [Try out ARAX/RTX...](#try-out-araxrtx)
  - [...in your web browser](#in-your-web-browser)
  - [...using our web API](#using-our-web-api)
  - [...by customizing a Jupyter Notebook](#by-customizing-a-jupyter-notebook)
- [Links](#links)
  - [General links for the Biomedical Data Translator project](#general-links-for-the-biomedical-data-translator-project)
  - [ARAX- and RTX-specific links](#arax--and-rtx-specific-links)

# About the Translator project, Team Expander Agent, and ARAX

We are **Expander Agent**, a team of researchers and software experts
working within a consortium effort called the **Biomedical Data Translator**
program ("Translator program"). Initiated by the NIH National Center for
Advancing Translational Sciences (NCATS), the Translator program's goal is to
accelerate the development of disease therapies by harnessing artificial
intelligence, Web-based distributed computing, and computationally-assisted
biomedical knowledge exploration. Although the Translator program is only in its
third year, Translator tools are already being used by biomedical researchers
for hypothesis generation and by analysts who are supporting clinical management
of rare disease cases. Key intended applications of Translator include
repositioning already-approved drugs for new indications (thus accelerating
time-to-market), identifying molecular targets for developing new therapeutic
agents, and providing software infrastructure that could enable development of
more powerful clinical decision support tools. The 18 teams on the translator
project are working with NCATS to achieve this goal by building the
**Translator** software, a modular system of Web services for biomedical
knowledge exploration, reasoning, and hypothesis generation.

After a multiyear feasibility assessment (2017-2019), during which our team built a
prototype Translator reasoning tool called **RTX,** the Translator program is
moving into a prototype development phase during early 2020. In this new phase,
our team is building a modular web-based software system, **ARAX**, that enables
expressive, automatable, and reproducible exploration and analysis of biomedical
knowledge graphs without requiring computer programming expertise. As we develop
ARAX, we will provide up-to-date software code for ARAX and RTX to the
scientific community via this software repository. While ARAX is currently under
development, our team will demonstrate ARAX for NCATS in mid-March 2020; at that
time we will make ARAX publicly available on the Web and we will provide
demonstration web pages and notebooks that illustrate how to access and operate
it.

# ARAX analyzes knowledge graphs to answer biomedical questions

ARAX is a tool for querying, manipulating, filtering, and exploring biomedical
knowledge graphs. It is designed to be a type of middleware—an *autonomous
relay agent*—within the Translator system. The top-level layer of
Translator (which is called the *autonomous relay system*) will issue structured
queries to ARAX via ARAX's web application programming interface. Then, based on the
query type, ARAX will determine which *knowledge providers* it needs to consult in
order to be able to answer the query; ARAX will then query the required
knowledge providers, synthesize the information that it gets from those queries,
and respond to the top-level layer in a standardized structured data
format. When completed, ARAX will contribute to and advance the Translator
program in four key ways:

1. ARAX provides a powerful **domain-specific language (DSL)**, called **ARAXi**
(technical documentation on ARAXi can be found
[here](code/ARAX/Documentation/DSL_Documentation.md)), that is designed
to enable researchers and clinicians to formulate, reuse, comprehend, and share
workflows for biomedical knowledge exploration. One of the key advantages of
ARAXi is that it is *not* a general-purpose programming language; it is
purpose-built for the task of describing—in user friendly syntax—a
knowledge graph manipulation workflow in terms of ARAX's modular
capabilities. All of ARAX's capabilities are exposed through ARAXi. Using ARAXi,
an analyst can:
  - define a small *query graph*; query for entities (e.g., proteins, pathways, or
diseases) that match the search criteria represented in the query graph
  - *expand* a knowledge graph, pulling in concepts that are related to
  concepts that are already in the knowledge graph
  - *filter* a knowledge graph, eliminating concepts or relationships that do
  not match a given set of search criteria
  - *overlay* contextual information from large datasets (such as co-occurrence of
terms in clinical health records or in abstracts of articles in the biomedical
literature)
  - *resultify*: enumerate and return matches of a query graph against a larger
  knowledge graph; as a sub-case of this step, ARAX can return as a single
  result, all concepts from the knowledge graph that match a given concept type
  and that match a given pattern of neighbor-concept-type relationships.

2. ARAX is based on a *modular architecture*. It provides distinct, orthogonal,
and human-understandable knowledge graph manipulation and analysis capabilities
via five operations (`query graph`, `expand`, `overlay`, `filter`, and
`resultify`) that can be accessed individually or in combination by the
Translator top-level layer, other Translator tools, or by individual researchers
directly using ARAX. Due to this transparent and modular design, the five ARAX
operations are easy-to-use in isolation and easy to compose into
workflows. Unifying these modules within a single service framework (ARAX) also
provides significant speed benefits for workflows that are implemented
end-to-end within ARAX, because the knowledge graph is stored on the server and
does not need to round-trip to the client with each operation. Through example
ARAX-powered analysis vignettes linked below, we describe how an ARAX workflow
using ARAXi can be much more powerful than the sum of its individual
parts.

3. ARAX is a Web service that speaks the information standard—called the
**Reasoners Standard Application Programming Interface**—that Translator
has adopted for data interchange between Translator components. Team
Expander Agent has been at the forefront of the development and stewardship
of the Reasoners Standard API (as described
[below](#the-reasoners-standard-application-programming-interface)), and with
this perspective, ARAX was built from the ground up to seamlessly interoperate
with other Translator software components.  In addition to complying with the
Translator standard for inter-reasoner communication, ARAX uses knowledge
sources (see [below](#what-knowledge-providers-does-arax-use)) that comply with
the *Biomedical Data Translator Knowledge Graph Standard* (for which our team
has been an active participant in the standards development process) which is
based on the [Biolink Model](https://biolink.github.io/biolink-model).

4. ARAX is integrated with the _RTX_ reasoning tool's knowledge graphs and graph
visualization capabilities. During the NCATS Translator feasibility assessment
phase, our team built a prototype reasoning tool system called RTX, whose
knowledge graphs (both the first-generation knowledge graph **RTX-KG1** and the
second-generation knowledge graph **RTX-KG2**) and user interface capabilities
are now available through ARAX. This integration enables a user of ARAX to
seamlessly refer a server-side knowledge graph or result-set for RTX-based
graphical visualization within a web browser. It also provides ARAX with
significant speed efficiencies for graph expansion and identifier mapping.

# How does ARAX work?

When the ARAX server is queried by the Autonomous Relay System or by another
application, four things happen in sequence:

1. From the query data structure that is provided to ARAX in accordance with the
Reasoners Standard API, ARAX extracts a series of ARAXi commands, a query graph,
or a natural-language question that has been interpreted by ARAX to be of a
specific question type.

2. ARAX chooses—based on the ARAXi commands, the query graph, or the
interpreted question—which upstream *knowledge providers* to query in
order to obtain the information required to answer the question

3. ARAX integrates and processes the information returned from the knowledge
providers, in accordance with the query type.

4. ARAX responds to the question with an answer that complies with the Reasoners
   Standard API. ARAX's responses to questions or queries typically contain
   three parts:

   - a recapitulation of the original query (which may involve a restatement of
the question or a structured representation of a small *query graph* of
biomedical concepts)
   - a list of *results*, each of which may be a concept (e.g., "imatinib" or
     "BCR-Abl tyrosine kinase") or a small graph of concepts and relationships
     between the concepts
   - a *knowledge graph* representing the union of concepts in all of the results,
   along with all known relationships among the concepts in the union.

# The Reasoners Standard Application Programming Interface

During the Feasibility Assessment Phase of the Biomedical Data Translator
program, a data interchange standard (the Reasoners Standard Application
Programming Interface) for communication to/from Translator reasoning tools was
developed and ratified by the Translator stakeholders. The Reasoners Standard
API is formally defined by a Yet Another Markup Language (YAML)
[file](https://github.com/NCATS-Tangerine/NCATS-ReasonerStdAPI/blob/master/API/TranslatorReasonersAPI.yaml)
that is in OpenAPI 3.0.1 format and that is versioned on GitHub. The current
version of the Reasoners Standard API is 0.9.1. The GitHub
[repository](https://github.com/NCATS-Tangerine/NCATS-ReasonerStdAPI) for the
Reasoners Standard API contains an issue list (the primary forum for documented
and archived discussion of issues and proposed changes with the standard),
automated regression tests, some presentation slide decks that provide information
about the standard, and (of course) the YAML file that defines the standard.
Team Expander Agent (and under our previous name during the Feasibility
Assessment phase, Team X-ray) is an active participant in the ongoing
development process for the Reasoners Standard API.

# What knowledge providers does ARAX use?

Currently, ARAX/RTX directly accesses four main knowledge providers in order to
handle queries, along with several additional APIs for identifier mapping.

## RTX-KG2

RTX-KG2 (GitHub project area is [RTXteam/RTX-KG2](https://github.com/RTXteam/RTX-KG2))
is a knowledge graph comprising 7.5M nodes and 34.3M relationships
that is built by integrating concepts and concept-predicate-concept triples
obtained from:

1. Unified Medical Language System (UMLS; including SNOMED CT)
2. NCBI Genes
3. Ensembl Genes
4. UniChem
5. Semantic Medline Database (SemMedDB)

RTX-KG2 complies with the Biomedical Data Translator Knowledge Graph object
model standard, which is based on the Biolink model. RTX-KG2 is hosted in a
Neo4j graph database server and can be accessed at
[kg2endpoint2.rtx.ai:7474](http://kg2endpoint2.rtx.ai:7474) (username is
`neo4j`; contact Team Expander Agent for the password). Alternatively, a
JSON dump of KG2 is available from the
[RTX-KG2 S3 bucket](http://rtx-kg2-public.s3-website-us-west-2.amazonaws.com/).
A version of KG2 that is formatted and indexed for the knowledge graph
exploration tool [mediKanren](https://github.com/webyrd/mediKanren) is
available; contact Team Expander Agent for details. For extensive technical
documentation on RTX-KG2, see this repository's [KG2 subdirectory](code/kg2).

## Columbia Open Health Data (COHD)

ARAX accesses the Columbia Open Health Data (COHD) resource (provided by the
Red Team and the Tatonetti Lab from the NCATS Translator Feasibility
Assessment Phase) for overlaying clinical health record co-occurrence
significance information for biomedical concepts in a knowledge graph,
via the `overlay` feature (for more information, see `code/ARAX`). ARAX
accesses COHD via a web API.

## PubMed

ARAX accesses PubMed for overlaying biomedical literature abstract co-occurrence
significance information for biomedical concepts in a knowledge graph, via the
`overlay` feature (for more information, see `code/ARAX`). For overlaying
literature co-occurrence information, ARAX uses a pre-indexed version of PubMed
(indexed for Medical Subject Heading or MeSH terms). For any concepts that
cannot be mapped to MeSH, ARAX queries PubMed via a web API.

## Identifier mapping

RTX's reasoning code uses several different web services for on-the-fly mapping
between certain identifier types:
1. Ontology Lookup Service
2. MyChem.info
3. Disease Ontology
4. PubChem
5. NCBI eUtils
6. Human Metabolome Database

A computable file enumerating and summarizing the external APIs that are
used by ARAX/RTX, in YAML format, can be found
[here](code/ARAX/KnowledgeSources/API_LIST.yaml).

# Team Expander Agent: who we are

Our team includes investigators from Oregon State University, the Pennsylvania
State University, Institute for Systems Biology, and Radboud University in the
Netherlands.

## Principal investigators

| Name           | Role                                        | Email                             | GitHub username                               | Areas of relevant expertise      |
| -------------- | ------------------------------------------- | --------------------------------- | --------------------------------------------- | -------------------------------- |
| Stephen Ramsey | OSU                 | `stephen.ramsey@oregonstate.edu`  | [saramsey](https://github.com/saramsey)       | compbio, systems biology         |
| David Koslicki | PSU                   | `dmk333@psu.edu`                  | [dkoslicki](https://github.com/dkoslicki)     | compbio, graph algorithms        |
| Eric Deutsch   | ISB           | `eric.deutsch@systemsbiology.org` | [edeutsch](https://github.com/edeutsch)       | bioinformatics, data management, standards development  |

## Team members

| Name             | Affiliation                   | Email                                 | GitHub username                                     | Areas of relevant expertise |
| ---------------- | ----------------------------- | ------------------------------------- | --------------------------------------------------- | --------------------------- |
| Jared Roach      | ISB | `jared.roach@systemsbiology.org`      |                                                     | genomics, genetics, medicine, systems biology |
| Luis Mendoza     | ISB | `luis.mendoza@systemsbiology.org`     | [isbluis](https://github.com/isbluis)               | software engineering, proteomics, systems biology |
| Finn Womack           | OSU       | `womackf@oregonstate.edu`        | [finnagin](https://github.com/finnagin)   | drug repositioning, Neo4j  |
| Amy Glen           | OSU       | `glena@oregonstate.edu`        | [amykglen](https://github.com/amykglen)   | knowledge graphs  |
| Arun Muluka          | PSU       | `avm6604@psu.edu`        | [aruntejam1](https://github.com/aruntejam1)   | knowledge graphs  |
| Chunyu Ma        | PSU     | `cqm5886@psu.edu` | [chunyuma](https://github.com/chunyuma) | programmer/analyst |

For our work on the Translator program, we also extensively
collaborate and cooperate with investigators at Oregon Health & Science
University, Lawrence Berkeley National Laboratory, University of North Carolina
Chapel Hill, and the University of Alabama Birmingham.

# What is RTX? How does it differ from ARAX?

During the Translator program's feasibility assessment phase (2017-2019), our
team—under the name "X-ray" that was assigned in accordance with the
feasibility assessment's team-naming scheme based on the electromagnetic
spectrum—built and released a prototype biomedical reasoning tool called
RTX, which is why this software repository is called `RTX`. RTX's capabilities
center around answering questions from a list of natural-language question
templates (e.g., *what proteins does acetaminophen target?* or *what drugs
target proteins associated with the glycolysis pathway?*) and around graphical
construction of a "query graph" that is used as a template for finding subgraphs
of the RTX-KG1 knowledge graph. The design for the ARAX software system
relates to RTX in four ways:
1. ARAX builds on the code-base for RTX and leverages the
already-built user interface and knowledge graphs for RTX (RTX-KG1 and RTX-KG2).
2. Through the expressive (but user-friendly) domain-specific language ARAXi, ARAX exposes RTX's graph
exploration and analysis capabilities so that they can be used (in combination
or individually) by Translator tools or workflows in accordance with the
Translator standard for inter-tool communication (Reasoners Standard API).
3. ARAX adds new and powerful graph exploration and analysis capabilities, such
as `expand`, `overlay` and `filter`, that make ARAX significantly more flexible
(in terms of the types of graph exploration/analysis workflows that it can
implement) than RTX.
4. RTX could produce results in the Reasoners Standard API format. However, its
more extensive reasoning capabilities could not be queried in this API format without
specialized knowledge of the RTX system. In contrast, ARAX can now perform its complex
reasoning capabilities upon receiving any input Reasoners Standard API, while still
producing such a standardized output format. As such, ARAX and its reasoning capabilities
will be accessible to any automated reasoning agent, automated reasoning system, or
knowledge provider capable of sending a Reasoners Standard API message.

# Organization of the ARAX/RTX software repository

ARAX and RTX are mostly written in the Python programming language and a small
amount of JavaScript and bash shell. Yet Another Markup Language (YAML) and
JavaScript Object Notation (JSON) are extensively used for configuration
files. Many examples of analysis workflow code that access RTX and/or ARAX are
provided in Jupyter notebook format, in several places in the code-base.

## subdirectory `code`

All software code files for ARAX and RTX are stored under this directory [link](code).

### subdirectory `code/ARAX`

Contains the core software code for ARAX [link](code/ARAX).

### subdirectory `code/ARAX/Examples`

Contains example Jupyter notebooks for using ARAX from
software [link](code/ARAX/Examples)

### subdirectory `code/UI/OpenAPI`

Contains (1) the YAML code that defines the Reasoners Standard
API and (2) the code for the Reasoners Standard API python object model that is
used to describe a knowledge graph, query nodes, and results
[(link)](code/UI/OpenAPI/python-flask-server).

### subdirectory `code/UI/Feedback`

Contains the code for the server-side logging system for the
RTX web browser-based user interface [(link)](code/UI/Feedback).

### subdirectory `code/UI/interactive`

Contains the code for the RTX web browser-based user interface
[(link)](code/UI/interactive).

### subdirectory `code/kg2`

Contains the code for building the RTX second-generation
knowledge graph (RTX-KG2) and hosting it in Neo4j [(link)](code/kg2).

### subdirectory `code/kg2/mediKanren`

Contains the code for exporting a version of the RTX-KG2
knowledge graph that is formatted and indexed for use with the [mediKanren](https://github.com/webyrd/mediKanren) knowledge graph exploration tool [(link)](code/kg2/mediKanren).

### subdirectory `code/reasoningtool/kg-construction`

Contains the code for building the RTX first-generation
knowledge graph (RTX-KG1) [(link)](code/reasoningtool/kg-construction).

### subdirectory `code/reasoningtool/SemMedDB`

Contains the code for a python interface to an instance of the
Semantic Medline Database (SemMedDB) that is being hosted in a MySQL database
[(link)](code/reasoningtool/SemMedDB).

### subdirectory `code/reasoningtool/QuestionAnswering`

Contains the code for parsing and answering questions
posed to the RTX reasoning tool [(link)](code/reasoningtool/QuestionAnswering).

### subdirectory `code/reasoningtool/MLDrugRepurposing`

Contains the code that is used for the machine-learning model for drug
repositioning that was described in the article *Leveraging distributed
biomedical knowledge sources to discover novel uses for known drugs*
[article](https://doi.org/10.1101/765305) by Finn Womack, Jason McClelland, and
David Koslicki [(link)](code/reasoningtool/MLDrugRepurposing).

### subdirectory `code/autocomplete`

Contains the code for the concept autocomplete feature in the
RTX web browser-based user interface [(link)](code/autocomplete).

## subdirectory `data`

Text data files for the RTX system that are deployed using git are stored under
this subdirectory. There are only a few such files because the RTX software
obtains most of the information that makes up the RTX-KG1 knowledge graph by
querying external knowledge providers via web APIs, rather than by loading flat
files [(link)](data).

## Key repository branches

The most up-to-date branch of the RTX repository (including the latest code for
the ARAX system) is `demo`. The `master` branch contains the most stable recent
release of RTX.

# License

ARAX and RTX are furnished under the MIT open-source software license; see the
`LICENSE` file for details. For the copyright on the code in the
`code/NLPCode` subdirectory, see the `LICENSE` file in that subdirectory.

# Disclaimer

Per the MIT [license](LICENSE), the ARAX and RTX software are provided
"as-is" without warranty of any kind. The content of this site and the RTX and
ARAX software is solely the responsibility of the authors and does not
necessarily represent the official views of the National Institutes of Health.

# Credits

Many people contributed to the development of ARAX and RTX. A list of code
contributors can be found under the
[contributors tab for this repository](https://github.com/RTXteam/RTX/graphs/contributors), in addition to the
current team members listed above. In addition to the code contributors, we
gratefully acknowledge technical assistance, contributions, and helpful feedback
from NCATS staff (Christine Colvis, Noel Southall, Mark Williams, Trung Nguyen,
Tyler Beck, Sarah Stemann, Debbi Adelakun, Dena Procaccini, and Tyler Peryea),
and Will Byrd, Greg Rosenblatt, Michael Patton, Chunlei Wu, Kevin Xin, Tom
Conlin, Harold Solbrig, Matt Brush, Karamarie Fecho, Julie McMurray, Kent
Shefchek, Chris Bizon, Steve Cox, Deepak Unni, Tim Putman, Patrick Wang, Sui
Huang, Theo Knijnenburg, Gustavo Glusman, John Earls, Andrew Su, Chris Mungall,
Marcin Joachimiak, Michel Dumontier, Richard Bruskiewich, and Melissa
Haendel. Support for the development of RTX was provided by NCATS through the
Translator program award OT2TR002520.  Support for the development of ARAX was
provided by NCATS through the Translator program award OT2TR003428.

# Installation and dependencies

ARAX is designed to be installed on an Amazon Web Services Elastic Compute Cloud
(EC2) instance with the following minimum requirements (we use a `m5a.4xlarge` instance):

- 16 vCPUs
- 64 GiB of RAM
- 1,023 GiB of elastic block storage
- host OS Ubuntu v18.04.

The host OS has nginx v1.14.0 installed and configured
(see `notes/ARAX/rtx-host-os-nginx-config` for configuration details) for
SSL/TLS termination and proxying of HTTP traffic to `localhost:8080`. The SSL
site certificate was generated using Letsencrypt (certbot v0.27.0). ARAX
and all of its database dependencies run inside a Docker container
(Docker v19.03.5) that is configured to map TCP ports as
follows (host-port:container-port):

- 7473:7473
- 7474:7474
- 7687:7687
- 8080:80

(for the specific Docker run command, see
`notes/ARAX/arax-run-container-nodes.md`). Within the Docker container,
ARAX uses

- Ubuntu v16.04
- Apache v2.4.18
- python v3.7.3
- Neo4j v3.2.6 (see [`code/reasoningtool/kg-construction`](code/reasoningtool/kg-construction) on how to set up Neo4j for running ARAX/RTX)
- OpenJDK v1.8.0_131
- mysql v5.7.19-0ubuntu0.16.04.1

The python package requirements for ARAX are described in the top-level
`requirements.txt` file.  RTX makes extensive use of internal caching via
SQLite v3.11.0.

# Contact us

The best way to contact Team Expander Agent is by

- sending an email message to
**[`expander.agent@gmail.com`](mailto:expander.agent@gmail.com)**
- logging an [issue](../../issues/) in this GitHub repository
- (for members of the Biomedical Data Translator consortium) messaging us on the
[NCATS Translator Slack](https://ncatstranslator.slack.com).

See also the contact information for the Team Expander Agent PIs above.

# Try out ARAX/RTX...

## ...in your web browser

Here is the link to access the web browser interface to RTX:
[arax.rtx.ai](https://arax.rtx.ai)

## ...using our web API

Here is the link to documentation on the web API interface to RTX:
[arax.rtx.ai/api/arax/v1.2/ui/](https://arax.rtx.ai/api/arax/v1.2/ui/).

## ...by customizing a Jupyter Notebook

Three Jupyter notebooks that demonstrate how to programmatically use ARAX are provided
[here](code/ARAX/Examples).

# Links

## General links for the Biomedical Data Translator project

- [Biomedical Data Translator consortium-wide project information (ncats.nih.gov)](https://ncats.nih.gov/translator/about)
- [Biomedical Data Translator 2020 funding opportunity announcement (grants.nih.gov)](https://grants.nih.gov/grants/guide/notice-files/NOT-TR-19-028.html)
- [Biomedical Data Translator Feasibility Assessment Phase open-source software repository (github.com)](https://github.com/NCATS-Tangerine/)
- [Biomedical Data Translator open-source software repository (github.com)](https://github.com/NCATSTranslator)
- [Biomedical Data Translator Knowledge Graph object model standard (github.io)](https://biolink.github.io/biolink-model)
- [Biomedical Data Translator Reasoners Standard API (github.com)](https://github.com/NCATS-Tangerine/NCATS-ReasonerStdAPI)

## ARAX- and RTX-specific links

- [ARAX browser-based interface (arax.rtx.ai)](https://arax.rtx.ai)
- [Technical documentation on ARAXi, the ARAX domain-specific language (this repo)](code/ARAX/Documentation/DSL_Documentation.md)
- [Technical documentation on the ARAX web API interface (arax.rtx.ai)](https://arax.rtx.ai/api/rtx/v1/ui/)
- [Example notebooks on how to programmatically use ARAX (this repo)](code/ARAX/Examples)


# About #

Pycroft is the current user management system of the AG DSN student
network.  It is based on [Flask](https://flask.palletsprojects.com/) and expects a
[Postgres](https://www.postgresql.org/) database making use of the
[SQLAlchemy ORM](https://www.sqlalchemy.org/).

# Documentation #
The sphinx documentation can be found at https://agdsn.github.io/pycroft/.
It contains a [setup guide](https://agdsn.github.io/pycroft/guides/setup.html).


# usegalaxy.\* tools [![Build Status](https://build.galaxyproject.eu/buildStatus/icon?style=flat-square&subject=Update%20Trusted&job=usegalaxy-eu%2Fupdate-trusted-tools)](https://build.galaxyproject.eu/job/usegalaxy-eu/job/update-trusted-tools/) [![Build Status](https://build.galaxyproject.eu/buildStatus/icon?style=flat-square&subject=Install%20Tools&job=usegalaxy-eu%2Finstall-tools)](https://build.galaxyproject.eu/job/usegalaxy-eu/job/install-tools/)

Originally this repository was solely for the use of usegalaxy.eu, but we are looking at expanding that to the other usegalaxy.\* instances.
Some documentation may be outdated while we figure out the new policies and procedures.

Currently only UseGalaxy.eu is installing tools from this repository.

All installed tools are also listed on our [webpage](https://galaxyproject.eu/tools).

## Usage

With [ephemeris](https://ephemeris.readthedocs.io/en/latest/) installed, you can use our tool defintions easily:

```bash
make tools_iuc.yaml.lock.install GALAXY_SERVER_URL=https://... GALAXY_API_KEY=...
```

## Setup

- `yaml` files are manually curated
- `yaml.lock` files are automatically generated
- Only IUC tools are automatically updated with the latest version each week

## Requesting Tools in UseGalaxy.\*

Policies are not set in stone and we would be happy to seek feedback and discussions.
Currenty, we encourage everyone to submit tools via PRs to this repo. The tools are losely grouped into several categories based on the yaml files. Please make your changes in the appropriate file and avoid creating a new yaml file unless necessary.

However, we promise a high-quality service to
our users and we need to ensure sustainability of the installed tools. That means that tools are regularly updated, tested
and can be adjusted to new developments.
Therefore, we encourage everyone to follow the [IUC Guidelines](https://galaxy-iuc-standards.readthedocs.io/en/latest/index.html) for tool development and have automatic testing enabled via [CI](https://en.wikipedia.org/wiki/Continuous_integration).

We encourage you to submit your tool to one of the larger community repositories, like

 * [Galaxy Tools maintained by IUC](https://github.com/galaxyproject/tools-iuc)
 * [Björn Grüning repo](https://github.com/bgruening/galaxytools)
 * Peter Cock's repos:
   * [pico repo](https://github.com/peterjc/pico_galaxy)
 * [Galaxy Proteomics repo](https://github.com/galaxyproteomics/tools-galaxyp)
 * [EI repo](https://github.com/TGAC/earlham-galaxytools)

 These repositories have planemo CI testing enabled and have a larger communities in place that help you with maintaining the
 tools.

### Updating an Existing Tool

- Edit the .yaml.lock file to add the latest/specific changeset revision for the tool. You can use `python scripts/update-tool.py --owner <repo-owner> --name <repo-name> <file.yaml>` in order to do this if you just want to add the latest revision.
- Open a pull request

### Requesting a New Tool

- If you just want the latest version:
	- Edit the .yaml file to add name/owner/section
- If you want a specific version:
	- Edit the .yaml file to add name/owner/section
	- Run `make fix`
	- Edit the .yaml.lock to correct the version number.
- Open a pull request


# CICD, Delivery, Compliance and Security Automation for Gardener
[![REUSE status](https://api.reuse.software/badge/github.com/gardener/cc-utils)](https://api.reuse.software/info/github.com/gardener/cc-utils)

![tests](https://concourse.ci.gardener.cloud/api/v1/teams/cicd/pipelines/cc-utils-master/jobs/master-head-update-job/badge?title=tests)
![release](https://concourse.ci.gardener.cloud/api/v1/teams/cicd/pipelines/cc-utils-master/jobs/master-release_job_image-job/badge?title=build)

![libs](https://badge.fury.io/py/gardener-cicd-libs.svg)

[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)

## What is it

`cc-utils` is a collection of re-usable utils intended to be used in the
context of Continuous Integration and output qualification of components
relevant for the [gardener](https://github.com/gardener) project.

[End-User Documentation](https://gardener.github.io/cc-utils)

## How to contribute

Be sure to run tests, linter and codestyle checks:

- `.ci/lint`
- `.ci/test`

Run `.ci/install_git_hooks` to register recommended git hooks.

## How to use it

### Install using pip

`pip install gardener-cicd-libs` - install libraries (no CLI)

`pip install gardener-cicd-cli` - install CLI

### Consume from Container Image

A copy of cc-utils is contained in the default container image in which gardener
CI/CD jobs are run (`europe-docker.pkg.dev/gardener-project/releases/cicd/job-image`):

- `gardener-ci` is available from PATH

## Runtime environment requirements

### Python Runtime

`Python 3.11` or greater is required as a runtime (see requirements.txt for additional
runtime dependencies). Earlier Python versions.

As a general rule, contained sources are always qualified using the python3-version from
[alpine](https://endoflife.date/alpine)'s greatest release version.

In addition to the Python API, some functions are exposed via a command line interface
(`./cli.py`).

## Special Modules

* `cli/gardener_ci/*.py`: all defined functions are exposed via
gardener-ci <module_name> <function_name>


![Content logo](https://raw.githubusercontent.com/demisto/content/813c5e98d9ebace80d3f4e0ff88734cac6d5b5d5/xsoar_content_logo.png)

[![CircleCI](https://circleci.com/gh/demisto/content-docs.svg?style=svg)](https://circleci.com/gh/demisto/content-docs)
[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/demisto/content-docs.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/demisto/content-docs/context:javascript)
[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/demisto/content-docs.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/demisto/content-docs/context:python)

# Cortex XSOAR Content Developer Docs
This repo holds Cortex XSOAR Content Developer Docs. The Content Developer Docs provide information on how to develop and contribute content to the Cortex XSOAR platform. The actual content is hosted at the following repo: https://github.com/demisto/content.

This website is built using [Docusaurus 2](https://v2.docusaurus.io/), a modern static website generator.

You can access the generated website at: https://xsoar.pan.dev/

## Running the site locally
### Node Setup
We use node 14.x for running the project. It is recommended to use `nvm`. See: https://github.com/nvm-sh/nvm for install instructions. It is recommended to setup `auto use` as specified here: https://github.com/nvm-sh/nvm#calling-nvm-use-automatically-in-a-directory-with-a-nvmrc-file . The root dir of the project contains a proper `.nvmrc` file with the recommended version to use.

### Install and Start
Install all dependencies:
```
npm install
```
Start the development server on: http://localhost:3000
```
npm start
```
Start writing docs...

## Generating Reference Docs
Reference docs are generated from the [Cortex XSOAR Content repository](https://github.com/demisto/content). If you are working on general site docs which are not reference docs from the Content repo, you don't need to run this step to preview your docs.

To generate the docs we use [pipenv](https://github.com/pypa/pipenv). Make sure to install pipenv by running: `pip3 install pipenv`.

When working locally you can generate the reference docs by running:
```
npm run reference-docs
```
This task will checkout the content repository and generate the docs. The generated docs are ignored by `.gitignore` and shouldn't be checked in as they are generated during the build.

When generating the docs the `master` branch of the `content` repo will be used unless there is a matching branch with the same name as the current branch of `content-docs`, that branch will be used.

If you have the `content` repo checked out locally and you want to use it for generating the reference docs (for example when working on a Content Integration Doc), you can set the path to your `content` repo dir with the environment variable: `CONTENT_REPO_DIR`. For example:
```bash
 CONTENT_REPO_DIR=~/dev/demisto/content npm run reference-docs
```
Additional env vars that affect the generation of the docs:
* `FILE_REGEX`: specify a regex to select which docs to generate docs for. For example:
  ```bash
  FILE_REGEX=Proofpoint CONTENT_REPO_DIR=~/dev/demisto/content npm run reference-docs
  ```
* `MAX_FILES`: max files to generate in each category. Set this for faster generation when developing. Note also in CI builds on branches this is automatically set to speed up the preview site if there are only modifications related to docs. For example:
  ```bash
  MAX_FILES=10 CONTENT_REPO_DIR=~/dev/demisto/content npm run reference-docs
  ```
* `UPDATE_PACK_DOCS`: set this when you want to update [pack-docs.md](docs/documentation/pack-docs.md) with the supported tags. Make sure NOT to check-in the modified doc.
* `SKIP_DEPRECATED`: skip calculating and generating deprecated doc. Set this for faster file generation when developing. For example:
  ```bash
  SKIP_DEPRECATED=true MAX_FILES=10 CONTENT_REPO_DIR=~/dev/demisto/content npm run reference-docs
  ```

### Generation Code
Code used for generating content reference docs is written in Python and resides in the `content-repo` folder. To set up a development environment we use [pipenv](https://github.com/pypa/pipenv). Make sure to install pipenv by running: `pip3 install pipenv`. Setup a proper Python env by running:
```
pipenv install --dev
```
To run linting and unit tests you can use npm:
```
npm run test
```
Linting and unit tests are run on each commit using Github Actions. They are required to pass in-order to merge PRs.

## Build

```
$ npm run build
```

This command generates static content into the `build` directory and can be served using any static contents hosting service.

## Contributing
Contributions are welcome and appreciated. If you find a bug or have suggestions for improvements, feel free to open an [Issue](https://github.com/demisto/content-docs/issues) or (better yet) submit a [Pull Request](https://github.com/demisto/content-docs/pulls).

Before merging any PRs, we need all contributors to sign a contributor license agreement. By signing a contributor license agreement, we ensure that the community is free to use your contributions.

When you open a new pull request, a bot will evaluate whether you have signed the CLA. If required, the bot will comment on the pull request, including a link to accept the agreement. The CLA document is also available for review as a [PDF](https://github.com/demisto/content/blob/master/docs/cla.pdf).

If the `license/cla` status check remains on *Pending*, even though all contributors have accepted the CLA, you can recheck the CLA status by visiting the following link (replace **[PRID]** with the ID of your PR): https://cla-assistant.io/check/demisto/content-docs?pullRequest=[PRID] .


# m1n1: an experimentation playground for Apple Silicon

(And to some extent a Linux bootloader)

## Building

You need an `aarch64-linux-gnu-gcc` cross-compiler toolchain (or a native one, if running on ARM64).

```shell
$ git clone --recursive https://github.com/AsahiLinux/m1n1.git
$ cd m1n1
$ make
```

The output will be in build/m1n1.macho.

To build on a native arm64 machine, use `make ARCH=`.

To build verbosely, use `make V=1`.

Building on ARM64 macOS is supported with clang and LLVM; you need to use Homebrew to
install the required dependencies:

```shell
$ brew install llvm
```

After that, just type `make`.

### Building using the container setup

If you have a container runtime installed, like Podman or Docker, you can make use of the compose setup, which contains all build dependencies.

```shell
$ git clone --recursive https://github.com/AsahiLinux/m1n1.git
$ cd m1n1
$ podman-compose run m1n1 make
$ # or
$ docker-compose run m1n1 make
```

## Usage

Our [wiki](https://github.com/AsahiLinux/docs/wiki/m1n1%3AUser-Guide) has more information on how to
use m1n1.

To install on an OS container based on macOS <12.1, use `m1n1.macho`:

```shell
kmutil configure-boot -c m1n1.macho -v <path to your OS volume>
```

To install on an OS container based on macOS >=12.1, use `m1n1.bin`:

```shell
kmutil configure-boot -c m1n1.bin --raw --entry-point 2048 --lowest-virtual-address 0 -v <path to your OS volume>
```

## Payloads

m1n1 supports running payloads by simple concatenation:

```shell
$ cat build/m1n1.macho Image.gz build/dtb/apple-j274.dtb initramfs.cpio.gz > m1n1-payload.macho
$ cat build/m1n1.bin Image.gz build/dtb/apple-j274.dtb initramfs.cpio.gz > m1n1-payload.bin
```

Supported payload file formats:

* Kernel images (or compatible). Must be compressed or last payload.
* Devicetree blobs (FDT). May be uncompressed or compressed.
* Initramfs cpio images. Must be compressed.

Supported compression formats:

* gzip
* xz

## License

m1n1 is licensed under the MIT license, as included in the [LICENSE](LICENSE) file.

* Copyright The Asahi Linux Contributors

Please see the Git history for authorship information.

Portions of m1n1 are based on mini:

* Copyright (C) 2008-2010 Hector Martin "marcan" <marcan@marcan.st>
* Copyright (C) 2008-2010 Sven Peter <sven@svenpeter.dev>
* Copyright (C) 2008-2010 Andre Heider <a.heider@gmail.com>

m1n1 embeds libfdt, which is dual [BSD](3rdparty_licenses/LICENSE.BSD-2.libfdt) and
[GPL-2](3rdparty_licenses/LICENSE.GPL-2) licensed and copyright:

* Copyright (C) 2014 David Gibson <david@gibson.dropbear.id.au>
* Copyright (C) 2018 embedded brains GmbH
* Copyright (C) 2006-2012 David Gibson, IBM Corporation.
* Copyright (C) 2012 David Gibson, IBM Corporation.
* Copyright 2012 Kim Phillips, Freescale Semiconductor.
* Copyright (C) 2016 Free Electrons
* Copyright (C) 2016 NextThing Co.

The ADT code in mini is also based on libfdt and subject to the same license.

m1n1 embeds [minlzma](https://github.com/ionescu007/minlzma), which is
[MIT](3rdparty_licenses/LICENSE.minlzma) licensed and copyright:

* Copyright (c) 2020 Alex Ionescu

m1n1 embeds a slightly modified version of [tinf](https://github.com/jibsen/tinf), which is
[ZLIB](3rdparty_licenses/LICENSE.tinf) licensed and copyright:

* Copyright (c) 2003-2019 Joergen Ibsen

m1n1 embeds portions taken from
[arm-trusted-firmware](https://github.com/ARM-software/arm-trusted-firmware), which is
[BSD](3rdparty_licenses/LICENSE.BSD-3.arm) licensed and copyright:

* Copyright (c) 2013-2020, ARM Limited and Contributors. All rights reserved.

m1n1 embeds [Doug Lea's malloc](ftp://gee.cs.oswego.edu/pub/misc/malloc.c) (dlmalloc), which is in
the public domain ([CC0](3rdparty_licenses/LICENSE.CC0)).

m1n1 embeds portions of [PDCLib](https://github.com/DevSolar/pdclib), which is in the public
domain ([CC0](3rdparty_licenses/LICENSE.CC0)).

m1n1 embeds the [Source Code Pro](https://github.com/adobe-fonts/source-code-pro) font, which is
licensed under the [OFL-1.1](3rdparty_licenses/LICENSE.OFL-1.1) license and copyright:

* Copyright 2010-2019 Adobe (http://www.adobe.com/), with Reserved Font Name 'Source'. All Rights Reserved. Source is a trademark of Adobe in the United States and/or other countries.
* This Font Software is licensed under the SIL Open Font License, Version 1.1.

m1n1 embeds portions of the [dwc3 usb linux driver](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/usb/dwc3/core.h?id=7bc5a6ba369217e0137833f5955cf0b0f08b0712), which was [BSD-or-GPLv2 dual-licensed](3rdparty_licenses/LICENSE.BSD-3.dwc3) and copyright
* Copyright (C) 2010-2011 Texas Instruments Incorporated - http://www.ti.com

m1n1 embeds portions of [musl-libc](https://musl.libc.org/)'s floating point library, which are MIT licensed and copyright
* Copyright (c) 2017-2018, Arm Limited.

m1n1 embeds some rust crates. Licenses can be found in the vendor directory for every crate.


    FastAPI framework, high performance, easy to learn, fast to code, ready for production

---

**Documentation**: https://fastapi.tiangolo.com

**Source Code**: https://github.com/tiangolo/fastapi

---

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints.

The key features are:

* **Fast**: Very high performance, on par with **NodeJS** and **Go** (thanks to Starlette and Pydantic). [One of the fastest Python frameworks available](#performance).
* **Fast to code**: Increase the speed to develop features by about 200% to 300%. *
* **Fewer bugs**: Reduce about 40% of human (developer) induced errors. *
* **Intuitive**: Great editor support. Completion everywhere. Less time debugging.
* **Easy**: Designed to be easy to use and learn. Less time reading docs.
* **Short**: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
* **Robust**: Get production-ready code. With automatic interactive documentation.
* **Standards-based**: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.

* estimation based on tests on an internal development team, building production applications.

## Sponsors

Other sponsors

## Opinions

"_[...] I'm using **FastAPI** a ton these days. [...] I'm actually planning to use it for all of my team's **ML services at Microsoft**. Some of them are getting integrated into the core **Windows** product and some **Office** products._"

Kabir Khan - Microsoft (ref)

---

"_We adopted the **FastAPI** library to spawn a **REST** server that can be queried to obtain **predictions**. [for Ludwig]_"

Piero Molino, Yaroslav Dudin, and Sai Sumanth Miryala - Uber (ref)

---

"_**Netflix** is pleased to announce the open-source release of our **crisis management** orchestration framework: **Dispatch**! [built with **FastAPI**]_"

Kevin Glisson, Marc Vilanova, Forest Monsen - Netflix (ref)

---

"_I’m over the moon excited about **FastAPI**. It’s so fun!_"

Brian Okken - Python Bytes podcast host (ref)

---

"_Honestly, what you've built looks super solid and polished. In many ways, it's what I wanted **Hug** to be - it's really inspiring to see someone build that._"

Timothy Crosley - Hug creator (ref)

---

"_If you're looking to learn one **modern framework** for building REST APIs, check out **FastAPI** [...] It's fast, easy to use and easy to learn [...]_"

"_We've switched over to **FastAPI** for our **APIs** [...] I think you'll like it [...]_"

Ines Montani - Matthew Honnibal - Explosion AI founders - spaCy creators (ref) - (ref)

---

"_If anyone is looking to build a production Python API, I would highly recommend **FastAPI**. It is **beautifully designed**, **simple to use** and **highly scalable**, it has become a **key component** in our API first development strategy and is driving many automations and services such as our Virtual TAC Engineer._"

Deon Pillsbury - Cisco (ref)

---

## **Typer**, the FastAPI of CLIs

If you are building a CLI app to be used in the terminal instead of a web API, check out **Typer**.

**Typer** is FastAPI's little sibling. And it's intended to be the **FastAPI of CLIs**. ⌨ 🚀

## Requirements

FastAPI stands on the shoulders of giants:

* Starlette for the web parts.
* Pydantic for the data parts.

## Installation

```console
$ pip install fastapi

---> 100%
```

## Example

### Create it

* Create a file `main.py` with:

```Python
from typing import Union

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}
```

Or use async def...

If your code uses `async` / `await`, use `async def`:

```Python hl_lines="9  14"
from typing import Union

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def read_root():
    return {"Hello": "World"}

@app.get("/items/{item_id}")
async def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}
```

**Note**:

If you don't know, check the _"In a hurry?"_ section about `async` and `await` in the docs.

### Run it

Run the server with:

```console
$ fastapi dev main.py

 ╭────────── FastAPI CLI - Development mode ───────────╮
 │                                                     │
 │  Serving at: http://127.0.0.1:8000                  │
 │                                                     │
 │  API docs: http://127.0.0.1:8000/docs               │
 │                                                     │
 │  Running in development mode, for production use:   │
 │                                                     │
 │  fastapi run                                        │
 │                                                     │
 ╰─────────────────────────────────────────────────────╯

INFO:     Will watch for changes in these directories: ['/home/user/code/awesomeapp']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [2248755] using WatchFiles
INFO:     Started server process [2248757]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
```

About the command fastapi dev main.py...

The command `fastapi dev` reads your `main.py` file, detects the **FastAPI** app in it, and starts a server using Uvicorn.

By default, `fastapi dev` will start with auto-reload enabled for local development.

You can read more about it in the FastAPI CLI docs.

### Check it

Open your browser at http://127.0.0.1:8000/items/5?q=somequery.

You will see the JSON response as:

```JSON
{"item_id": 5, "q": "somequery"}
```

You already created an API that:

* Receives HTTP requests in the _paths_ `/` and `/items/{item_id}`.
* Both _paths_ take `GET` operations (also known as HTTP _methods_).
* The _path_ `/items/{item_id}` has a _path parameter_ `item_id` that should be an `int`.
* The _path_ `/items/{item_id}` has an optional `str` _query parameter_ `q`.

### Interactive API docs

Now go to http://127.0.0.1:8000/docs.

You will see the automatic interactive API documentation (provided by Swagger UI):

![Swagger UI](https://fastapi.tiangolo.com/img/index/index-01-swagger-ui-simple.png)

### Alternative API docs

And now, go to http://127.0.0.1:8000/redoc.

You will see the alternative automatic documentation (provided by ReDoc):

![ReDoc](https://fastapi.tiangolo.com/img/index/index-02-redoc-simple.png)

## Example upgrade

Now modify the file `main.py` to receive a body from a `PUT` request.

Declare the body using standard Python types, thanks to Pydantic.

```Python hl_lines="4  9-12  25-27"
from typing import Union

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Item(BaseModel):
    name: str
    price: float
    is_offer: Union[bool, None] = None

@app.get("/")
def read_root():
    return {"Hello": "World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

@app.put("/items/{item_id}")
def update_item(item_id: int, item: Item):
    return {"item_name": item.name, "item_id": item_id}
```

The `fastapi dev` server should reload automatically.

### Interactive API docs upgrade

Now go to http://127.0.0.1:8000/docs.

* The interactive API documentation will be automatically updated, including the new body:

![Swagger UI](https://fastapi.tiangolo.com/img/index/index-03-swagger-02.png)

* Click on the button "Try it out", it allows you to fill the parameters and directly interact with the API:

![Swagger UI interaction](https://fastapi.tiangolo.com/img/index/index-04-swagger-03.png)

* Then click on the "Execute" button, the user interface will communicate with your API, send the parameters, get the results and show them on the screen:

![Swagger UI interaction](https://fastapi.tiangolo.com/img/index/index-05-swagger-04.png)

### Alternative API docs upgrade

And now, go to http://127.0.0.1:8000/redoc.

* The alternative documentation will also reflect the new query parameter and body:

![ReDoc](https://fastapi.tiangolo.com/img/index/index-06-redoc-02.png)

### Recap

In summary, you declare **once** the types of parameters, body, etc. as function parameters.

You do that with standard modern Python types.

You don't have to learn a new syntax, the methods or classes of a specific library, etc.

Just standard **Python**.

For example, for an `int`:

```Python
item_id: int
```

or for a more complex `Item` model:

```Python
item: Item
```

...and with that single declaration you get:

* Editor support, including:
    * Completion.
    * Type checks.
* Validation of data:
    * Automatic and clear errors when the data is invalid.
    * Validation even for deeply nested JSON objects.
* Conversion of input data: coming from the network to Python data and types. Reading from:
    * JSON.
    * Path parameters.
    * Query parameters.
    * Cookies.
    * Headers.
    * Forms.
    * Files.
* Conversion of output data: converting from Python data and types to network data (as JSON):
    * Convert Python types (`str`, `int`, `float`, `bool`, `list`, etc).
    * `datetime` objects.
    * `UUID` objects.
    * Database models.
    * ...and many more.
* Automatic interactive API documentation, including 2 alternative user interfaces:
    * Swagger UI.
    * ReDoc.

---

Coming back to the previous code example, **FastAPI** will:

* Validate that there is an `item_id` in the path for `GET` and `PUT` requests.
* Validate that the `item_id` is of type `int` for `GET` and `PUT` requests.
    * If it is not, the client will see a useful, clear error.
* Check if there is an optional query parameter named `q` (as in `http://127.0.0.1:8000/items/foo?q=somequery`) for `GET` requests.
    * As the `q` parameter is declared with `= None`, it is optional.
    * Without the `None` it would be required (as is the body in the case with `PUT`).
* For `PUT` requests to `/items/{item_id}`, Read the body as JSON:
    * Check that it has a required attribute `name` that should be a `str`.
    * Check that it has a required attribute `price` that has to be a `float`.
    * Check that it has an optional attribute `is_offer`, that should be a `bool`, if present.
    * All this would also work for deeply nested JSON objects.
* Convert from and to JSON automatically.
* Document everything with OpenAPI, that can be used by:
    * Interactive documentation systems.
    * Automatic client code generation systems, for many languages.
* Provide 2 interactive documentation web interfaces directly.

---

We just scratched the surface, but you already get the idea of how it all works.

Try changing the line with:

```Python
    return {"item_name": item.name, "item_id": item_id}
```

...from:

```Python
        ... "item_name": item.name ...
```

...to:

```Python
        ... "item_price": item.price ...
```

...and see how your editor will auto-complete the attributes and know their types:

![editor support](https://fastapi.tiangolo.com/img/vscode-completion.png)

For a more complete example including more features, see the Tutorial - User Guide.

**Spoiler alert**: the tutorial - user guide includes:

* Declaration of **parameters** from other different places as: **headers**, **cookies**, **form fields** and **files**.
* How to set **validation constraints** as `maximum_length` or `regex`.
* A very powerful and easy to use **Dependency Injection** system.
* Security and authentication, including support for **OAuth2** with **JWT tokens** and **HTTP Basic** auth.
* More advanced (but equally easy) techniques for declaring **deeply nested JSON models** (thanks to Pydantic).
* **GraphQL** integration with Strawberry and other libraries.
* Many extra features (thanks to Starlette) as:
    * **WebSockets**
    * extremely easy tests based on HTTPX and `pytest`
    * **CORS**
    * **Cookie Sessions**
    * ...and more.

## Performance

Independent TechEmpower benchmarks show **FastAPI** applications running under Uvicorn as one of the fastest Python frameworks available, only below Starlette and Uvicorn themselves (used internally by FastAPI). (*)

To understand more about it, see the section Benchmarks.

## Dependencies

Used by Pydantic:

* email_validator - for email validation.
* pydantic-settings - for settings management.
* pydantic-extra-types - for extra types to be used with Pydantic.

Used by Starlette:

* httpx - Required if you want to use the `TestClient`.
* jinja2 - Required if you want to use the default template configuration.
* python-multipart - Required if you want to support form "parsing", with `request.form()`.

Used by FastAPI / Starlette:

* uvicorn - for the server that loads and serves your application.
* orjson - Required if you want to use `ORJSONResponse`.
* ujson - Required if you want to use `UJSONResponse`.
* `fastapi-cli` - to provide the `fastapi` command.

When you install `fastapi` it comes these standard dependencies.

## `fastapi-slim`

If you don't want the extra standard optional dependencies, install `fastapi-slim` instead.

When you install with:

```bash
pip install fastapi
```

...it includes the same code and dependencies as:

```bash
pip install "fastapi-slim[standard]"
```

The standard extra dependencies are the ones mentioned above.

## License

This project is licensed under the terms of the MIT license.


.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png
   :alt: Luigi Logo
   :align: center

.. image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fspotify%2Fluigi%2Fbadge&label=build&logo=none&%3Fref%3Dmaster&style=flat
    :target: https://actions-badge.atrox.dev/spotify/luigi/goto?ref=master

.. image:: https://img.shields.io/codecov/c/github/spotify/luigi/master.svg?style=flat
    :target: https://codecov.io/gh/spotify/luigi?branch=master

.. image:: https://img.shields.io/pypi/v/luigi.svg?style=flat
   :target: https://pypi.python.org/pypi/luigi

.. image:: https://img.shields.io/pypi/l/luigi.svg?style=flat
   :target: https://pypi.python.org/pypi/luigi

.. image:: https://readthedocs.org/projects/luigi/badge/?version=stable
    :target: https://luigi.readthedocs.io/en/stable/?badge=stable
    :alt: Documentation Status

Luigi is a Python (3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 tested) package that helps you build complex
pipelines of batch jobs. It handles dependency resolution, workflow management,
visualization, handling failures, command line integration, and much more.

Getting Started
---------------

Run ``pip install luigi`` to install the latest stable version from `PyPI
<https://pypi.python.org/pypi/luigi>`_. `Documentation for the latest release
<https://luigi.readthedocs.io/en/stable/>`__ is hosted on readthedocs.

Run ``pip install luigi[toml]`` to install Luigi with `TOML-based configs
<https://luigi.readthedocs.io/en/stable/configuration.html>`__ support.

For the bleeding edge code, ``pip install
git+https://github.com/spotify/luigi.git``. `Bleeding edge documentation
<https://luigi.readthedocs.io/en/latest/>`__ is also available.

Background
----------

The purpose of Luigi is to address all the plumbing typically associated
with long-running batch processes. You want to chain many tasks,
automate them, and failures *will* happen. These tasks can be anything,
but are typically long running things like
`Hadoop <http://hadoop.apache.org/>`_ jobs, dumping data to/from
databases, running machine learning algorithms, or anything else.

There are other software packages that focus on lower level aspects of
data processing, like `Hive <http://hive.apache.org/>`__,
`Pig <http://pig.apache.org/>`_, or
`Cascading <http://www.cascading.org/>`_. Luigi is not a framework to
replace these. Instead it helps you stitch many tasks together, where
each task can be a `Hive query <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
a `Hadoop job in Java <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop_jar.html>`_,
a  `Spark job in Scala or Python <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.spark.html>`_,
a Python snippet,
`dumping a table <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.sqla.html>`_
from a database, or anything else. It's easy to build up
long-running pipelines that comprise thousands of tasks and take days or
weeks to complete. Luigi takes care of a lot of the workflow management
so that you can focus on the tasks themselves and their dependencies.

You can build pretty much any task you want, but Luigi also comes with a
*toolbox* of several common task templates that you use. It includes
support for running
`Python mapreduce jobs <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop.html>`_
in Hadoop, as well as
`Hive <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
and `Pig <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`__,
jobs. It also comes with
`file system abstractions for HDFS <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hdfs.html>`_,
and local files that ensures all file system operations are atomic. This
is important because it means your data pipeline will not crash in a
state containing partial data.

Visualiser page
---------------

The Luigi server comes with a web interface too, so you can search and filter
among all your tasks.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/visualiser_front_page.png
   :alt: Visualiser page

Dependency graph example
------------------------

Just to give you an idea of what Luigi does, this is a screen shot from
something we are running in production. Using Luigi's visualiser, we get
a nice visual overview of the dependency graph of the workflow. Each
node represents a task which has to be run. Green tasks are already
completed whereas yellow tasks are yet to be run. Most of these tasks
are Hadoop jobs, but there are also some things that run locally and
build up data files.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/user_recs.png
   :alt: Dependency graph

Philosophy
----------

Conceptually, Luigi is similar to `GNU
Make <http://www.gnu.org/software/make/>`_ where you have certain tasks
and these tasks in turn may have dependencies on other tasks. There are
also some similarities to `Oozie <http://oozie.apache.org/>`_
and `Azkaban <https://azkaban.github.io/>`_. One major
difference is that Luigi is not just built specifically for Hadoop, and
it's easy to extend it with other kinds of tasks.

Everything in Luigi is in Python. Instead of XML configuration or
similar external data files, the dependency graph is specified *within
Python*. This makes it easy to build up complex dependency graphs of
tasks, where the dependencies can involve date algebra or recursive
references to other versions of the same task. However, the workflow can
trigger things not in Python, such as running
`Pig scripts <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`_
or `scp'ing files <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.ssh.html>`_.

Who uses Luigi?
---------------

We use Luigi internally at `Spotify <https://www.spotify.com>`_ to run
thousands of tasks every day, organized in complex dependency graphs.
Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
that powers all kinds of stuff including recommendations, toplists, A/B
test analysis, external reports, internal dashboards, etc.

Since Luigi is open source and without any registration walls, the exact number
of Luigi users is unknown. But based on the number of unique contributors, we
expect hundreds of enterprises to use it. Some users have written blog posts
or held presentations about Luigi:

* `Spotify <https://www.spotify.com>`_ `(presentation, 2014) <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`__
* `Foursquare <https://foursquare.com/>`_ `(presentation, 2013) <http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897>`__
* `Mortar Data (Datadog) <https://www.datadoghq.com/>`_ `(documentation / tutorial) <http://help.mortardata.com/technologies/luigi>`__
* `Stripe <https://stripe.com/>`_ `(presentation, 2014) <http://www.slideshare.net/PyData/python-as-part-of-a-production-machine-learning-stack-by-michael-manapat-pydata-sv-2014>`__
* `Buffer <https://buffer.com/>`_ `(blog, 2014) <https://overflow.bufferapp.com/2014/10/31/buffers-new-data-architecture/>`__
* `SeatGeek <https://seatgeek.com/>`_ `(blog, 2015) <http://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/>`__
* `Treasure Data <https://www.treasuredata.com/>`_ `(blog, 2015) <http://blog.treasuredata.com/blog/2015/02/25/managing-the-data-pipeline-with-git-luigi/>`__
* `Growth Intelligence <http://growthintel.com/>`_ `(presentation, 2015) <http://www.slideshare.net/growthintel/a-beginners-guide-to-building-data-pipelines-with-luigi>`__
* `AdRoll <https://www.adroll.com/>`_ `(blog, 2015) <http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html>`__
* 17zuoye `(presentation, 2015) <https://speakerdeck.com/mvj3/luiti-an-offline-task-management-framework>`__
* `Custobar <https://www.custobar.com/>`_ `(presentation, 2016) <http://www.slideshare.net/teemukurppa/managing-data-workflows-with-luigi>`__
* `Blendle <https://launch.blendle.com/>`_ `(presentation) <http://www.anneschuth.nl/wp-content/uploads/sea-anneschuth-streamingblendle.pdf#page=126>`__
* `TrustYou <http://www.trustyou.com/>`_ `(presentation, 2015) <https://speakerdeck.com/mfcabrera/pydata-berlin-2015-processing-hotel-reviews-with-python>`__
* `Groupon <https://www.groupon.com/>`_ / `OrderUp <https://orderup.com>`_ `(alternative implementation) <https://github.com/groupon/luigi-warehouse>`__
* `Red Hat - Marketing Operations <https://www.redhat.com>`_ `(blog, 2017) <https://github.com/rh-marketingops/rh-mo-scc-luigi>`__
* `GetNinjas <https://www.getninjas.com.br/>`_ `(blog, 2017) <https://labs.getninjas.com.br/using-luigi-to-create-and-monitor-pipelines-of-batch-jobs-eb8b3cd2a574>`__
* `voyages-sncf.com <https://www.voyages-sncf.com/>`_ `(presentation, 2017) <https://github.com/voyages-sncf-technologies/meetup-afpy-nantes-luigi>`__
* `Open Targets <https://www.opentargets.org/>`_ `(blog, 2017) <https://blog.opentargets.org/using-containers-with-luigi>`__
* `Leipzig University Library <https://ub.uni-leipzig.de>`_ `(presentation, 2016) <https://de.slideshare.net/MartinCzygan/build-your-own-discovery-index-of-scholary-eresources>`__ / `(project) <https://finc.info/de/datenquellen>`__
* `Synetiq <https://synetiq.net/>`_ `(presentation, 2017) <https://www.youtube.com/watch?v=M4xUQXogSfo>`__
* `Glossier <https://www.glossier.com/>`_ `(blog, 2018) <https://medium.com/glossier/how-to-build-a-data-warehouse-what-weve-learned-so-far-at-glossier-6ff1e1783e31>`__
* `Data Revenue <https://www.datarevenue.com/>`_ `(blog, 2018) <https://www.datarevenue.com/en/blog/how-to-scale-your-machine-learning-pipeline>`_
* `Uppsala University <http://pharmb.io>`_ `(tutorial) <http://uppnex.se/twiki/do/view/Courses/EinfraMPS2015/Luigi.html>`_   / `(presentation, 2015) <https://www.youtube.com/watch?v=f26PqSXZdWM>`_ / `(slides, 2015) <https://www.slideshare.net/SamuelLampa/building-workflows-with-spotifys-luigi>`_ / `(poster, 2015) <https://pharmb.io/poster/2015-sciluigi/>`_ / `(paper, 2016) <https://doi.org/10.1186/s13321-016-0179-6>`_ / `(project) <https://github.com/pharmbio/sciluigi>`_
* `GIPHY <https://giphy.com/>`_ `(blog, 2019) <https://engineering.giphy.com/luigi-the-10x-plumber-containerizing-scaling-luigi-in-kubernetes/>`__
* `xtream <https://xtreamers.io/>`__ `(blog, 2019) <https://towardsdatascience.com/lessons-from-a-real-machine-learning-project-part-1-from-jupyter-to-luigi-bdfd0b050ca5>`__
* `CIAN <https://cian.ru/>`__ `(presentation, 2019) <https://www.highload.ru/moscow/2019/abstracts/6030>`__

Some more companies are using Luigi but haven't had a chance yet to write about it:

* `Schibsted <http://www.schibsted.com/>`_
* `enbrite.ly <http://enbrite.ly/>`_
* `Dow Jones / The Wall Street Journal <http://wsj.com>`_
* `Hotels.com <https://hotels.com>`_
* `Newsela <https://newsela.com>`_
* `Squarespace <https://www.squarespace.com/>`_
* `OAO <https://adops.com/>`_
* `Grovo <https://grovo.com/>`_
* `Weebly <https://www.weebly.com/>`_
* `Deloitte <https://www.Deloitte.co.uk/>`_
* `Stacktome <https://stacktome.com/>`_
* `LINX+Neemu+Chaordic <https://www.chaordic.com.br/>`_
* `Foxberry <https://www.foxberry.com/>`_
* `Okko <https://okko.tv/>`_
* `ISVWorld <http://isvworld.com/>`_
* `Big Data <https://bigdata.com.br/>`_
* `Movio <https://movio.co.nz/>`_
* `Bonnier News <https://www.bonniernews.se/>`_
* `Starsky Robotics <https://www.starsky.io/>`_
* `BaseTIS <https://www.basetis.com/>`_
* `Hopper <https://www.hopper.com/>`_
* `VOYAGE GROUP/Zucks <https://zucks.co.jp/en/>`_
* `Textpert <https://www.textpert.ai/>`_
* `Tracktics <https://www.tracktics.com/>`_
* `Whizar <https://www.whizar.com/>`_
* `xtream <https://www.xtreamers.io/>`__
* `Skyscanner <https://www.skyscanner.net/>`_
* `Jodel <https://www.jodel.com/>`_
* `Mekar <https://mekar.id/en/>`_
* `M3 <https://corporate.m3.com/en/>`_
* `Assist Digital <https://www.assistdigital.com/>`_
* `Meltwater <https://www.meltwater.com/>`_
* `DevSamurai <https://www.devsamurai.com/>`_
* `Veridas <https://veridas.com/>`_

We're more than happy to have your company added here. Just send a PR on GitHub.

External links
--------------

* `Mailing List <https://groups.google.com/d/forum/luigi-user/>`_ for discussions and asking questions. (Google Groups)
* `Releases <https://pypi.python.org/pypi/luigi>`_ (PyPI)
* `Source code <https://github.com/spotify/luigi>`_ (GitHub)
* `Hubot Integration <https://github.com/houzz/hubot-luigi>`_ plugin for Slack, Hipchat, etc (GitHub)

Authors
-------

Luigi was built at `Spotify <https://www.spotify.com>`_, mainly by
`Erik Bernhardsson <https://github.com/erikbern>`_ and
`Elias Freider <https://github.com/freider>`_.
`Many other people <https://github.com/spotify/luigi/graphs/contributors>`_
have contributed since open sourcing in late 2012.
`Arash Rouhani <https://github.com/tarrasch>`_ was the chief maintainer from 2015 to 2019, and now
Spotify's Data Team maintains Luigi.


.. image:: https://raw.githubusercontent.com/scipy/scipy/main/doc/source/_static/logo.svg
  :target: https://scipy.org
  :width: 110
  :height: 110
  :align: left

.. image:: https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A
  :target: https://numfocus.org

.. image:: https://img.shields.io/pypi/dm/scipy.svg?label=Pypi%20downloads
  :target: https://pypi.org/project/scipy/

.. image:: https://img.shields.io/conda/dn/conda-forge/scipy.svg?label=Conda%20downloads
  :target: https://anaconda.org/conda-forge/scipy

.. image:: https://img.shields.io/badge/stackoverflow-Ask%20questions-blue.svg
  :target: https://stackoverflow.com/questions/tagged/scipy

.. image:: https://img.shields.io/badge/DOI-10.1038%2Fs41592--019--0686--2-blue.svg
  :target: https://www.nature.com/articles/s41592-019-0686-2

SciPy (pronounced "Sigh Pie") is an open-source software for mathematics,
science, and engineering. It includes modules for statistics, optimization,
integration, linear algebra, Fourier transforms, signal and image processing,
ODE solvers, and more.

- **Website:** https://scipy.org
- **Documentation:** https://docs.scipy.org/doc/scipy/
- **Development version of the documentation:** https://scipy.github.io/devdocs
- **SciPy development forum:** https://discuss.scientific-python.org/c/contributor/scipy
- **Source code:** https://github.com/scipy/scipy
- **Contributing:** https://scipy.github.io/devdocs/dev/index.html
- **Bug reports:** https://github.com/scipy/scipy/issues
- **Code of Conduct:** https://docs.scipy.org/doc/scipy/dev/conduct/code_of_conduct.html
- **Report a security vulnerability:** https://tidelift.com/docs/security
- **Citing in your work:** https://www.scipy.org/citing-scipy/

SciPy is built to work with
NumPy arrays, and provides many user-friendly and efficient numerical routines,
such as routines for numerical integration and optimization. Together, they
run on all popular operating systems, are quick to install, and are free of
charge. NumPy and SciPy are easy to use, but powerful enough to be depended
upon by some of the world's leading scientists and engineers. If you need to
manipulate numbers on a computer and display or publish the results, give
SciPy a try!

For the installation instructions, see `our install
guide <https://scipy.org/install/>`__.

Call for Contributions
----------------------

We appreciate and welcome contributions. Small improvements or fixes are always appreciated; issues labeled as "good
first issue" may be a good starting point. Have a look at `our contributing
guide <https://scipy.github.io/devdocs/dev/index.html>`__.

Writing code isn’t the only way to contribute to SciPy. You can also:

- review pull requests
- triage issues
- develop tutorials, presentations, and other educational materials
- maintain and improve `our website <https://github.com/scipy/scipy.org>`__
- develop graphic design for our brand assets and promotional materials
- help with outreach and onboard new contributors
- write grant proposals and help with other fundraising efforts

If you’re unsure where to start or how your skills fit in, reach out! You can
ask on the `forum <https://discuss.scientific-python.org/c/contributor/scipy>`__
or here, on GitHub, by leaving a comment on a relevant issue that is already
open.

If you are new to contributing to open source, `this
guide <https://opensource.guide/how-to-contribute/>`__ helps explain why, what,
and how to get involved.


MoviePy
=======

.. image:: https://badge.fury.io/py/moviepy.svg
    :target: PyPI_
    :alt: MoviePy page on the Python Package Index
.. image:: https://img.shields.io/gitter/room/movie-py/gitter?color=46BC99&logo=gitter
    :target: Gitter_
    :alt: Discuss MoviePy on Gitter
.. image:: https://img.shields.io/github/actions/workflow/status/Zulko/moviepy/test_suite.yml?logo=github
    :target: https://github.com/Zulko/moviepy/actions/workflows/test_suite.yml
    :alt: Build status on gh-actions
.. image:: https://img.shields.io/coveralls/github/Zulko/moviepy/master?logo=coveralls
    :target: https://coveralls.io/github/Zulko/moviepy?branch=master
    :alt: Code coverage from coveralls.io

MoviePy (full documentation_) is a Python library for video editing: cutting, concatenations, title insertions, video compositing (a.k.a. non-linear editing), video processing, and creation of custom effects. See the gallery_ for some examples of use.

MoviePy can read and write all the most common audio and video formats, including GIF, and runs on Windows/Mac/Linux, with Python 3.6+. Here it is in action in an IPython notebook:

.. image:: https://raw.githubusercontent.com/Zulko/moviepy/master/docs/demo_preview.jpeg
    :alt: [logo]
    :align: center

Example
-------

In this example we open a video file, select the subclip between t=50s and t=60s, add a title at the center of the screen, and write the result to a new file:

.. code:: python

    from moviepy import *

    video = VideoFileClip("myHolidays.mp4").subclip(50,60)

    # Make the text. Many more options are available.
    txt_clip = ( TextClip("My Holidays 2013",fontsize=70,color='white')
                 .with_position('center')
                 .with_duration(10) )

    result = CompositeVideoClip([video, txt_clip]) # Overlay text on video
    result.write_videofile("myHolidays_edited.webm",fps=25) # Many options...

*Note:* This example uses the new 2.x API, for MoviePy 1.0.3, currently on PyPI, see `this snippet <https://gist.github.com/Zulko/57e6e50debef1834fb9b60700b1b9f99>`_.

Maintainers wanted!
-------------------

As there are more and more people seeking support (270 open issues as of Jan. 2021!) and all the MoviePy maintainers seem busy, we'd love to hear about developers interested in giving a hand and solving some of the issues (especially the ones that affect you) or reviewing pull requests. Open an issue or contact us directly if you are interested. Thanks!

Installation
------------

MoviePy depends on the Python modules NumPy_, Imageio_, Decorator_, and Proglog_, which will be automatically installed during MoviePy's installation. The software FFMPEG should be automatically downloaded/installed (by imageio) during your first use of MoviePy (installation will take a few seconds). If you want to use a specific version of FFMPEG, follow the instructions in ``config_defaults.py``. In case of trouble, provide feedback.

**Installation by hand:** download the sources, either from PyPI_ or, if you want the development version, from GitHub_, unzip everything into one folder, open a terminal and type:

.. code:: bash

    $ (sudo) python setup.py install

**Installation with pip:** if you have ``pip`` installed, just type this in a terminal:

.. code:: bash

    $ (sudo) pip install moviepy

If you have neither ``setuptools`` nor ``ez_setup`` installed, the command above will fail. In this case type this before installing:

.. code:: bash

    $ (sudo) pip install setuptools

Optional but useful dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can install ``moviepy`` with all dependencies via:

.. code:: bash

    $ (sudo) pip install moviepy[optional]

ImageMagick_ is not strictly required, but needed if you want to incorporate texts. It can also be used as a backend for GIFs, though you can also create GIFs with MoviePy without ImageMagick.

Once you have installed ImageMagick, MoviePy will try to autodetect the path to its executable. If it fails, you can still configure it by setting environment variables (see the documentation).

PyGame_ is needed for video and sound previews (not relevant if you intend to work with MoviePy on a server but essential for advanced video editing by hand).

For advanced image processing, you will need one or several of the following packages:

- The Python Imaging Library (PIL) or, even better, its branch Pillow_.
- Scipy_ (for tracking, segmenting, etc.) can be used to resize video clips if PIL and OpenCV are not installed.
- `Scikit Image`_ may be needed for some advanced image manipulation.
- `OpenCV 2.4.6`_ or a more recent version (one that provides the package ``cv2``) may be needed for some advanced image manipulation.
- `Matplotlib`_

For instance, using the method ``clip.resize`` requires that at least one of Scipy, PIL, Pillow or OpenCV is installed.

Documentation
-------------

Building the documentation has additional dependencies that require installation.

.. code:: bash

    $ (sudo) pip install moviepy[doc]

The documentation can be generated and viewed via:

.. code:: bash

    $ python setup.py build_docs

You can pass additional arguments to the documentation build, such as clean build:

.. code:: bash

    $ python setup.py build_docs -E

More information is available from the `Sphinx`_ documentation.

New in 1.0.0: Progress bars and messages with Proglog
-------------------------------------------------------

Non-backwards-compatible changes were introduced in 1.0.0 to
manage progress bars and messages using
`Proglog <https://github.com/Edinburgh-Genome-Foundry/Proglog>`_, which
enables to display nice progress bars in the console as well as in
a Jupyter notebook or any user interface, like a website.

To display notebook friendly progress bars, first install IPyWidgets:

.. code::

    sudo pip install ipywidgets
    sudo jupyter nbextension enable --py --sys-prefix widgetsnbextension

Then at the beginning of your notebook enter:

.. code:: python

    import proglog
    proglog.notebook()

Have a look at the Proglog project page for more options.

Contribute
----------

MoviePy is open-source software originally written by Zulko_ and released under the MIT licence. The project is hosted on GitHub_, where everyone is welcome to contribute, ask for help or simply give feedback. Please read our `Contributing Guidelines`_ for more information about how to contribute!

You can also discuss the project on Reddit_ or Gitter_. These are preferred over GitHub issues for usage questions and examples.

Maintainers
-----------

- Zulko_ (owner)
- `@tburrows13`_
- `@mgaitan`_
- `@earney`_
- `@mbeacom`_
- `@overdrivr`_
- `@keikoro`_
- `@ryanfox`_
- `@mondeja`_

.. MoviePy links
.. _gallery: https://zulko.github.io/moviepy/gallery.html
.. _documentation: https://zulko.github.io/moviepy/
.. _`download MoviePy`: https://github.com/Zulko/moviepy
.. _`Label Wiki`: https://github.com/Zulko/moviepy/wiki/Label-Wiki
.. _Contributing Guidelines: https://github.com/Zulko/moviepy/blob/master/CONTRIBUTING.md

.. Websites, Platforms
.. _Reddit: https://www.reddit.com/r/moviepy/
.. _PyPI: https://pypi.python.org/pypi/moviepy
.. _GitHub: https://github.com/Zulko/moviepy
.. _Gitter: https://gitter.im/movie-py/Lobby

.. Software, Tools, Libraries
.. _Pillow: https://pillow.readthedocs.org/en/latest/
.. _Scipy: https://www.scipy.org/
.. _`OpenCV 2.4.6`: https://github.com/skvark/opencv-python
.. _Pygame: https://www.pygame.org/download.shtml
.. _Numpy: https://www.scipy.org/install.html
.. _imageio: https://imageio.github.io/
.. _`Scikit Image`: https://scikit-image.org/docs/stable/install.html
.. _Decorator: https://pypi.python.org/pypi/decorator
.. _proglog: https://github.com/Edinburgh-Genome-Foundry/Proglog
.. _ffmpeg: https://www.ffmpeg.org/download.html
.. _ImageMagick: https://www.imagemagick.org/script/index.php
.. _`Matplotlib`: https://matplotlib.org/
.. _`Sphinx`: https://www.sphinx-doc.org/en/master/setuptools.html

.. People
.. _Zulko: https://github.com/Zulko
.. _`@mgaitan`: https://github.com/mgaitan
.. _`@tburrows13`: https://github.com/tburrows13
.. _`@earney`: https://github.com/earney
.. _`@mbeacom`: https://github.com/mbeacom
.. _`@overdrivr`: https://github.com/overdrivr
.. _`@keikoro`: https://github.com/keikoro
.. _`@ryanfox`: https://github.com/ryanfox
.. _`@mondeja`: https://github.com/mondeja


pip - The Python Package Installer
==================================

.. |pypi-version| image:: https://img.shields.io/pypi/v/pip.svg
   :target: https://pypi.org/project/pip/
   :alt: PyPI

.. |python-versions| image:: https://img.shields.io/pypi/pyversions/pip
   :target: https://pypi.org/project/pip
   :alt: PyPI - Python Version

.. |docs-badge| image:: https://readthedocs.org/projects/pip/badge/?version=latest
   :target: https://pip.pypa.io/en/latest
   :alt: Documentation

|pypi-version| |python-versions| |docs-badge|

pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.

Please take a look at our documentation for how to install and use pip:

* `Installation`_
* `Usage`_

We release updates regularly, with a new version every 3 months. Find more details in our documentation:

* `Release notes`_
* `Release process`_

If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms:

* `Issue tracking`_
* `Discourse channel`_
* `User IRC`_

If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:

* `GitHub page`_
* `Development documentation`_
* `Development IRC`_

Code of Conduct
---------------

Everyone interacting in the pip project's codebases, issue trackers, chat
rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_.

.. _package installer: https://packaging.python.org/guides/tool-recommendations/
.. _Python Package Index: https://pypi.org
.. _Installation: https://pip.pypa.io/en/stable/installation/
.. _Usage: https://pip.pypa.io/en/stable/
.. _Release notes: https://pip.pypa.io/en/stable/news.html
.. _Release process: https://pip.pypa.io/en/latest/development/release-process/
.. _GitHub page: https://github.com/pypa/pip
.. _Development documentation: https://pip.pypa.io/en/latest/development
.. _Issue tracking: https://github.com/pypa/pip/issues
.. _Discourse channel: https://discuss.python.org/c/packaging
.. _User IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa
.. _Development IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa-dev
.. _PSF Code of Conduct: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md