===================
Jira Python Library
===================

.. image:: https://img.shields.io/pypi/v/jira.svg
    :target: https://pypi.python.org/pypi/jira/

.. image:: https://img.shields.io/pypi/l/jira.svg
    :target: https://pypi.python.org/pypi/jira/

.. image:: https://img.shields.io/github/issues/pycontribs/jira.svg
    :target: https://github.com/pycontribs/jira/issues

.. image:: https://img.shields.io/badge/irc-%23pycontribs-blue
    :target: irc:///#pycontribs

------------

.. image:: https://readthedocs.org/projects/jira/badge/?version=main
    :target: https://jira.readthedocs.io/

.. image:: https://codecov.io/gh/pycontribs/jira/branch/main/graph/badge.svg
    :target: https://codecov.io/gh/pycontribs/jira

.. image:: https://img.shields.io/bountysource/team/pycontribs/activity.svg
    :target: https://www.bountysource.com/teams/pycontribs/issues?tracker_ids=3650997

This library eases the use of the Jira REST API from Python and it has been used in production for years.

As this is an open-source project that is community maintained, do not be surprised if some bugs or features are not implemented quickly enough. You are always welcomed to use BountySource_ to motivate others to help.

.. _BountySource: https://www.bountysource.com/teams/pycontribs/issues?tracker_ids=3650997

Quickstart
----------

Feeling impatient? I like your style.

.. code-block:: python

    from jira import JIRA

    jira = JIRA('https://jira.atlassian.com')

    issue = jira.issue('JRA-9')
    print(issue.fields.project.key)            # 'JRA'
    print(issue.fields.issuetype.name)         # 'New Feature'
    print(issue.fields.reporter.displayName)   # 'Mike Cannon-Brookes [Atlassian]'

Installation
------------

Download and install using ``pip install jira`` or ``easy_install jira``

You can also try ``pip install --user --upgrade jira`` which will install or
upgrade jira to your user directory. Or maybe you ARE using a virtualenv_
right?

By default only the basic library dependencies are installed, so if you want
to use the ``cli`` tool or other optional dependencies do perform a full
installation using ``pip install jira[opt,cli,test]``

.. _virtualenv: https://virtualenv.pypa.io/

Usage
-----

See the documentation_ for full details.

.. _documentation: https://jira.readthedocs.org/

Development
-----------

Development takes place on GitHub_ using the default repository branch. Each
version is tagged.

Setup
=====
* Fork_ repo
* Keep it sync_'ed while you are developing

Automatic (VS Code)
```````````````````
.. image:: https://img.shields.io/static/v1?label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode
    :target: https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/pycontribs/jira
    :alt: Open in Remote - Containers

Follow the instructions in the `contributing guide`_, which will describe how to use the dev container
that will automatically setup a suitable environment.

Manual
``````
* Install pyenv_ to install a suitable python version.
* Launch docker jira server
    - ``docker run -dit -p 2990:2990 --name jira addono/jira-software-standalone``

tox envs
````````
* Lint
    - ``tox -e lint``
* Run tests
    - ``tox``
* Build and publish with TWINE
    - ``tox -e publish``

.. _Fork: https://help.github.com/articles/fork-a-repo/
.. _sync: https://help.github.com/articles/syncing-a-fork/
.. _pyenv: https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
.. _pytest: https://docs.pytest.org/en/stable/usage.html#specifying-tests-selecting-tests
.. _contributing guide: https://jira.readthedocs.io/contributing.html

Jira REST API Reference Links
=============================

When updating interactions with the Jira REST API please refer to the documentation below. We aim to support both Jira Cloud and Jira Server / Data Center.

1. `Jira Cloud`_                / `Jira Server`_ (main REST API reference)
2. `Jira Software Cloud`_       / `Jira Software Server`_ (former names include: Jira Agile, Greenhopper)
3. `Jira Service Desk Cloud`_   / `Jira Service Desk Server`_

.. _`Jira Cloud`: https://developer.atlassian.com/cloud/jira/platform/rest/v2/
.. _`Jira Server`: https://docs.atlassian.com/software/jira/docs/api/REST/latest/
.. _`Jira Software Cloud`: https://developer.atlassian.com/cloud/jira/software/rest/
.. _`Jira Software Server`: https://docs.atlassian.com/jira-software/REST/latest/
.. _`Jira Service Desk Cloud`: https://docs.atlassian.com/jira-servicedesk/REST/cloud/
.. _`Jira Service Desk Server`: https://docs.atlassian.com/jira-servicedesk/REST/server/

Credits
-------

In addition to all the contributors we would like to thank to these companies:

* Atlassian_ for developing such a powerful issue tracker and for providing a free on-demand Jira_ instance that we can use for continuous integration testing.
* JetBrains_ for providing us with free licenses of PyCharm_
* GitHub_ for hosting our continuous integration and our git repo
* Navicat_ for providing us free licenses of their powerful database client GUI tools.

.. _Atlassian: https://www.atlassian.com/
.. _Jira: https://pycontribs.atlassian.net
.. _JetBrains: https://www.jetbrains.com/
.. _PyCharm: https://www.jetbrains.com/pycharm/
.. _GitHub: https://github.com/pycontribs/jira
.. _Navicat: https://www.navicat.com/

.. image:: https://raw.githubusercontent.com/pycontribs/resources/main/logos/x32/logo-atlassian.png
   :target: https://www.atlassian.com/

.. image:: https://raw.githubusercontent.com/pycontribs/resources/main/logos/x32/logo-pycharm.png
    :target: https://www.jetbrains.com/

.. image:: https://raw.githubusercontent.com/pycontribs/resources/main/logos/x32/logo-navicat.png
    :target: https://www.navicat.com/


PyParsing -- A Python Parsing Module
====================================

|Version| |Build Status| |Coverage| |License| |Python Versions| |Snyk Score|

Introduction
============

The pyparsing module is an alternative approach to creating and
executing simple grammars, vs. the traditional lex/yacc approach, or the
use of regular expressions. The pyparsing module provides a library of
classes that client code uses to construct the grammar directly in
Python code.

*[Since first writing this description of pyparsing in late 2003, this
technique for developing parsers has become more widespread, under the
name Parsing Expression Grammars - PEGs. See more information on PEGs*
`here <https://en.wikipedia.org/wiki/Parsing_expression_grammar>`__
*.]*

Here is a program to parse ``"Hello, World!"`` (or any greeting of the form
``"salutation, addressee!"``):

.. code:: python

    from pyparsing import Word, alphas
    greet = Word(alphas) + "," + Word(alphas) + "!"
    hello = "Hello, World!"
    print(hello, "->", greet.parseString(hello))

The program outputs the following::

    Hello, World! -> ['Hello', ',', 'World', '!']

The Python representation of the grammar is quite readable, owing to the
self-explanatory class names, and the use of '+', '|' and '^' operator
definitions.

The parsed results returned from ``parseString()`` is a collection of type
``ParseResults``, which can be accessed as a
nested list, a dictionary, or an object with named attributes.

The pyparsing module handles some of the problems that are typically
vexing when writing text parsers:

- extra or missing whitespace (the above program will also handle ``"Hello,World!"``, ``"Hello , World !"``, etc.)
- quoted strings
- embedded comments

The examples directory includes a simple SQL parser, simple CORBA IDL
parser, a config file parser, a chemical formula parser, and a four-
function algebraic notation parser, among many others.

Documentation
=============

There are many examples in the online docstrings of the classes
and methods in pyparsing. You can find them compiled into `online docs <https://pyparsing-docs.readthedocs.io/en/latest/>`__. Additional
documentation resources and project info are listed in the online
`GitHub wiki <https://github.com/pyparsing/pyparsing/wiki>`__. An
entire directory of examples can be found `here <https://github.com/pyparsing/pyparsing/tree/master/examples>`__.

License
=======

MIT License. See header of the `pyparsing __init__.py <https://github.com/pyparsing/pyparsing/blob/master/pyparsing/__init__.py#L1-L23>`__ file.

History
=======

See `CHANGES <https://github.com/pyparsing/pyparsing/blob/master/CHANGES>`__ file.

.. |Build Status| image:: https://github.com/pyparsing/pyparsing/actions/workflows/ci.yml/badge.svg
   :target: https://github.com/pyparsing/pyparsing/actions/workflows/ci.yml

.. |Coverage| image:: https://codecov.io/gh/pyparsing/pyparsing/branch/master/graph/badge.svg
  :target: https://codecov.io/gh/pyparsing/pyparsing

.. |Version| image:: https://img.shields.io/pypi/v/pyparsing?style=flat-square
    :target: https://pypi.org/project/pyparsing/
    :alt: Version

.. |License| image:: https://img.shields.io/pypi/l/pyparsing.svg?style=flat-square
    :target: https://pypi.org/project/pyparsing/
    :alt: License

.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pyparsing.svg?style=flat-square
    :target: https://pypi.org/project/python-liquid/
    :alt: Python versions

.. |Snyk Score| image:: https://snyk.io//advisor/python/pyparsing/badge.svg
   :target: https://snyk.io//advisor/python/pyparsing
   :alt: pyparsing


**Português (BR)** | [English (US)](/docs/README-en-US.md)

# Querido Diário
Dentro do [ecossistema do Querido Diário](https://docs.queridodiario.ok.org.br/pt-br/latest/contribuindo/guia-de-contribuicao.html#ecossistema-do-querido-diario), este repositório é o responsável pela tarefa de **raspagem dos sites publicadores de diários oficiais**.

Conheça mais sobre as [tecnologias](https://queridodiario.ok.org.br/tecnologia) e a [história](https://queridodiario.ok.org.br/sobre) do projeto no [site do Querido Diário](https://queridodiario.ok.org.br)

# Sumário
- [Como contribuir](#como-contribuir)
- [Ambiente de desenvolvimento](#ambiente-de-desenvolvimento)
- [Template para raspadores](#template-para-raspadores)
- [Como executar](#como-executar)
  - [Dicas de execução](#dicas-de-execução)
- [Solução de problemas](#solução-de-problemas)
- [Suporte](#suporte)
- [Agradecimentos](#agradecimentos)
- [Open Knowledge Brasil](#open-knowledge-brasil)
- [Licença](#licença)

# Como contribuir

    <img alt="catarse" src="https://img.shields.io/badge/Catarse-Apoie%20o%20projeto-orange?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB2ZXJzaW9uPSIxLjIiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgdmlld0JveD0iMCAwIDMyNSA0NTUiIHdpZHRoPSIzMjUiIGhlaWdodD0iNDU1Ij4KCTx0aXRsZT5sb2dvLXYtY29yLXBvc2l0aXZvLWFpPC90aXRsZT4KCTxzdHlsZT4KCQkuczAgeyBmaWxsOiAjMTdhMzM4IH0gCgkJLnMxIHsgZmlsbDogIzdkYjkzZCB9IAoJCS5zMiB7IGZpbGw6ICNmMmJmMDAgfSAKCQkuczMgeyBmaWxsOiAjZjE5MTA2IH0gCgkJLnM0IHsgZmlsbDogI2VhNjYwYiB9IAoJCS5zNSB7IGZpbGw6ICNkZTMyODEgfSAKCTwvc3R5bGU+Cgk8ZyBpZD0iTGF5ZXIgMSI+CgkJPGcgaWQ9IiZsdDtHcm91cCZndDsiPgoJCQk8cGF0aCBpZD0iJmx0O1BhdGgmZ3Q7IiBjbGFzcz0iczAiIGQ9Im01Ni40IDI1Ni40cS0xLjggMy4xLTMuNCA2LjRjLTE1LjQgMzMuMS00LjcgNjMuMSAxNC44IDkyLjQgMTEuNiAxNy40IDguNiAzNi40LTEuOSA0Ny4zLTYgNi4zLTE0LjUgOS44LTIzLjIgOS43LTQuNSAwLTkuMS0wLjktMTMuNS0zLTYuMy0yLjktMTUuMS05LjEtMjIuMi0yOC43LTguOS0yNC4yLTkuMS01MS42IDIuNi03Ni44IDEwLjEtMjEuNiAyNi45LTM3LjggNDYuOC00Ny4zeiIvPgoJCQk8cGF0aCBpZD0iJmx0O1BhdGgmZ3Q7IiBjbGFzcz0iczEiIGQ9Im00Ni41IDMwNS44YzAuNSAyLjYgMSA1LjIgMS43IDcuOCAxMC41IDM4LjcgNDAuNyA1Ni41IDc3LjggNjcuNCAyMi4xIDYuNSAzMyAyNC43IDMxLjkgNDEuMi0wLjcgOS42LTUuMyAxOC41LTEyLjcgMjQuNi0zLjggMy4yLTguNCA1LjctMTMuNSA3LTcuNCAyLTE5LjIgMy0zOS4xLTguNC0yNC41LTE0LjItNDQuMS0zNy4yLTUyLTY2LjctNi44LTI1LjItNC4xLTUwLjggNS45LTcyLjl6Ii8+CgkJCTxwYXRoIGlkPSImbHQ7UGF0aCZndDsiIGNsYXNzPSJzMiIgZD0ibTczLjIgMzU0LjVjMi4yIDEuOCA0LjUgMy42IDYuOSA1LjMgMzYuMiAyNS4zIDc0LjMgMTguOSAxMTMuMiAxLjggMjMuMi0xMC4xIDQ1LjMtMi41IDU2IDEyLjIgNi4zIDguNiA4LjYgMTkuNCA2LjggMjkuNy0xIDUuNC0zLjEgMTAuNy02LjUgMTUuNS00LjggNi45LTE0IDE2LjEtMzguOSAyMC41LTMwLjcgNS40LTYzLjQtMC4xLTkwLjktMTkuNC0yMy42LTE2LjUtMzkuNC0zOS45LTQ2LjYtNjUuNnoiLz4KCQkJPHBhdGggaWQ9IiZsdDtQYXRoJmd0OyIgY2xhc3M9InMzIiBkPSJtMTMwLjEgMzc2LjZxNC43IDAuMSA5LjUtMC40YzQ4LjQtNC4zIDc2LTM2LjUgOTYuNy03OC41IDEyLjQtMjUgMzYuNC0zNC4xIDU1LjgtMjkuMyAxMS40IDIuOCAyMSAxMC4yIDI2LjcgMjAuMyAzIDUuMiA1IDExLjEgNS41IDE3LjYgMC45IDkuMi0wLjQgMjMuNC0xOC4zIDQ0LjctMjIgMjYuMy01My41IDQ0LjgtOTAuMyA0OC0zMS41IDIuOC02MS40LTUuOC04NS42LTIyLjR6Ii8+CgkJCTxwYXRoIGlkPSImbHQ7UGF0aCZndDsiIGNsYXNzPSJzNCIgZD0ibTE5My42IDM1NS4xYzIuNy0yLjIgNS4zLTQuNiA3LjgtNy4xIDM3LjctMzcuOCAzOC4xLTg0LjUgMjUuOS0xMzQuNS03LjItMjkuOCA2LjUtNTQuNSAyNi4zLTY0LjIgMTEuNS01LjYgMjQuOS02LjEgMzYuOC0xLjggNi4zIDIuMyAxMi4xIDUuOSAxNy4xIDExIDcuMiA3LjEgMTYuMiAyMC4xIDE2LjMgNTAuNiAwIDM3LjctMTMuNSA3NS42LTQyLjIgMTA0LjMtMjQuNiAyNC43LTU1LjkgMzguNi04OCA0MS43eiIvPgoJCQk8cGF0aCBpZD0iJmx0O1BhdGgmZ3Q7IiBjbGFzcz0iczUiIGQ9Im0yMzEuOSAyOTJjMC44LTQuNCAxLjQtOC45IDEuOC0xMy41IDYtNjkuMi0zMi42LTExNi04Ni41LTE1NS43LTMyLjItMjMuNi0zOS4xLTU5LjYtMjcuNS04NS44IDYuNy0xNS4zIDE5LjctMjcgMzUuMi0zMi42IDguMS0yLjkgMTctNC4yIDI2LjEtMy40IDEzLjIgMS4xIDMzIDYuNSA1OC42IDM2LjkgMzEuNSAzNy41IDQ5LjcgODYuNSA0NS4xIDEzOS4xLTMuOSA0NS4xLTIzLjQgODUuMS01Mi44IDExNXoiLz4KCQk8L2c+Cgk8L2c+Cjwvc3ZnPg==">

Agradecemos por considerar contribuir com o Querido Diário! :tada:

Você encontra como fazê-lo no [CONTRIBUTING.md](/docs/CONTRIBUTING.md)!

Além disso, consulte a [documentação do Querido Diário](https://docs.queridodiario.ok.org.br/pt-br/latest/) para te ajudar.

# Ambiente de desenvolvimento
Você precisa ter [Python](https://docs.python.org/3/) (+3.0) e o framework [Scrapy](https://scrapy.org) instalados.

Os comandos abaixo preparam o ambiente em sistema operacional Linux. Eles consistem em criar um [ambiente virtual de Python](https://docs.python.org/pt-br/3/library/venv.html), instalar os requisitos listados em `requirements-dev` e a ferramenta para padronização de código `pre-commit`.

``` console
python3 -m venv .venv
source .venv/bin/activate
pip install -r data_collection/requirements-dev.txt
pre-commit install
```

> A configuração em outros sistemas operacionais está disponível em ["como configurar o ambiente de desenvolvimento"](/docs/CONTRIBUTING.md#como-configurar-o-ambiente-de-desenvolvimento), incluindo mais detalhes para quem deseja contribuir com o desenvolvimento do repositório.

# Template para raspadores

Ao invés de começar um arquivo de raspador do zero, você pode inicializar um arquivo de código de raspador já no padrão do Querido Diário, a partir de um template. Para isso, faça:

1. Vá para o diretório `data_collection`:
```console
cd data_collection
```
2. Acione o template:
```console
scrapy genspider -t qdtemplate <uf_nome_do_municipio> <https://sitedomunicipio...>
```

Um arquivo `uf_nome_do_municipio.py` será criado no diretório `spiders`, com alguns campos já preenchidos. O diretório é organizado por UF, lembre-se de mover o arquivo para o diretório adequado.

# Como executar
Para experimentar a execução de um raspador já integrado ao projeto ou testar o que esteja desenvolvendo, siga os comandos:

1. Se ainda não o fez, ative o ambiente virtual no diretório `/querido-diario`:
``` console
source .venv/bin/activate
```
2. Vá para o diretório `data_collection`:
```console
cd data_collection
```
3. Verifique a lista de raspadores disponíveis:
```console
scrapy list
```
4. Execute um raspador da lista:
```console
scrapy crawl <nome_do_raspador>       //exemplo: scrapy crawl ba_acajutiba
```
5. Os diários coletados na raspagem serão salvos no diretório `data_collection/data`

## Dicas de execução
Além dos comandos acima, o Scrapy oferece outros recursos para configurar o comando de raspagem. Os recursos a seguir podem ser usados sozinhos ou combinados.

* **Limite de data**
Ao executar o item 4, o raspador coletará todos os diários oficiais do site publicador daquele município. Para execuções menores, utilize a flag de atributo `-a` seguida de:

`start_date=AAAA-MM-DD`: definirá a data inicial de coleta de diários.
```console
scrapy crawl <nome_do_raspador> -a start_date=<AAAA-MM-DD>
```
`end_date=AAAA-MM-DD`: definirá a data final de coleta de diários. Caso omitido, assumirá a data do dia em que está sendo executado.
```console
scrapy crawl <nome_do_raspador> -a end_date=<AAAA-MM-DD>
```

* **Arquivo de log**
É possível enviar o log da raspagem para um arquivo ao invés de deixá-lo no terminal. Isto é particularmente útil quando se desenvolve um raspador que apresenta problemas e você quer enviar o arquivo de log no seu PR para obter ajuda. Para isso, use a flag de configuração `-s` seguida de:

`LOG_FILE=log_<nome_do_municipio>.txt`: definirá o arquivo para armazenar as mensagens de log.
```console
scrapy crawl <nome_do_raspador> -s LOG_FILE=log_<nome_do_municipio>.txt
```

* **Tabela de raspagem**
Também é possível construir uma tabela que lista todos os diários e metadados coletados pela raspagem, ficando mais fácil de ver como o raspador está se comportando. Para isso, use a flag de saída `-o` seguida de um nome para o arquivo.
```console
scrapy crawl <nome_do_raspador> -o <nome_do_municipio>.csv
```

# Solução de problemas
Confira o arquivo de [solução de problemas](/docs/TROUBLESHOOTING.md) para resolver os problemas mais frequentes com a configuração do ambiente do projeto.

# Suporte

Ingresse em nosso [canal de comunidade](https://go.ok.org.br/discord) para trocas sobre os projetos, dúvidas, pedidos de ajuda com contribuição e conversar sobre inovação cívica em geral.

# Agradecimentos
Este projeto é mantido pela Open Knowledge Brasil e possível graças às comunidades técnicas, às [Embaixadoras de Inovação Cívica](https://embaixadoras.ok.org.br/), às pessoas voluntárias e doadoras financeiras, além de universidades parceiras, empresas apoiadoras e financiadoras.

Conheça [quem apoia o Querido Diário](https://queridodiario.ok.org.br/apoie#quem-apoia).

# Open Knowledge Brasil

A [Open Knowledge Brasil](https://ok.org.br/) é uma organização da sociedade civil sem fins lucrativos, cuja missão é utilizar e desenvolver ferramentas cívicas, projetos, análises de políticas públicas, jornalismo de dados para promover o conhecimento livre nos diversos campos da sociedade.

Todo o trabalho produzido pela OKBR está disponível livremente.

# Licença

Código licenciado sob a [Licença MIT](LICENSE.md).


# JSON Schema Test Suite

[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](https://github.com/json-schema-org/.github/blob/main/CODE_OF_CONDUCT.md)
[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
[![Financial Contributors on Open Collective](https://opencollective.com/json-schema/all/badge.svg?label=financial+contributors)](https://opencollective.com/json-schema)

[![DOI](https://zenodo.org/badge/5952934.svg)](https://zenodo.org/badge/latestdoi/5952934)
[![Build Status](https://github.com/json-schema-org/JSON-Schema-Test-Suite/workflows/Test%20Suite%20Sanity%20Checking/badge.svg)](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22)

This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators.

It is meant to be language agnostic and should require only a JSON parser.
The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.

The recommended workflow of this test suite is to clone the `main` branch of this repository as a `git submodule` or `git subtree`. The `main` branch is always stable.

## Coverage

All JSON Schema specification releases should be well covered by this suite, including drafts 2020-12, 2019-09, 07, 06, 04 and 03.
Drafts 04 and 03 are considered "frozen" in that less effort is put in to backport new tests to these versions.

Additional coverage is always welcome, particularly for bugs encountered in real-world implementations.
If you see anything missing or incorrect, please feel free to [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues) or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite).

@gregsdennis has also started a separate [test suite](https://github.com/gregsdennis/json-schema-vocab-test-suites) that is modelled after this suite to cover third-party vocabularies.

## Introduction to the Test Suite Structure

The tests in this suite are contained in the `tests` directory at the root of this repository.
Inside that directory is a subdirectory for each released version of the specification.

The structure and contents of each file in these directories is described below.

In addition to the version-specific subdirectories, two additional directories are present:

1. `draft-next/`: containing tests for the next version of the specification whilst it is in development
2. `latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)

Inside each version directory there are a number of `.json` files each containing a collection of related tests.
Often the grouping is by property under test, but not always.
In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-draft), and which contain additional `.json` files.

Each `.json` file consists of a single JSON array of test cases.

### Terminology

For clarity, we first define this document's usage of some testing terminology:

| term            | definition                                                                                |
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **test suite**  | the entirety of the contents of this repository, containing tests for multiple different releases of the JSON Schema specification                                |
| **test case**   | a single schema, along with a description and an array of *test*s                                                                                |
| **test**        | within a *test case*, a single test example, containing a description, instance and a boolean indicating whether the instance is valid under the test case schema |
| **test runner** | a program, external to this repository and authored by a user of this suite, which is executing each of the tests in the suite                                    |

An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test cases can be found [alongside this README](./test-schema.json).

### Sample Test Case

Here is a single *test case*, containing one or more tests:

```json
{
    "description": "The test case description",
    "schema": { "type": "string" },
    "tests": [
        {
            "description": "a test with a valid instance",
            "data": "a string",
            "valid": true
        },
        {
            "description": "a test with an invalid instance",
            "data": 15,
            "valid": false
        }
    ]
}
```

### Subdirectories Within Each Draft

There is currently only one additional subdirectory that may exist within each draft test directory.

This is:

1. `optional/`: Contains tests that are considered optional.

Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in
which case they are not truly optional in such a language).
In the future this directory structure will be made richer to reflect these differences more clearly.

## Using the Suite to Test a Validator Implementation

The test suite structure was described [above](#introduction-to-the-test-suite-structure).

If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes:

1. How to implement a test runner which passes tests to your validator
2. Assumptions the suite makes about how the test runner will configure your validator
3. Invariants the test suite claims to hold for its tests

### How to Implement a Test Runner

Presented here is a possible implementation of a test runner.
The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects.

To test a specific version:

* For 2019-09 and later published drafts, implementations that are able to detect the draft of each schema via `$schema` SHOULD be configured to do so
* For draft-07 and earlier, draft-next, and implementations unable to detect via `$schema`, implementations MUST be configured to expect the draft matching the test directory name
* Load any remote references [described below](#additional-assumptions) and configure your implementation to retrieve them via their URIs
* Walk the filesystem tree for that version's subdirectory and for each `.json` file found:

    * if the file is located in the root of the version directory:

        * for each test case present in the file:

            * load the schema from the `"schema"` property
            * load (or log) the test case description from the `"description"` property for debugging or outputting
            * for each test in the `"tests"` property:

                * load the instance to be tested from the `"data"` property
                * load (or log) the individual test description from the `"description"` property for debugging or outputting

                * use the schema loaded above to validate whether the instance is considered valid under your implementation

                * if the result from your implementation matches the value found in the `"valid"` property, your implementation correctly implements the specific example
                * if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific example

    * otherwise it is located in a special subdirectory as described above.
      Follow the additional assumptions and restrictions for the containing subdirectory, then run the test case as above.

If your implementation supports multiple versions, run the above procedure for each version supported, configuring your implementation as appropriate to call each version individually.

### Additional Assumptions

1. The suite, notably in its `refRemote.json` file in each draft, expects a number of remote references to be configured.
   These are JSON documents, identified by URI, which are used by the suite to test the behavior of the `$ref` keyword (and related keywords).
   Depending on your implementation, you may configure how to "register" these *either*:

    * by directly retrieving them off the filesystem from the `remotes/` directory, in which case you should load each schema with a retrieval URI of `http://localhost:1234` followed by the relative path from the remotes directory -- e.g. a `$ref` to `http://localhost:1234/foo/bar/baz.json` is expected to resolve to the contents of the file at `remotes/foo/bar/baz.json`

    * or alternatively, by executing `bin/jsonschema_suite remotes` using the executable in the `bin/` directory, which will output a JSON object containing all of the remotes combined, e.g.:

    ```

    $  bin/jsonschema_suite remotes
    ```
    ```json
    {
        "http://localhost:1234/baseUriChange/folderInteger.json": {
            "type": "integer"
        },
        "http://localhost:1234/baseUriChangeFolder/folderInteger.json": {
            "type": "integer"
        }
    }
    ```

2. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run.
   In particular, when running tests within the `optional/format` subdirectory, test runners should configure implementations to enable format validation, where the implementation supports it.

### Invariants & Guarantees

The test suite guarantees a number of things about tests it defines.
Any deviation from the below is generally considered a bug.
If you suspect one, please [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/new):

1. All files containing test cases are valid JSON.
2. The contents of the `"schema"` property in a test case are always valid
   JSON Schemas under the corresponding specification.

   The rationale behind this is that we are testing instances in a test's `"data"` element, and not the schema itself.
   A number of tests *do* test the validity of a schema itself, but do so by representing the schema as an instance inside a test, with the associated meta-schema in the `"schema"` property (via the `"$ref"` keyword):

   ```json
   {
       "description": "Test the \"type\" schema keyword",
       "schema": {
           "$ref": "https://json-schema.org/draft/2019-09/schema"
        },
       "tests": [
           {
               "description": "Valid: string",
               "data": {
                   "type": "string"
               },
               "valid": true
           },
           {
               "description": "Invalid: null",
               "data": {
                   "type": null
               },
               "valid": false
           }
       ]
   }
   ```
   See below for some [known limitations](#known-limitations).

## Known Limitations

This suite expresses its assertions about the behavior of an implementation *within* JSON Schema itself.
Each test is the application of a schema to a particular instance.
This means that the suite of tests can test against any behavior a schema can describe, and conversely cannot test against any behavior which a schema is incapable of representing, even if the behavior is mandated by the specification.

For example, a schema can require that a string is a _URI-reference_ and even that it matches a certain pattern, but even though the specification contains [recommendations about URIs being normalized](https://json-schema.org/draft/2020-12/json-schema-core.html#name-the-id-keyword), a JSON schema cannot today represent this assertion within the core vocabularies of the specifications, so no test covers this behavior.

## Who Uses the Test Suite

This suite is being used by:

### Clojure

* [jinx](https://github.com/juxt/jinx)
* [json-schema](https://github.com/tatut/json-schema)

### Coffeescript

* [jsck](https://github.com/pandastrike/jsck)

### Common Lisp

* [json-schema](https://github.com/fisxoj/json-schema)

### C++

* [Modern C++ JSON schema validator](https://github.com/pboettch/json-schema-validator)
* [Valijson](https://github.com/tristanpenman/valijson)

### Dart

* [json\_schema](https://github.com/patefacio/json_schema)

### Elixir

* [ex\_json\_schema](https://github.com/jonasschmidt/ex_json_schema)

### Erlang

* [jesse](https://github.com/for-GET/jesse)

### Go

* [gojsonschema](https://github.com/sigu-399/gojsonschema)
* [validate-json](https://github.com/cesanta/validate-json)

### Haskell

* [aeson-schema](https://github.com/timjb/aeson-schema)
* [hjsonschema](https://github.com/seagreen/hjsonschema)

### Java

* [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations)
* [json-schema-validator](https://github.com/daveclayton/json-schema-validator)
* [everit-org/json-schema](https://github.com/everit-org/json-schema)
* [networknt/json-schema-validator](https://github.com/networknt/json-schema-validator)
* [Justify](https://github.com/leadpony/justify)
* [Snow](https://github.com/ssilverman/snowy-json)
* [jsonschemafriend](https://github.com/jimblackler/jsonschemafriend)
* [OpenAPI JSON Schema Generator](https://github.com/openapi-json-schema-tools/openapi-json-schema-generator)

### JavaScript

* [json-schema-benchmark](https://github.com/Muscula/json-schema-benchmark)
* [direct-schema](https://github.com/IreneKnapp/direct-schema)
* [is-my-json-valid](https://github.com/mafintosh/is-my-json-valid)
* [jassi](https://github.com/iclanzan/jassi)
* [JaySchema](https://github.com/natesilva/jayschema)
* [json-schema-valid](https://github.com/ericgj/json-schema-valid)
* [Jsonary](https://github.com/jsonary-js/jsonary)
* [jsonschema](https://github.com/tdegrunt/jsonschema)
* [request-validator](https://github.com/bugventure/request-validator)
* [skeemas](https://github.com/Prestaul/skeemas)
* [tv4](https://github.com/geraintluff/tv4)
* [z-schema](https://github.com/zaggino/z-schema)
* [jsen](https://github.com/bugventure/jsen)
* [ajv](https://github.com/epoberezkin/ajv)
* [djv](https://github.com/korzio/djv)

### Kotlin

* [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations)

### Node.js

For node.js developers, the suite is also available as an [npm](https://www.npmjs.com/package/@json-schema-org/tests) package.

Node-specific support is maintained in a [separate repository](https://github.com/json-schema-org/json-schema-test-suite-npm) which also welcomes your contributions!

### .NET

* [JsonSchema.Net](https://github.com/gregsdennis/json-everything)
* [Newtonsoft.Json.Schema](https://github.com/JamesNK/Newtonsoft.Json.Schema)

### Perl

* [Test::JSON::Schema::Acceptance](https://github.com/karenetheridge/Test-JSON-Schema-Acceptance) (a wrapper of this test suite)
* [JSON::Schema::Modern](https://github.com/karenetheridge/JSON-Schema-Modern)
* [JSON::Schema::Tiny](https://github.com/karenetheridge/JSON-Schema-Tiny)

### PHP

* [opis/json-schema](https://github.com/opis/json-schema)
* [json-schema](https://github.com/justinrainbow/json-schema)
* [json-guard](https://github.com/thephpleague/json-guard)

### PostgreSQL

* [postgres-json-schema](https://github.com/gavinwahl/postgres-json-schema)
* [is\_jsonb\_valid](https://github.com/furstenheim/is_jsonb_valid)

### Python

* [jsonschema](https://github.com/Julian/jsonschema)
* [fastjsonschema](https://github.com/seznam/python-fastjsonschema)
* [hypothesis-jsonschema](https://github.com/Zac-HD/hypothesis-jsonschema)
* [jschon](https://github.com/marksparkza/jschon)
* [OpenAPI JSON Schema Generator](https://github.com/openapi-json-schema-tools/openapi-json-schema-generator)

### Ruby

* [json-schema](https://github.com/hoxworth/json-schema)
* [json\_schemer](https://github.com/davishmcclurg/json_schemer)

### Rust

* [jsonschema](https://github.com/Stranger6667/jsonschema-rs)
* [valico](https://github.com/rustless/valico)

### Scala

* [json-schema-validation-comparison](https://www.creekservice.org/json-schema-validation-comparison/functional) (Comparison site for JVM-based validator implementations)
* [typed-json](https://github.com/frawa/typed-json)

### Swift

* [JSONSchema](https://github.com/kylef/JSONSchema.swift)

If you use it as well, please fork and send a pull request adding yourself to
the list :).

## Contributing

If you see something missing or incorrect, a pull request is most welcome!

There are some sanity checks in place for testing the test suite. You can run
them with `bin/jsonschema_suite check` or `tox`. They will be run automatically
by [GitHub Actions](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22)
as well.

This repository is maintained by the JSON Schema organization, and will be governed by the JSON Schema steering committee (once it exists).


# [MTGJSON v5](https://mtgjson.com/)

MTGJSON is an open-source repository of [Magic: The Gathering](https://magic.wizards.com/) card data, specifically in [JSON](https://json.org/) and [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) formats. This repository contains the build scripts we use to create our data sets.

## How to Contribute

Running a project as large as ours isn't easy, and we rely on the community to help keep our project going. The community can support us through two different ways: financial contributions and code contributions.

[![Github Sponsors](https://img.shields.io/static/v1.svg?label=GitHub%20Sponsors&message=Support%20MTGJSON&color=34d058&logo=github)](https://github.com/sponsors/ZeldaZach)
We accept reoccurring donations via GitHub Sponsors, that grant priority support from MTGJSON maintainers and a special role on our Discord indicating your support.
**This is the preferred approach for financial contributions, as the MTGJSON team pays _no fees_ for these types of donations!**

[![Patreon](https://img.shields.io/static/v1.svg?label=Patreon&message=Support%20MTGJSON&color=f96854&logo=patreon)](https://patreon.com/mtgjson)
We accept reoccurring donations via Patreon, that grant priority support from MTGJSON maintainers and a special role on our Discord indicating your support.

[![PayPal](https://img.shields.io/static/v1.svg?label=PayPal&message=Support%20MTGJSON&color=009cde&logo=paypal)](https://paypal.me/zachhalpern)
We accept one-time donations via PayPal, for those who want to say thank you to the project.

[![Code Contributions](https://img.shields.io/static/v1.svg?label=GitHub&message=Development&color=aaa&logo=github)](https://github.com/mtgjson)
We love it when the community contributes back to the project! If you'd like to help improve our data for the hundreds of projects and stores we support, do reach out via [Discord](https://mtgjson.com/discord)!

## Connect With Us

[![Discord](https://img.shields.io/discord/224178957103136779?label=Discord&logo=discord&logoColor=white&color=7289da)](https://mtgjson.com/discord)
The team stays in contact via [Discord](https://mtgjson.com/discord). The server is open to the public and is a great place to collaborate with other like-minded people. Stop by today and say hi!

## About Us

### The Team
The MTGJSON team has been led by Zach Halpern since 2018, with support from an awesome group of people. The full team lineup can be found [MTGJSON's homepage](https://mtgjson.com/).

### Our Product
MTGJSON at its core is a database that can be downloaded for offline access to Magic: the Gathering card data. We pride ourselves on our [documentation](https://mtgjson.com/data-models/), and aim for full transparency with the community.

### Our Partners
Over time, MTGJSON has gone through a number of transitions to bring the best product for our consumers. We'd like to thank the following groups, in alphabetical order, for helping to support our mission by enriching our data:
- [CardHoarder](https://www.cardhoarder.com/?affiliate_id=mtgjson&utm_source=mtgjson&utm_campaign=affiliate&utm_medium=card)
- [CardKingdom](https://www.cardkingdom.com/?partner=mtgjson&utm_source=mtgjson&utm_medium=affiliate&utm_campaign=mtgjson)
- [CardMarket](https://www.cardmarket.com/en/Magic?utm_campaign=card_prices&utm_medium=text&utm_source=mtgjson)
- [Gatherer](https://gatherer.wizards.com)
- [MTG.wtf](https://mtg.wtf/)
- [MTGBan](https://www.mtgban.com/)
- [Scryfall](https://scryfall.com)
- [TCGPlayer](https://www.tcgplayer.com/?partner=mtgjson&utm_campaign=affiliate&utm_medium=mtgjson&utm_source=mtgjson)
- [What's in Standard](https://whatsinstandard.com/)

## How to Use
### For 99% of our Users
MTGJSON supplies precompiled databases at https://mtgjson.com/api/v5/. **This is the recommended way to use our service**.

As stated before, we pride ourselves on our documentation which can be found at https://mtgjson.com/. If you find anything to be unclear or ambiguous, please [open a ticket](https://github.com/mtgjson/mtgjson-website/issues) on our documentation repository so we can address your concern immediately.

We fully rebuild our API data once a week (on Monday afternoons) and our price dataset once a day. You can poll the [Meta.json](https://mtgjson.com/api/v5/Meta.json) file to see when our data was last updated.

### For those who want to build MTGJSON locally
Most of our users shouldn't have a need to build MTGJSON locally. However, there are always exceptions and we aren't ones to judge.
#### Build Box
While MTGJSON will work on Windows, Mac, and Linux, we prefer working within the Linux environment for lower overheads and less manual dependency management.

- For **Linux** based build boxes (we recommend Ubuntu 20.04), your build box should have at least 2 cores and 4 GiB of RAM available.
- For **Mac** based build boxes, your build box should have at least 2 cores and 4 GiB of RAM available.
- For **Windows** based build boxes, your build box should have at least 4 cores and 8 GiB of RAM available.

#### Install Python3
MTGJSON is built on and tested against a wide range of Python3 versions. Currently, we maintain support for the following versions:
- Python 3.8
- Python 3.9
- Python 3.10
- Python 3.11
- Python 3.12

#### Install MTGJSON
##### Local Installation
`python3 -m pip install /path/to/mtgjson5/`
##### PyPi Repository
We intend to put MTGJSON5 on the pip package archive in the near future, once the first set of revisions takes place.

#### Using MTGJSON
A fully up-to-date help menu can be achieved via `python3 -m mtgjson5 -h`, but for your convenience here is a recent rundown:
```
usage: mtgjson5 [-h] [-s [SET [SET ...]] | -a] [-c] [-x] [-z] [-p]
                [-SS [SET [SET ...]]] [-PB] [-R] [-NA]

optional arguments:
  -h, --help            show this help message and exit
  -s [SET [SET ...]], --sets [SET [SET ...]]
                        Set(s) to build, using Scryfall set code notation.
                        Non-existent sets silently ignored.
  -a, --all-sets        Build all possible sets, overriding the --sets option.
  -c, --full-build      Build new prices, MTGSQLive, and compiled outputs like
                        AllPrintings.
  -x, --resume-build    While determining what sets to build, ignore
                        individual set files found in the output directory.
  -z, --compress        Compress the output folder's contents for
                        distribution.
  -p, --pretty          When dumping JSON files, prettify the contents instead
                        of minifying them.
  -SS [SET [SET ...]], --skip-sets [SET [SET ...]]
                        Purposely exclude sets from the build that may have
                        been set using --sets or --all-sets.

mtgjson maintainer arguments:
  -PB, --price-build    Build updated pricing data then exit.
  -R, --referrals       Create and maintain a referral map for referral
                        linkages.
  -NA, --no-alerts      Prevent push notifications from sending when property
                        keys are defined.
```

#### MTGJSON Environment Variables
Due to how the new system is built, a few advanced values can be set by the user in the shell environment.
- `MTGJSON5_DEBUG` When set to 1 or true, additional logging will be dumped to the output files
- `MTGJSON5_OUTPUT_PATH` When set, MTGJSON will dump all outputs to a specific directory
    - Ex:  `MTGJSON5_OUTPUT_PATH=~/Desktop` will dump database files to `/home/USER/Desktop/mtgjson_build_5XXX` and log files to `/home/USER/Desktop/logs`

## Licensing
MTGJSON is a freely available product under the [MIT License](https://github.com/mtgjson/mtgjson/blob/master/LICENSE.txt), allowing our users to enjoy Magic: the Gathering data free of charge, in perpetuity.


See https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md


# Cloud Foundry PHP Buildpack

[![CF Slack](https://www.google.com/s2/favicons?domain=www.slack.com) Join us on Slack](https://cloudfoundry.slack.com/messages/buildpacks/)

A buildpack to deploy PHP applications to Cloud Foundry based systems, such as a [cloud provider](https://www.cloudfoundry.org/learn/certified-providers/) or your own instance.

### Buildpack User Documentation

Official buildpack documentation can be found here: [php buildpack docs](http://docs.cloudfoundry.org/buildpacks/php/index.html).

### Building the Buildpack

#### Option 1: Using the `package.sh` script
1. Run `./scripts/package.sh [ --uncached | --cached ] [ --stack=STACK ]`

This requires that you have `docker` installed on your local machine, as it
will run packaging setup within a `ruby` image.

#### Option 2: Manually use the `buildpack-packager`
1. Make sure you have fetched submodules

   ```bash
   git submodule update --init
   ```

1. Check out a tagged release. It is not recommended to bundle buildpacks based on master or develop as these are moving targets.

   ```bash
   git checkout v4.4.2  # or whatever version you want, see releases page for available versions
   ```

1. Get latest buildpack dependencies, this will require having Ruby 3.0 or running in a Ruby 3.0 container image

   ```shell
   BUNDLE_GEMFILE=cf.Gemfile bundle
   ```

1. Build the buildpack. Please note that the PHP buildpack still uses the older Ruby based buildpack packager. This is different than most of the other buildpacks which use a newer Golang based buildpack packager. You must use the Ruby based buildpack packager with the PHP buildpack.

   ```shell
   BUNDLE_GEMFILE=cf.Gemfile bundle exec buildpack-packager [ --uncached | --cached ] [ --any-stack | --stack=STACK ]
   ```

1. Use in Cloud Foundry

    Upload the buildpack to your Cloud Foundry instance and optionally specify it by name

    ```bash
    cf create-buildpack custom_php_buildpack php_buildpack-cached-custom.zip 1
    cf push my_app -b custom_php_buildpack
    ```

### Contributing
Find our guidelines [here](https://github.com/cloudfoundry/php-buildpack/blob/develop/CONTRIBUTING.md).

### Integration Tests
Buildpacks use the [Cutlass](https://github.com/cloudfoundry/libbuildpack/tree/master/cutlass) framework for running integration tests.

To run integration tests, run the following command:

```
./scripts/integration.sh
```

### Unit Tests

To run unit tests, run the following command:

```bash
./scripts/unit
```

### Requirements
 1. [PyEnv] - This will allow you to easily install Python 2.6.6, which is the same version available through the staging environment of CloudFoundry.
 1. [virtualenv] & [pip] - The buildpack uses virtualenv and pip to setup the [required packages].  These are used by the unit test and not required by the buildpack itself.

### Setup
```bash
git clone https://github.com/cloudfoundry/php-buildpack
cd php-buildpack
python -V  # should report 2.6.6, if not fix PyEnv before creating the virtualenv
virtualenv `pwd`/env
. ./env/bin/activate
pip install -r requirements.txt
```

### Project Structure

The project is broken down into the following directories:

  - `bin` contains executable scripts, including `compile`, `release` and `detect`
  - `defaults` contains the default configuration
  - `docs` contains project documentation
  - `extensions` contains non-core extensions
  - `env` virtualenv environment
  - `lib` contains core extensions, helper code and the buildpack utils
  - `scripts` contains the Python scripts that run on compile, release and detect
  - `tests` contains test scripts and test data
  - `run_tests.sh` a convenience script for running the full suite of tests

### Understanding the Buildpack

The easiest way to understand the buildpack is to trace the flow of the scripts.  The buildpack system calls the `compile`, `release` and `detect` scripts provided by the buildpack.  These are located under the `bin` directory and are generic.  They simply redirect to the corresponding Python script under the `scripts` directory.

Of these, the `detect` and `release` scripts are straightforward, providing the minimal functionality required by a buildpack.  The `compile` script is more complicated but works like this.

  - load configuration
  - setup the `WEBDIR` directory
  - install the buildpack utils and the core extensions (HTTPD, Nginx & PHP)
  - install other extensions
  - install the `rewrite` and `start` scripts
  - setup the runtime environment and process manager
  - generate a startup.sh script

### Extensions

The buildpack relies heavily on extensions.  An extension is simply a set of Python methods that will get called at various times during the staging process.

Included non-core extensions:
- [`composer`](extensions/composer) - [Downloads, installs and runs Composer](http://docs.cloudfoundry.org/buildpacks/php/gsg-php-composer.html)
- [`dynatrace`](extensions/dynatrace) - Downloads and configures Dynatrace OneAgent
  - Looks for a bound service with name `dynatrace` and value `credentials` with sub-keys
    - `apiurl`
    - `environmentid`
    - `apitoken`
- [`newrelic`](extensions/newrelic) - [Downloads, installs and configures the NewRelic agent for PHP](http://docs.cloudfoundry.org/buildpacks/php/gsg-php-newrelic.html)
- [`sessions`](extensions/sessions) - [Configures PHP to store session information in a bound Redis or Memcached service instance](http://docs.cloudfoundry.org/buildpacks/php/gsg-php-sessions.html)

### Adding extensions

In general, you shouldn't need to modify the buildpack itself.  Instead creating your own extension should be the way to go.

To create an extension, simply create a folder.  The name of the folder will be the name of the extension.  Inside that folder, create a file called `extension.py`. That file will contain your code.  Inside that file, put your extension methods and any additional required code.

It's not necessary to fork the buildpack to add extensions for your app. The buildpack will notice and use extensions if you place them in a `.extensions` folder at your application root. See the [extensions directory in the this example](./fixtures/custom_extension/.extensions/phpmyadmin/extension.py) for a sample.

#### Methods

Here is an explanation of the methods offered to an extension developer.  All of them are optional and if a method is not implemented, it is simply skipped.

```python
def configure(ctx):
    pass
```

The `configure` method gives extension authors a chance to adjust the configuration of the buildpack prior to *any* extensions running.  The method is called very early on in the lifecycle of the buildpack, so keep this in mind when using this method.  The purpose of this method is to allow an extension author the opportunity to modify the configuration for PHP, the web server or another extension prior to those components being installed.

An example of when to use this method would be to adjust the list of PHP extensions that are going to be installed.

The method takes one argument, which is the buildpack context.  You can edit the context to update the state of the buildpack.  Return value is ignore / not necessary.

```python
def preprocess_commands(ctx):
    return ()
```

The `preprocess_commands` method gives extension authors the ability to contribute a list of commands that should be run prior to the services.  These commands are run in the execution environment, not the staging environment and should execute and complete quickly.  The purpose of these commands is to give extension authors the chance to run any last-minute code to adjust to the environment.

As an example, this is used by the core extensions rewrite configuration files with information that is specific to the runtime environment.

The method takes the context as an argument and should return a tuple of tuples (i.e. list of commands to run).

```python
def service_commands(ctx):
    return {}
```

The `service_commands` method gives extension authors the ability to contribute a set of services that need to be run.  These commands are run and should continue to run.  If any service exits, the process manager will halt all of the other services and the application will be restarted by Cloud Foundry.

The method takes the context as an argument and should return a dictionary of services to run.  The key should be the service name and the value should be a tuple which is the command and arguments.

```python
def service_environment(ctx):
    return {}
```

The `service_environment` method gives extension authors the ability to contribute environment variables that will be set and available to the services.

The method takes the buildpack context as its argument and should return a dictionary of the environment variables to be added to the environment where services (see `service_commands`) are executed.

The key should be the variable name and the value should be the value.  The value can either be a string, in which case the environment variable will be set with the value of the string or it can be a list.

If it's a list, the contents will be combined into a string and separated by the path separation character (i.e. ':' on Unix / Linux or ';' on Windows).  Keys that are set multiple times by the same or different extensions are automatically combined into one environment variable using the same path separation character.  This is helpful when two extensions both want to contribute to the same variable, for example LD_LIBRARY_PATH.

Please note that environment variables are not evaluated as they are set.  This would not work because they are set in the staging environment which is different than the execution environment.  This means you cannot do things like `PATH=$PATH:/new/path` or `NEWPATH=$HOME/some/path`.  To work around this, the buildpack will rewrite the environment variable file before it's processed.  This process will replace any `@<env-var>` markers with the value of the environment variable from the execution environment.  Thus if you do `PATH=@PATH:/new/path` or `NEWPATH=@HOME/some/path`, the service end up with a correctly set `PATH` or `NEWPATH`.

```python
def compile(install):
    return 0
```

The `compile` method is the main method and where extension authors should perform the bulk of their logic.  This method is called  by the buildpack while it's installing extensions.

The method is given one argument which is an Installer builder object.  The object can be used to install packages, configuration files or access the context (for examples of all this, see the core extensions like [HTTPD], [Nginx], [PHP], [Dynatrace] and [NewRelic]).  The method should return 0 when successful or any other number when it fails.  Optionally, the extension can raise an exception.  This will also signal a failure and it can provide more details about why something failed.

#### Method Order

It is sometimes useful to know what order the buildpack will use to call the methods in an extension.  They are called in the following order.

1. `configure`
2. `compile`
3. `service_environment`
4. `service_commands`
5. `preprocess_commands`

#### Example

Here is an example extension.  While technically correct, it doesn't actually do anything.

Here's the directory.

```bash
$ ls -lRh
total 0
drwxr-xr-x  3 daniel  staff   102B Mar  3 10:57 testextn

./testextn:
total 8
-rw-r--r--  1 daniel  staff   321B Mar  3 11:03 extension.py
```

Here's the code.

```python
import logging

_log = logging.getLogger('textextn')

# Extension Methods
def configure(ctx):
    pass

def preprocess_commands(ctx):
    return ()

def service_commands(ctx):
    return {}

def service_environment(ctx):
    return {}

def compile(install):
    return 0
```

#### Tips

 1. To be consistent with the rest of the buildpack, extensions should import and use the standard logging module.  This will allow extension output to be incorporated into the output for the rest of the buildpack.
 1. The buildpack will run every extension that is included with the buildpack and the application.  There is no mechanism to disable specific extensions.  Thus, when you write an extension, you should make some way for the user to enable / disable it's functionality.  See the [NewRelic] extension for an example of this.
 1. If an extension requires configuration, it should be included with the extension.  The `defaults/options.json` file is for the buildpack and its core extensions.  See the [NewRelic] buildpack for an example of this.
 1. Extensions should have their own test module.  This generally takes the form `tests/test_<extension_name>.py`.
 1. Run [bosh-lite].  It'll speed up testing and allow you to inspect the environment manually, if needed.
 1. Run a local web server for your binaries.  It'll seriously speed up download times.
 1. Test, test and test again.  Create unit and integration tests for your code and extensions.  This gives you quick and accurate feedback on your code.  It also makes it easier for you to make changes in the future and be confident that you're not breaking stuff.
 1. Check your code with flake8.  This linting tool can help to detect problems quickly.

[PyEnv]:https://github.com/yyuu/pyenv
[virtualenv]:http://www.virtualenv.org/en/latest/
[pip]:http://www.pip-installer.org/en/latest/
[required packages]:https://github.com/cloudfoundry/php-buildpack/blob/master/requirements.txt
[bosh-lite]:https://github.com/cloudfoundry/bosh-lite
[HTTPD]:https://github.com/cloudfoundry/php-buildpack/tree/master/lib/httpd
[Nginx]:https://github.com/cloudfoundry/php-buildpack/tree/master/lib/nginx
[PHP]:https://github.com/cloudfoundry/php-buildpack/tree/master/lib/php
[Dynatrace]:https://github.com/cloudfoundry/php-buildpack/tree/master/extensions/dynatrace
[NewRelic]:https://github.com/cloudfoundry/php-buildpack/tree/master/extensions/newrelic
[unit tests]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/development.md#testing

### Help and Support

Join the #buildpacks channel in our [Slack community](http://slack.cloudfoundry.org/)

### Reporting Issues

This project is managed through GitHub.  If you encounter any issues, bug or problems with the buildpack please open an issue.

### Active Development

The project backlog is on [Pivotal Tracker](https://www.pivotaltracker.com/projects/1042066)

[Configuration Options]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/config.md
[Development]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/development.md
[Troubleshooting]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/troubleshooting.md
[Usage]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/usage.md
[Binaries]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/binaries.md
[php-info]:https://github.com/dmikusa-pivotal/cf-ex-php-info
[PHPMyAdmin]:https://github.com/dmikusa-pivotal/cf-ex-phpmyadmin
[PHPPgAdmin]:https://github.com/dmikusa-pivotal/cf-ex-phppgadmin
[Wordpress]:https://github.com/dmikusa-pivotal/cf-ex-worpress
[Drupal]:https://github.com/dmikusa-pivotal/cf-ex-drupal
[CodeIgniter]:https://github.com/dmikusa-pivotal/cf-ex-code-igniter
[Stand Alone]:https://github.com/dmikusa-pivotal/cf-ex-stand-alone
[pgbouncer]:https://github.com/dmikusa-pivotal/cf-ex-pgbouncer
[Apache License]:http://www.apache.org/licenses/LICENSE-2.0
[vcap-dev]:https://groups.google.com/a/cloudfoundry.org/forum/#!forum/vcap-dev
[support forums]:http://support.run.pivotal.io/home
[Composer support]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/composer.md
["offline" mode]:https://github.com/cloudfoundry/php-buildpack/blob/master/docs/binaries.md#bundling-binaries-with-the-build-pack
[phalcon]:https://github.com/dmikusa-pivotal/cf-ex-phalcon
[Phalcon]:http://phalconphp.com/en/
[composer]:https://github.com/dmikusa-pivotal/cf-ex-composer
[Proxy Support]:http://docs.cloudfoundry.org/buildpacks/proxy-usage.html


![A toolbox for processing and analysing air traffic data](./docs/_static/logo/logo_full.png)

[![Documentation Status](https://github.com/xoolive/traffic/workflows/docs/badge.svg)](https://traffic-viz.github.io/)
[![tests](https://github.com/xoolive/traffic/actions/workflows/run-tests.yml/badge.svg?branch=master&event=push)](https://github.com/xoolive/traffic/actions/workflows/run-tests.yml)
[![Code Coverage](https://img.shields.io/codecov/c/github/xoolive/traffic.svg)](https://codecov.io/gh/xoolive/traffic)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue.svg)](https://mypy.readthedocs.io/)
![License](https://img.shields.io/pypi/l/traffic.svg)
[![Join the chat at https://gitter.im/xoolive/traffic](https://badges.gitter.im/xoolive/traffic.svg)](https://gitter.im/xoolive/traffic?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
![PyPI version](https://img.shields.io/pypi/v/traffic)
[![PyPI downloads](https://img.shields.io/pypi/dm/traffic)](https://pypi.org/project/traffic)
![Conda version](https://img.shields.io/conda/vn/conda-forge/traffic)
[![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/traffic.svg)](https://anaconda.org/conda-forge/traffic)
[![JOSS paper](http://joss.theoj.org/papers/10.21105/joss.01518/status.svg)](https://doi.org/10.21105/joss.01518)

The traffic library helps to work with common sources of air traffic data.

Its main purpose is to provide data analysis methods commonly applied to trajectories and airspaces. When a specific function is not provided, the access to the underlying structure is direct, through an attribute pointing to a pandas dataframe.

The library also offers facilities to parse and/or access traffic data from open sources of ADS-B traffic like the [OpenSky Network](https://opensky-network.org/) or Eurocontrol DDR files. It is designed to be easily extendable to other sources of data.

Static visualization (images) exports are accessible via Matplotlib/Cartopy. More dynamic visualization frameworks are easily accessible in Jupyter environments with [ipyleaflet](http://ipyleaflet.readthedocs.io/) and [altair](http://altair-viz.github.io/); or through exports to other formats, including CesiumJS or Google Earth.

## Installation

**Full installation instructions are to be found in the [documentation](https://traffic-viz.github.io/installation.html).**

- If you are not familiar/comfortable with your Python environment, please install the latest `traffic` release in a new, fresh conda environment.

  ```sh
  conda create -n traffic -c conda-forge python=3.10 traffic
  ```

- Adjust the Python version you need (>=3.9) and append packages you need for working efficiently, such as Jupyter Lab, xarray, PyTorch or more.
- Then activate the environment every time you need to use the `traffic` library:

  ```sh
  conda activate traffic
  ```

**Warning!** Dependency resolution may be tricky, esp. if you use an old conda environment where you overwrote `conda` libraries with `pip` installs. **Please only report installation issues in new, fresh conda environments.**

If conda fails to resolve an environment in a reasonable time, consider using a [Docker image](https://traffic-viz.github.io/user_guide/docker.html) with a working installation.

For troubleshooting, refer to the appropriate [documentation section](https://traffic-viz.github.io/troubleshooting/installation.html).

## Credits

[![JOSS
badge](http://joss.theoj.org/papers/10.21105/joss.01518/status.svg)](https://doi.org/10.21105/joss.01518)

- Like [other researchers before](https://scholar.google.com/scholar?cites=18420568209924139259&scisbd=1), if you find this project useful for your research and use it in an academic work, you may cite it as:

  ```bibtex
  @article{olive2019traffic,
      author={Xavier {Olive}},
      journal={Journal of Open Source Software},
      title={traffic, a toolbox for processing and analysing air traffic data},
      year={2019},
      volume={4},
      pages={1518},
      doi={10.21105/joss.01518},
      issn={2475-9066},
  }
  ```

- Additionally, you may consider adding a star to the repository. This token of appreciation is often interpreted as positive feedback and improves the visibility of the library.

## Documentation

[![Documentation Status](https://github.com/xoolive/traffic/workflows/docs/badge.svg)](https://traffic-viz.github.io/)
[![Join the chat at https://gitter.im/xoolive/traffic](https://badges.gitter.im/xoolive/traffic.svg)](https://gitter.im/xoolive/traffic?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Documentation available at <https://traffic-viz.github.io/>
Join the Gitter chat for assistance: https://gitter.im/xoolive/traffic

## Tests and code quality

[![tests](https://github.com/xoolive/traffic/actions/workflows/run-tests.yml/badge.svg?branch=master&event=push)](https://github.com/xoolive/traffic/actions/workflows/run-tests.yml)
[![Code Coverage](https://img.shields.io/codecov/c/github/xoolive/traffic.svg)](https://codecov.io/gh/xoolive/traffic)
[![Codacy Badge](https://img.shields.io/codacy/grade/eea673ed15304f1b93490726295d6de0)](https://www.codacy.com/manual/xoolive/traffic)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-blue.svg)](https://mypy.readthedocs.io/)

Unit and non-regression tests are written in the `tests/` directory. You may run `pytest` from the root directory.

Tests are checked on [Github Actions](https://github.com/xoolive/traffic/actions/workflows/run-tests.yml) platform upon each commit. Latest status and coverage are displayed with standard badges hereabove.

In addition to unit tests, code is checked against:

- linting and formatting with [ruff](https://beta.ruff.rs/docs/);
- static typing with [mypy](https://mypy.readthedocs.io/)

[pre-commit](https://pre-commit.com/) hooks are available in the repository.

## Feedback and contribution

Any input, feedback, bug report or contribution is welcome.

- Should you encounter any [issue](https://github.com/xoolive/traffic/issues/new), you may want to file it in the [issue](https://github.com/xoolive/traffic/issues/new) section of this repository.
- If you intend to [contribute to traffic](https://traffic-viz.github.io/installation.html#contribute-to-traffic) or file a pull request, the best way to ensure continuous integration does not break is to reproduce an environment with the same exact versions of all dependency libraries. Please follow the [appropriate section](https://traffic-viz.github.io/installation.html#contribute-to-traffic) in the documentation.

  Let us know what you want to do just in case we're already working on an implementation of something similar. This way we can avoid any needless duplication of effort. Also, please don't forget to add tests for any new functions.


|pypi| |test| |coverage|

=======
pkgcore
=======

pkgcore is a framework for package management; via the appropriate class
plugins, the design should allow for any underlying repository/config/format to
be used; slackware's tgzs being exempted due to lack of any real metadata, and
autopackage format being exempted due to the fact they effectively embed the
manager in each package (pkgcore *does* require being able to treat the pkg as
data, instead of autopackage's method of handing resolution/all manager ops off
to the package script).

Tools
=====

**pclean**: clean distfiles, binpkgs, and builds dirs

**pclonecache**: clone a repository cache

**pconfig**: query configuration info

**pebuild**: low-level ebuild operations, go through phases manually

**pinspect**: inspect repository related info

**pmaint**: repository maintenance (syncing, copying...)

**pmerge**: dependency resolution, fetching, (un)merging, etc.

**pquery**: query repository info, revdeps, pkg search, vdb search, etc.

Requirements
============

At least python version 3.10, and snakeoil_ — a utility library split out of
pkgcore for others to use.

Installing
==========

Installing latest pypi release::

    pip install pkgcore

Installing from git::

    pip install https://github.com/pkgcore/pkgcore/archive/master.tar.gz

Installing from a tarball::

    pip install .

Developing
==========

Installing the dependencies for testing, formatting, and documentation building
into an editable environment::

    make dev-environment

Tests
=====

Standard python test runners can be used, for example:

    pytest -v

A tox config is provided so the testsuite can be run in a
virtualenv setup against all supported python versions. To run tests for all
environments just execute **tox** in the root directory of a repo or unpacked
tarball. Otherwise, for a specific python version execute something similar to
the following::

    tox -e py311

Docs
====

Documentation is available on Github_.

Contact
=======

For bugs and feature requests please create an issue_.

.. _Github: https://pkgcore.github.io/pkgcore/
.. _snakeoil: https://github.com/pkgcore/snakeoil
.. _issue: https://github.com/pkgcore/pkgcore/issues

.. |pypi| image:: https://img.shields.io/pypi/v/pkgcore.svg
    :target: https://pypi.python.org/pypi/pkgcore
.. |test| image:: https://github.com/pkgcore/pkgcore/workflows/test/badge.svg
    :target: https://github.com/pkgcore/pkgcore/actions?query=workflow%3A%22test%22
.. |coverage| image:: https://codecov.io/gh/pkgcore/pkgcore/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/pkgcore/pkgcore


.. figure:: https://raw.githubusercontent.com/jmchilton/planemo/master/docs/planemo_logo.png
   :alt: Planemo Logo
   :align: center
   :figwidth: 100%
   :target: https://github.com/galaxyproject/planemo

Command-line utilities to assist in developing Galaxy_ and `Common Workflow Language`_ artifacts -
including tools, workflows, and training materials.

.. image:: https://readthedocs.org/projects/planemo/badge/?version=latest
   :target: http://planemo.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status

.. image:: https://badge.fury.io/py/planemo.svg
   :target: https://pypi.python.org/pypi/planemo/
   :alt: Planemo on the Python Package Index (PyPI)

.. image:: https://github.com/galaxyproject/planemo/workflows/Python%20CI/badge.svg
   :target: https://github.com/galaxyproject/planemo/actions?query=workflow%3A%22Python+CI%22

* Free software: MIT License
* Documentation: https://planemo.readthedocs.io.
* Code: https://github.com/galaxyproject/planemo

Quick Start
-----------

-----------------
Obtaining
-----------------

For a traditional Python installation of Planemo, first set up a virtualenv
for ``planemo`` (this example creates a new one in ``.venv``) containing
Python 3.7 or newer and then install with ``pip``. Planemo must be installed
with pip 7.0 or newer.

::

    $ virtualenv .venv; . .venv/bin/activate
    $ pip install "pip>=7" # Upgrade pip if needed.
    $ pip install planemo

For information on updating Planemo, installing the latest development release,
or installing Planemo via `Bioconda <https://github.com/bioconda/bioconda-recipes>`__
- checkout the `installation <http://planemo.readthedocs.io/en/latest/installation.html>`__
documentation.

Planemo is also available as a `virtual appliance
<https://planemo.readthedocs.io/en/latest/appliance.html>`_ bundled
with a preconfigured Galaxy server and set up for Galaxy_ and
`Common Workflow Language`_ tool development.
You can choose from open virtualization format (OVA_, .ova) or Docker_
appliances.

-----------------
Basics - Galaxy
-----------------

This quick start will assume you have a directory with one or more Galaxy
tool XML files. If no such directory is available, one can be quickly created for
demonstrating ``planemo`` as follows ``project_init --template=demo mytools; cd mytools``.

Planemo can check tool XML files for common problems and best
practices using the ``lint`` `command <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
(also aliased as ``l``).

::

    $ planemo lint

Like many ``planemo`` commands - by default this will search the
current directory and use all tool files it finds. It can be explicitly
passed a path to tool files or a directory of tool files.

::

    $ planemo l randomlines.xml

The ``lint`` command takes in additional options related to
reporting levels, exit code, etc. These options are described
in the `docs <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
or (like with all commands) can be accessed by passing ``--help`` to it.

::

    $ planemo l --help
    Usage: planemo lint [OPTIONS] TOOL_PATH

Once tools are syntactically correct - it is time to test. The ``test``
`command <http://planemo.readthedocs.org/en/latest/commands.html#test-command>`__
can be used to test a tool or a directory of tools.

::

	$ planemo test --galaxy_root=../galaxy randomlines.xml

If no ``--galaxy_root`` is defined, Planemo will download and configure
a disposable Galaxy instance for testing.

Planemo will create a HTML output report in the current directory named
``tool_test_output.html`` (override with ``--test_output``). See an
`example <http://galaxyproject.github.io/planemo/tool_test_viewer.html?test_data_url=https://gist.githubusercontent.com/jmchilton/9d4351c9545d34209904/raw/9ed285d3cf98e435fc4a743320363275949ad63c/index>`_
of such a report for Tophat.

Once tools have been linted and tested - the tools can be viewed in a
Galaxy interface using the ``serve`` (``s``) `command
<http://planemo.readthedocs.org/en/latest/commands.html#serve-command>`__.

::

	$ planemo serve

Like ``test``, ``serve`` requires a Galaxy root and one can be
explicitly specified with ``--galaxy_root`` or installed dynamically
with ``--install_galaxy``.

For more information on building Galaxy tools in general please check out
`Building Galaxy Tools Using Planemo`_.

For more information on developing Galaxy workflows with Planemo checkout
`best practices for Galaxy Workflows`_ and the description of Planemo's
`test format`_. For information on developing Galaxy training materials
checkout the `contributing documentation <https://training.galaxyproject.org/training-material/topics/contributing/>`__
on training.galaxyproject.org.

----------------------------------
Basics - Common Workflow Language
----------------------------------

This quick start will assume you have a directory with one or more `Common Workflow
Language`_ YAML files. If no such directory is available, one can be quickly created for
demonstrating ``planemo`` as follows ``planemo project_init --template=seqtk_complete_cwl mytools; cd mytools``.

Planemo can check tools YAML files for common problems and best
practices using the ``lint`` `command <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
(also aliased as ``l``).

::

    $ planemo lint

Like many ``planemo`` commands - by default this will search the
current directory and use all tool files it finds. It can be explicitly
passed a path to tool files or a directory of tool files.

::

    $ planemo l seqtk_seq.cwl

The ``lint`` command takes in additional options related to
reporting levels, exit code, etc. These options are described
in the `docs <http://planemo.readthedocs.org/en/latest/commands.html#lint-command>`_
or (like with all commands) can be accessed by passing ``--help`` to it.

::

    $ planemo l --help
    Usage: planemo lint [OPTIONS] TOOL_PATH

Once tools are syntactically correct - it is time to test. The ``test``
`command <http://planemo.readthedocs.org/en/latest/commands.html#test-command>`__
can be used to test a CWL tool, workflow, or a directories thereof.

::

  $ planemo test --engine cwltool seqtk_seq.cwl

Planemo will create a HTML output report in the current directory named
``tool_test_output.html``. Check out the file ``seqtk_seq_tests.yml`` for
an example of Planemo test for a CWL tool. A test consists of any number of
jobs (with input descriptions) and corresponding output assertions.

Checkout the `Commmon Workflow User Guide`_ for more information on developing
CWL tools in general and  `Building Common Workflow Language Tools`_ for more
information on using Planemo to develop CWL tools.

---------
Tool Shed
---------

Planemo can help you publish tools to the Galaxy Tool Shed.
Check out `Publishing to the Tool Shed`_ for more information.

------
Conda
------

Planemo can help develop tools and Conda packages in unison.
Check out the `Galaxy <http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-conda>`__ or `CWL
<http://planemo.readthedocs.io/en/latest/writing_advanced_cwl.html#dependencies-and-conda-cwl>`__ version of the "Dependencies and Conda" tutorial
for more information.

-----------------------
Docker and Containers
-----------------------

Planemo can help develop tools that run in "Best Practice" containers for
scientific workflows. Check out the `Galaxy <http://planemo.readthedocs.io/en/latest/writing_advanced.html#dependencies-and-containers>`__ or `CWL
<http://planemo.readthedocs.io/en/latest/writing_advanced_cwl.html#dependencies-and-containers-cwl>`__ version of the "Dependencies and Containers" tutorial for more information.

.. _Galaxy: http://galaxyproject.org/
.. _GitHub: https://github.com/
.. _Conda: http://conda.pydata.org/
.. _Docker: https://www.docker.com/
.. _Vagrant: https://www.vagrantup.com/
.. _Travis CI: http://travis-ci.org/
.. _`tools-devteam`: https://github.com/galaxyproject/tools-devteam
.. _`tools-iuc`: https://github.com/galaxyproject/tools-iuc
.. _Building Galaxy Tools Using Planemo: http://planemo.readthedocs.io/en/latest/writing_standalone.html
.. _Publishing to the Tool Shed: http://planemo.readthedocs.org/en/latest/publishing.html
.. _Common Workflow Language: https://www.commonwl.org/
.. _Commmon Workflow User Guide: http://www.commonwl.org/user_guide/
.. _Building Common Workflow Language Tools: http://planemo.readthedocs.io/en/latest/writing_cwl_standalone.html
.. _OVA: https://en.wikipedia.org/wiki/Open_Virtualization_Format
.. _test format: https://planemo.readthedocs.io/en/latest/test_format.html
.. _best practices for Galaxy Workflows: https://planemo.readthedocs.io/en/latest/best_practices_workflows.html


# Tibanna

[![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/downloads/release/python-380/) [![Build Status](https://travis-ci.org/4dn-dcic/tibanna.svg?branch=master)](https://travis-ci.org/4dn-dcic/tibanna) [![Code Quality](https://api.codacy.com/project/badge/Grade/d2946b5bc0704e5c9a4893426a7e0314)](https://www.codacy.com/app/4dn/tibanna?utm_source=github.com&utm_medium=referral&utm_content=4dn-dcic/tibanna&utm_campaign=Badge_Grade) [![Test Coverage](https://api.codacy.com/project/badge/Coverage/d2946b5bc0704e5c9a4893426a7e0314)](https://www.codacy.com/app/4dn/tibanna?utm_source=github.com&utm_medium=referral&utm_content=4dn-dcic/tibanna&utm_campaign=Badge_Coverage) [![Documentation Status](https://readthedocs.org/projects/tibanna/badge/?version=latest)](https://tibanna.readthedocs.io/en/latest/?badge=latest)

***

Tibanna runs portable pipelines (in CWL/WDL/Snakemake/shell) on the AWS Cloud.

Install Tibanna.
```bash
pip install tibanna
```

Use CLI to set up the cloud component and run workflow.
```bash
# Deploy Unicorn to the Cloud (Unicorn = serverless scheduler/resource allocator).
tibanna deploy_unicorn --usergroup=mygroup

# Run CWL/WDL workflow on the Cloud.
tibanna run_workflow --input-json=myrun.json
```

Alternatively, use Python API.

```python
from tibanna.core import API

# Deploy Unicorn to the Cloud.
API().deploy_unicorn(usergroup='mygroup')

# Run CWL/WDL workflow on the Cloud.
API().run_workflow(input_json='myrun.json')
```

---
Note: Starting `0.8.2`, Tibanna supports local CWL/WDL files as well as shell commands and Snakemake workflows.

Note 2: As of Tibanna version `2.0.0`, Python 3.7 (and lower) is no longer supported. Please switch to Python 3.11! Python 3.8 is also supported as a fallback, but please prefer 3.11 if you can.

Note 3: Starting `0.8.0`, one no longer needs to `git clone` the Tibanna repo.
* Please switch from `invoke <command>` to `tibanna <command>`!
* We also renovated the Python API as an inheritable class to allow development around tibanna.

For more details, see Tibanna [**Documentation**](http://tibanna.readthedocs.io/en/latest).
* Also check out our [**paper in _Bioinformatics_**](https://doi.org/10.1093/bioinformatics/btz379).
* A preprint can also be found on [**biorxiv**](https://www.biorxiv.org/content/10.1101/440974v3).


[![Build Status](https://travis-ci.com/NCI-GDC/gdcdictionary.svg?branch=master)](https://travis-ci.com/NCI-GDC/gdcdictionary)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commitlogoColor=white)](https://github.com/pre-commit/pre-commit)

---
# GDC Data Dictionary

The Genomic Data Commons’ (GDC) data dictionary provides the first
level of validation for all data stored in and generated by the
GDC. JSON schemas define all the individual entities (nodes) in the
GDC data model. Moreover, these schemas define all of the
relationships (links) between the nodes. Finally, the schemas define
the valid key-value pairs that can be used to describe the nodes.

- [GDC Data Dictionary](#gdc-data-dictionary)
  - [GDC Data Dictionary Structure](#gdc-data-dictionary-structure)
  - [Node Properties and Examples](#node-properties-and-examples)
  - [Dictionary Changes](#dictionary-changes)
    - [Breaking Changes](#breaking-changes)
    - [Entity Relation Additions](#entity-relation-additions)
    - [Schema Additions](#schema-additions)
    - [Cosmetic Corrections](#cosmetic-corrections)
    - [Testing](#testing)
    - [Versioning](#versioning)
  - [Gdcdatamodel2 auto generation](#Gdcdatamodel2-auto-generation)
  - [Setup pre-commit hook to check for secrets](#setup-pre-commit-hook-to-check-for-secrets)
  - [Contributing](#contributing)

## GDC Data Dictionary Structure

The GDC Data Model is covers all of the nodes within the GDC as well as the relationships between
the different types of nodes. All of the nodes in the data model are strongly typed and individually
defined for a specific data type. For example, submitted files can come in two different forms,
aligned or unaligned; within the model we have two separately defined nodes for
`Submitted Unaligned Reads` and `Submitted Aligned Reads`. Doing such allows for faster querying of
the data model as well as providing a clear and concise representation of the data in the GDC.

Beyond node type, there are also a number of GDC extensions used to further define the nodes within
the data model. Nodes are grouped up into categories that represent broad roles for the node such
as `analysis` or `biospecimen`. Additionally, nodes are defined within their `Program` or `Project`
and have descriptions of their use. All nodes also have a series of `systemProperties`; these
properties are those that will be automatically filled by the system unless otherwise defined by
the user.  These basic properties define the node itself but it still needs to be placed into the model.

The model itself is represented as a graph. Within the schema are defined `links`; these links
point from child to parent with Program being the root of the graph. The links also contain a
`backref` that allows for a parent to point to a child. Other features of the link include a
semantic `label` that describes the relationship between the two nodes, a `multiplicity` property
that describes the numeric relationship from the child to the parent, and a requirement property
to define whether a node must have that link. Taken all together the nodes and links create the
directed graph of the GDC Data Model.

## Node Properties and Examples

Each node contains a series of potential key-value pairs (`properties`) that can be used to
characterize the data they represent. Some properties are categorized as `required` or `preferred`.
If a submission lacks a required property, it cannot be accepted. Preferred properties can denote
two things: the property is being highlighted as it has become more desired by the community or
the property is being promoted to required. All properties not designated either `required` or
`preferred` are still sought by GDC, but submissions without them are allowed.

The properties have further validation through their entries. Legal values are defined in each
property. For the most part these are represented in the `enum` categories although some keys,
such as `submitter_id`, will allow any string value as a valid entry. Other numeric properties
can have maximum and minimum values to limit valid entries.  For examples of what a valid entry
would look like, each node has a mock submission located in the `examples/valid/` directory.

## Dictionary Changes

The following is an attempt to layout guidelines for the level of
impact of changes to the dictionary by categorizing them into
**Breaking Changes**, **Entity Relation Additions**, **Schema Additions**,
**Cosmetic Corrections**.

### Breaking Changes

Breaking changes are changes to the dictionary such that previously
allowable data is invalid against the new schema, e.g. a **removal** of
part of the dictionary.

N.B. That not all changes classified here as Breaking Changes are
promised to require a data migration.  It is possible that no data
exists in the GDC that is invalidated by the change, e.g. making a
field required that has never been left blank.  This should be
confirmed against the corpus of data and the userbase should be
notified of a break in backwards-compatibility.

**Breaking Changes include**:
- Renaming/removing anything that is not a description or comment
  - Removing an entity schema
  - Removing a property's allowed `type`
  - Removing a property's allowed `enum` value
  - Changing an entity's `category`
  - Changing an entity's `unique_keys`
  - Changing an entity's `links`, including `label`, `backref`
  - Removing a property from an entity schema
- Changing existence requirements
  - Adding a property to the `required` list
  - Changing link `required` from `false` to `true`
  - Changing link `multiplicity` from `one_to_many` or `many_to_one` to `one_to_one`
  - Changing link subgroup exclusivity from `false` to `true`

**Handling breaking changes**:

Sometimes it may be best to introduce necessary breaking changes
incrementally.  Given you have State A and State B, which are
incomatible, if you can create a State AB that is compatible with
both, you can upgrade to State AB without breaking changes, update
data to be compliant with State B, then upgrade to State B.

1. State A is deployed
2. Upgrade to State AB
3. Update data while State AB is deployed to be valid under State B
4. Upgrade to State B

An example could be: _Introduce required property `color`_:

1. Property `color` does not exist
2. Deploy schema that allows but does not require `color`
3. Add color to all records
4. Deploy schema that requires `color`

### Entity Relation Additions

Additions to the dictionary that create entities or add links between
entities should not be considered breaking changes, however, they
should be carefully considered in context of downstream effects.

**Entity Relation Additions include**:
- Adding a new entity schema
- Adding a new link between entities

**Entity Relation downstream effects**:
- The GDC will have to update the database schema
- Users should be notified of additions

### Schema Additions

The GDC is setup to allow strict additions to properties have minimal
impact on existing data.

**Schema Additions include**:
- New properties
- New allowed types for properties
- New allowed `enum` members for properties

**Schema Addition downstream effects**:
- Users should be notified of additions

### Cosmetic Corrections

Cosmetic corrections are changes that have little to no behavioral
effects.

**Cosmetic Corrections include**:
- Changes to terms
- Changes to documentation
- Schema formatting changes

**Schema Addition downstream effects**:
- No large impacts

### Testing

Commits will automagically be run on TravisCI when a Pull Request is opened.
If you would like to test locally they are run via [tox](https://tox.readthedocs.io/en/latest/)

### Versioning

The GDC Dictionary should
follow [Semantic Versioning](http://semver.org/) by updating the
line in setup.py file to `MAJOR.MINOR.PATCH` accordingly:

1. MAJOR: version when you make incompatible API changes: **Breaking Changes**
   - e.g. 1.2.4 -> 2.0.0
2. MINOR: version when you add functionality in a backwards-compatible manner: **Relationship Additions**, **Schema Additions**
   - e.g. 1.2.4 -> 1.3.0
3. PATCH: version when you make backwards-compatible bug fixes: **Cosmetic Corrections**
   - e.g. 1.2.4 -> 1.2.5

## Gdcdatamodel2 auto generation

Gdcdatamodel2 should be generated automatically on gitlab for each commit push.
The generated python artifact should be in
https://nexus.osdc.io/#browse/browse:pypi-snapshots:gdcdatamodel2

The gitlab pipline also automatically push a new branch to gdcdatamodel2 on github.

If you want, you can also manually run the pipeline to generate a new version.
1. Go to https://gitlab.datacommons.io/nci-gdc/development/gdcdictionary/-/pipelines/new
2. Select the branch/tag of gdcdictionary you want to use, default: develop
3. (optional) the generated version of gdcdatamodel2 should based on the branch/tag you
selected in previous step. But if you want to generate from a different branch, change
`GDCDICTIONARY_TARGET_VERSION_OVERRIDE` in the variables.
4. click `Run Pipeline` button.

## Setup pre-commit hook to check for secrets

We use [pre-commit](https://pre-commit.com/) to setup pre-commit hooks for this repo.
We use [detect-secrets](https://github.com/Yelp/detect-secrets) to search for secrets being committed into the repo.

To install the pre-commit hook, run
```
pre-commit install
```

To update the .secrets.baseline file run
```
detect-secrets scan --update .secrets.baseline
```

`.secrets.baseline` contains all the string that were caught by detect-secrets but are not stored in plain text. Audit the baseline to view the secrets .

```
detect-secrets audit .secrets.baseline
```

## Contributing

Read how to contribute [here](https://github.com/NCI-GDC/gdcdictionary/blob/develop/CONTRIBUTING.md).


|Build Status| |codecov| |PyPI|

Sphinx Substitution Extensions
==============================

Extensions for Sphinx which allow substitutions within code blocks.

.. contents::

Installation
------------

Sphinx Substitution Extensions is compatible with Sphinx 7.2.0+ using Python 3.10+.

.. code:: console

   $ pip install Sphinx-Substitution-Extensions

rST setup
---------

1. Add the following to ``conf.py`` to enable the extension:

.. code:: python

   extensions += ['sphinx_substitution_extensions']

2. Set the following variable in ``conf.py`` to define substitutions:

.. code:: python

   rst_prolog = """
   .. |release| replace:: 0.1
   .. |author| replace:: Eleanor
   """

This will replace ``|release|`` in the new directives with ``0.1``, and ``|author|`` with ``Eleanor``.

Using substitutions in rST documents
------------------------------------

``code-block``
~~~~~~~~~~~~~~

This adds a ``:substitutions:`` option to Sphinx's built-in `code-block`_ directive.

.. code:: rst

   .. code-block:: bash
      :substitutions:

      echo "|author| released version |release|"

``prompt``
~~~~~~~~~~

This adds a ``:substitutions:`` option to `sphinx-prompt`_.

.. code:: rst

   .. prompt:: bash
      :substitutions:

      echo "|author| released version |release|"

Inline ``:substitution-code:``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: rst

   :substitution-code:`echo "|author| released version |release|"`

``substitution-download``
~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: rst

   :substitution-download:`|author|'s manuscript <|author|_manuscript.txt>`

MyST Markdown setup
-------------------

1. Add the following to ``conf.py`` to enable the extension:

.. code:: python

   extensions += ['sphinx_substitution_extensions']

2. Set the following variables in ``conf.py`` to define substitutions:

.. code:: python

   myst_enable_extensions += ['substitution']
   myst_substitutions = {
         "release": "0.1",
         "author": "Eleanor",
   }

This will replace ``|release|`` in the new directives with ``0.1``, and ``|author|`` with ``Eleanor``.

Using substitutions in MyST Markdown
------------------------------------

``code-block``
~~~~~~~~~~~~~~

This adds a ``:substitutions:`` option to Sphinx's built-in `code-block`_ directive.

.. code:: markdown

   ```{code-block} bash
      :substitutions:

      echo "|author| released version |release|"
   ```

Credits
-------

``sphinx-prompt`` authors
~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks to `@sbrunner`_ and other contributors for `sphinx-prompt`_.
``substitution-prompt`` is based on `sphinx-prompt`_.

ClusterHQ Developers
~~~~~~~~~~~~~~~~~~~~

This package is largely inspired by code written for Flocker by ClusterHQ.
Developers of the relevant code include, at least, Jon Giddy and Tom Prince.

Contributing
------------

See `CONTRIBUTING.rst <./CONTRIBUTING.rst>`_.

.. |Build Status| image:: https://github.com/adamtheturtle/sphinx-substitution-extensions/workflows/CI/badge.svg
   :target: https://github.com/adamtheturtle/sphinx-substitution-extensions/actions
.. _sphinx-prompt: https://github.com/sbrunner/sphinx-prompt
.. _code-block: http://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-code-block
.. _@sbrunner: https://github.com/sbrunner
.. |codecov| image:: https://codecov.io/gh/adamtheturtle/sphinx-substitution-extensions/branch/main/graph/badge.svg
   :target: https://codecov.io/gh/adamtheturtle/sphinx-substitution-extensions
.. |PyPI| image:: https://badge.fury.io/py/Sphinx-Substitution-Extensions.svg
   :target: https://badge.fury.io/py/Sphinx-Substitution-Extensions


[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![Gitter chat](https://badges.gitter.im/cs3org/wopiserver.svg)](https://gitter.im/cs3org/wopiserver)
 [![Build Status](https://github.com/cs3org/wopiserver/actions/workflows/release.yml/badge.svg)](https://github.com/cs3org/wopiserver/actions)
 [![codecov](https://codecov.io/gh/cs3org/wopiserver/branch/master/graph/badge.svg)](https://codecov.io/gh/cs3org/wopiserver)
========

# WOPI Server

This service is part of the ScienceMesh Interoperability Platform ([IOP](https://developer.sciencemesh.io)) and implements a vendor-neutral application gateway compatible with the Web-application Open Platform Interface ([WOPI](https://docs.microsoft.com/en-us/microsoft-365/cloud-storage-partner-program/online)) specifications.

It enables ScienceMesh EFSS storages to integrate Office Online platforms including Microsoft Office Online and Collabora Online. In addition it implements a [bridge](src/bridge/readme.md) module with dedicated extensions to support apps like CodiMD and Etherpad.

Author: Giuseppe Lo Presti (@glpatcern)

Contributors (oldest contributions first):
- Michael DSilva (@madsi1m)
- Lovisa Lugnegaard (@LovisaLugnegard)
- Samuel Alfageme (@SamuAlfageme)
- Diogo Castro (@diocas)
- Ishank Arora (@ishank011)
- Willy Kloucek (@wkloucek)
- Gianmaria Del Monte (@gmgigi96)
- Klaas Freitag (@dragotin)
- Jörn Friedrich Dreyer (@butonic)
- Michael Barz (@micbar)
- Robert Kaussow (@xoxys)
- Javier Ferrer (@javfg)
- Vasco Guita (@vascoguita)
- Thomas Mueller (@deepdiver1975)
- Andre Duffeck (@aduffeck)

Initial revision: December 2016

First production version for CERNBox: September 2017 (presented at [oCCon17](https://occon17.owncloud.org) - [slides](https://www.slideshare.net/giuseppelopresti/collaborative-editing-and-more-in-cernbox))

Integration in the CS3 Organisation: April 2020

## Modules

* [core](src/core/readme.md)
* [bridge](src/bridge/readme.md)

## Changelog

[Available here](CHANGELOG.md)

## Compatibility

This WOPI server implements the required APIs to ensure full compatibility with Microsoft Office (as provided via the CSPP Terms), Collabora Online, and ONLYOFFICE.

## Unit testing

The `/test` folder contains some unit tests for the supported storage interfaces.
No tests are provided (yet) for the core WOPI server, though the test suite aims at covering all
storage access patterns used by the WOPI server.

By default, the local storage is tested, and the CI runs it as well (_TODO_ test against Reva in the CI).
To run the tests, either run `pytest` if available in your system, or execute the following:

1. Run all tests: `python3 test/test_storageiface.py [-v]`
2. Run only one test: `python3 test/test_storageiface.py [-v] TestStorage.<the test you would like to run>`

### Test against a Reva CS3 endpoint:

1. Clone reva (https://github.com/cs3org/reva)
2. Run Reva according to <https://reva.link/docs/tutorials/share-tutorial/> (ie up until step 4 in the instructions)
4. Configure `test/wopiserver-test.conf` such that the wopiserver can talk to your Reva instance: use [this example](docker/etc/wopiserver.cs3.conf) for a skeleton configuration
5. Run the tests: `WOPI_STORAGE=cs3 python3 test/test_storageiface.py`
3. For a production deployment, configure your `wopiserver.conf` following the example above, and make sure the `iopsecret` file contains the same secret as configured in the [Reva appprovider](https://developer.sciencemesh.io/docs/technical-documentation/iop/iop-optional-configs/collabora-wopi-server/wopiserver)

### Test against an Eos endpoint:

1. Make sure your Eos instance is configured to accept connections from WOPI as a privileged gateway
2. Configure `test/wopiserver-test.conf` according to your Eos setup (the provided defaults are valid at CERN)
3. Run the tests: `WOPI_STORAGE=xroot python3 test/test_storageiface.py`
4. For a production deployment (CERN only), configure your `wopiserver.conf` according to the Puppet infrastructure

### Test using the Microsoft WOPI validator test suite

Refer to [these notes](test/wopi-validator.md). Microsoft also provides a graphical version of the test suite
as part of their Office 365 offer, which is also supported via the Reva open-in-app workflow.

## Run the WOPI server locally for development purposes

1. Install all requirements: `pip install -r requirements.txt`
2. Add log file directory: `sudo mkdir /var/log/wopi/ && sudo chmod a+rwx /var/log/wopi`
3. Create the folder for the wopi config: `sudo mkdir /etc/wopi/ && sudo chmod a+rwx /etc/wopi`
4. Create recoveryfolder: `sudo mkdir /var/spool/wopirecovery && sudo chmod a+rwx /var/spool/wopirecovery`
5. Create the files `iopsecret` and `wopisecret` in the folder `/etc/wopi/`, create random strings for the secrets
6. Copy the provided [wopiserver.conf](./wopiserver.conf) to `/etc/wopi/wopiserver.defaults.conf`
7. Create a config file `/etc/wopi/wopiserver.conf`: start from `docker/etc/wopiserver.conf` for a minimal configuration and add from the defaults file as needed
8. From the WOPI server folder run: `python3 src/wopiserver.py`

### Test the open-in-app workflow on the local WOPI server

Once the WOPI server runs on top of local storage, the `tools/wopiopen.py` script can be used
to test the open-in-app workflow.
For that, assuming you have e.g. CodiMD deployed in your cluster:

1. Create a `test.md` file in your local storage folder, e.g. `/var/wopi_local_storage`
2. From the WOPI server folder, execute `tools/wopiopen.py -a CodiMD -i "internal_CodiMD_URL" -u "user_visible_CodiMD_URL" -k CodiMD_API_Key test.md`
3. If everything was setup correctly, you'll get a JSON response including an `app-url`. Open it in a browser to access the file. Otherwise, the tool prints the response from the WOPI server and the logs should help troubleshooting the problem.