# cc-legal-tools-app

**Creative Commons (CC) Legal Tools Application.** This repository contains the
application that manages the license tools and public domain tools (static
HTML, internationalization and localization files, etc.). It consumes and
generates data in the [creativecommons/cc-legal-tools-data][repodata]
repository.

[repodata]:https://github.com/creativecommons/cc-legal-tools-data

## Code of conduct

[`CODE_OF_CONDUCT.md`][org-coc]:
> The Creative Commons team is committed to fostering a welcoming community.
> This project and all other Creative Commons open source projects are governed
> by our [Code of Conduct][code_of_conduct]. Please report unacceptable
> behavior to [conduct@creativecommons.org](mailto:conduct@creativecommons.org)
> per our [reporting guidelines][reporting_guide].

[org-coc]: https://github.com/creativecommons/.github/blob/main/CODE_OF_CONDUCT.md
[code_of_conduct]: https://opensource.creativecommons.org/community/code-of-conduct/
[reporting_guide]: https://opensource.creativecommons.org/community/code-of-conduct/enforcement/

## Contributing

See [`CONTRIBUTING.md`][org-contrib].

[org-contrib]: https://github.com/creativecommons/.github/blob/main/CONTRIBUTING.md

## About

This application manages 639 legal tools (636 licenses and 3 public domain
tools). The current version of the licenses is 4.0 and includes 6 licenses.
They are international and are designed to operate globally, ensuring they are
robust, enforceable and easily adopted worldwide. Prior versions were adapted
to specific jurisdictions ("ported"). That is why there are 636 licenses.

Broadly speaking, each legal tool consists of three layers:
1. `deed`: a plain language summary of the legal tool
2. `legalcode`: the legal tool itself
3. `rdf`: metadata about the legal tool in RDF/XML format

With translations of the deed and translations of the legal code, this
application manages over 30,000 documents.

### Not the live site

This project is not intended to serve the legal tools directly. Instead, a
command line tool can be used to save all the rendered HTML and RDF/XML pages
as files. Then those files are used as part of the CreativeCommons.org
site (served as static files).

## Software Versions

- [Python 3.11][python311] specified in:
  - [`.github/workflows/django-app-coverage.yml`][django-app-coverage]
  - [`.github/workflows/static-analysis.yml`][static-analysis]
  - [`.pre-commit-config.yaml`](.pre-commit-config.yaml)
  - [`Dockerfile`](Dockerfile)
  - [`Pipfile`](Pipfile)
  - [`pyproject.toml`](pyproject.toml)
- [Django 4.2 (LTS)][django42]
  - [`Pipfile`](Pipfile)

[django-app-coverage]: .github/workflows/django-app-coverage.yml
[static-analysis]: .github/workflows/static-analysis.yml
[python311]: https://docs.python.org/3.11/
[django42]: https://docs.djangoproject.com/en/4.2/

## Setting up the Project

### Data Repository

Visit [Cloning a Repository][gitclone] on how to clone a GitHub repository.

The [creativecommons/cc-legal-tools-data][repodata] project repository should
be cloned into a directory adjacent to this one:
```
PARENT_DIR
├── cc-legal-tools-app     (git clone of this repository)
└── cc-legal-tools-data    (git clone of the cc-legal-tools-data repository)
```

If it is not cloned into the default location, the Django
`DATA_REPOSITORY_DIR` Django configuration setting, or the
`DATA_REPOSITORY_DIR` environment variable can be used to configure its
location.

[gitclone]:https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository
[repodata]:https://github.com/creativecommons/cc-legal-tools-data

### Docker Compose Setup

Use the following instructions to start the project with Docker compose.
Pleaes note that CC staff use macOS for development--please help us with
documenting other operating systems if you encounter issues.

1. Ensure the [Data Repository](#data-repository), above, is in place
2. [Install Docker Engine](https://docs.docker.com/engine/install/)
3. Ensure you are at the top level of the directory where you cloned this repository (where `manage.py` is)
4. Create Django local settings file
    ```shell
    cp cc_legal_tools/settings/local.example.py cc_legal_tools/settings/local.py
    ```
5. Build the containers
    ```shell
    docker compose build
    ```
6. **Run the containers**
    ```shell
    docker compose up
    ```
   1. **app** ([127.0.0.1:8005](http://127.0.0.1:8005/)): this Django
      application
      - Any changes made to Python will be detected and rebuilt
        transparently as long as the development server is running.
   2. **static** ([127.0.0.1:8006](http://127.0.0.1:8006/)): a static web
      server serving [creativecommons/cc-legal-tools-data][repodata]:`docs/`
7. Run database migrations
    ```shell
    docker compose exec app ./manage.py migrate
    ```
8. Clear data in the database
    ```shell
    docker compose exec app ./manage.py clear_license_data
    ```
9. Load legacy HTML in the database
    ```shell
    docker compose exec app ./manage.py load_html_files
    ```

[repodata]:https://github.com/creativecommons/cc-legal-tools-data

### Manual Setup

> :warning: **This section may be helpful for maintaining the project, but
> should NOT be used for development. Please use the Docker Compose Setup,
> above.**

1. Development Environment
   1. Ensure the [Data Repository](#data-repository), above, is in place
   2. Install dependencies
      - Linux:
        ```shell
        sudo apt-get install python3.11 python3.11-dev python3-pip
        ```
        ```shell
        pip3 install pipenv
        ```
      - macOS: via [Homebrew](https://brew.sh/):
        ```shell
        brew install pipenv python@3.11
        ```
      - Windows: [install Python][python-windows] and then use `pip` to install
        `pipenv`:
        ```shell
        pip install pipenv
        ```
   3. Install Python environment and modules via pipenv to create a
      virtualenv
      - Linux:
        ```shell
        pipenv install --dev --python /usr/bin/python3.11
        ```
      - macOS: via [Homebrew](https://brew.sh/):
        ```shell
        pipenv install --dev --python /usr/local/opt/python@3.11/libexec/bin/python
        ```
      - Windows:
        ```shell
        pipenv install --dev --python \User\Appdata\programs\python
        ```
   4. Install pre-commit hooks
    ```shell
    pipenv run pre-commit install
    ```
2. Configure Django
   1. Create Django local settings file
    ```shell
    cp cc_legal_tools/settings/local.example.py cc_legal_tools/settings/local.py
    ```
   2. Create project database
      - Linux:
        ```shell
        sudo createdb -E UTF-8 cc_legal_tools
        ```
      - macOS:
        ```shell
        createdb -E UTF-8 cc_legal_tools
        ```
      - Windows:
        ```shell
        createdb -E UTF-8 cc_legal_tools
        ```
   3. Load database schema
    ```shell
    pipenv run ./manage.py migrate
    ```
3. Run development server ([127.0.0.1:8005](http://127.0.0.1:8005/))
    ```shell
    pipenv run ./manage.py runserver
    ```
   - Any changes made to Python will be detected and rebuilt transparently as
     long as the development server is running.

[python-windows]:https://www.pythontutorial.net/getting-started/install-python/

#### Manual Commands

> :information_source: The rest of the documentation assumes Docker. If you are
> using a manual setup, use `pipenv run` instead of `docker compose exec app`
> for the commands below.

### Tooling

- **[Python Guidelines — Creative Commons Open Source][ccospyguide]**
- [Black][black]: the uncompromising Python code formatter
- [Coverage.py][coveragepy]: Code coverage measurement for Python
- Docker
  - [Dockerfile reference | Docker Documentation][dockerfile]
  - [Compose file version 3 reference | Docker Documentation][compose3]
- [flake8][flake8]: a python tool that glues together pep8, pyflakes, mccabe,
  and third-party plugins to check the style and quality of some python code.
- [isort][isort]: A Python utility / library to sort imports.
- [pre-commit][precommit]: A framework for managing and maintaining
  multi-language pre-commit hooks.

[ccospyguide]: https://opensource.creativecommons.org/contributing-code/python-guidelines/
[black]: https://github.com/psf/black
[coveragepy]: https://github.com/nedbat/coveragepy
[dockerfile]: https://docs.docker.com/engine/reference/builder/
[compose3]: https://docs.docker.com/compose/compose-file/compose-file-v3/
[flake8]: https://gitlab.com/pycqa/flake8
[isort]: https://pycqa.github.io/isort/
[precommit]: https://pre-commit.com/

#### Helper Scripts

Best run before every commit:
- `./dev/coverage.sh` - Run coverage tests and report
- `./dev/tools.sh` - Run Python code tools (isort, black, flake8)

Run as needed:
- `./dev/copy_theme.sh` - Copy the portions of
  [creativecommons/vocabulary-theme][vocab-theme] needed for local development
  - Run after each new release of
    [creativecommons/vocabulary-theme][vocab-theme]

Esoteric and dangerous:
- `./dev/concatenatemessages.sh` - Concatenate legacy ccEngine translations
  into cc-legal-tools-app
  - rarely used (only after source strings are updated)
- `./dev/resetdb.sh` - Reset Django application database data (!!DANGER!!)
  - usually only helpful if you're doing model/schema work
- `./dev/updatemessages.sh` - Run Django Management nofuzzy_makemessages with
  helpful options (including excluding legalcode) and compilemessages

[vocab-theme]: https://github.com/creativecommons/vocabulary-theme

#### Coverage Tests and Report

The coverage tests and report are run as part of pre-commit and as a GitHub
Action. To run it manually:
1. Ensure the [Data Repository](#data-repository), above, is in place
2. Ensure [Docker Compose Setup](#docker-compose-setup), above, is complete
2. Coverage test
    ```shell
    docker compose exec app coverage run manage.py test --noinput --keepdb
    ```
3. Coverage report
    ```shell
    docker compose exec app coverage report
    ```

### Commit Errors

#### Error building trees

If you encounter an `error: Error building trees` error from pre-commit when
you commit, try adding your files (`git add <FILES>`) before committing them.

## Frontend Dependencies

The following CC projects are used to achieve a consistent look and feel:
- [creativecommons/vocabulary-theme][vocabulary-theme]: WordPress Theme
  implementation of the Vocabulary design system

[vocabulary-theme]: https://github.com/creativecommons/vocabulary-theme

## Data

The legal tools metadata is in a database. The metadata tracks which legal
tools exist, their translations, their ports, and their characteristics like
what they permit, require, and prohibit.

~~The metadata can be downloaded by visiting the URL path:
`127.0.0.1:8005`[`/licenses/metadata.yaml`][metadata]~~ (currently disabled)

[metadata]: http://127.0.0.1:8005/licenses/metadata.yaml

There are two main models (Django terminology for tables) in
[`legal_tools/models.py`](legal_tools/models.py):
1. `LegalCode`
2. `Tool`

A Tool can be identified by a `unit` (ex. `by`, `by-nc-sa`, `devnations`) which
is a proxy for the complete set of permissions, requirements, and prohibitions;
a `version` (ex. `4.0`, `3.0)`, and an optional `jurisdiction` for ports. So we
might refer to the tool by its **identifier** "BY 3.0 AM" which would be the
3.0 version of the BY license terms as ported to the Armenia jurisdiction. For
additional information see: [**Legal Tools Namespace** -
creativecommons/cc-legal-tools-data: CC Legal Tools Data (static HTML, language
files, etc.)][namespace].

There are three places legal code text could be:
1. **Gettext files** (`.po` and `.mo`) in the
   [creativecommons/cc-legal-tools-data][repodata] repository (legal tools with
   full translation support):
   - 4.0 Licenses
   - CC0
2. **Django template**
   ([`legalcode_licenses_3.0_unported.html`][unportedtemplate]):
   - Unported 3.0 Licenses (English-only)
3. **`html` field** (in the `LegalCode` model):
   - Everything else

The text that's in gettext files can be translated via Transifex at [Creative
Commons localization][cctransifex]. For additional information on the Django
translation domains / Transifex resources, see [How the license translation is
implemented](#how-the-tool-translation-is-implemented), below.

Documentation:
- [Models | Django documentation | Django][djangomodels]
- [Templates | Django documentation | Django][djangotemplates]

[namespace]: https://github.com/creativecommons/cc-legal-tools-data#legal-tools-namespace
[unportedtemplate]: templates/includes/legalcode_licenses_3.0_unported.html
[cctransifex]: https://www.transifex.com/creativecommons/public/
[djangomodels]: https://docs.djangoproject.com/en/4.2/topics/db/models/
[djangotemplates]: https://docs.djangoproject.com/en/4.2/topics/templates/

## Importing the existing legal tool text

The process of getting the text into the site varies by legal tool.

Note that once the site is up and running in production, the data in the site
will become the canonical source, and the process described here should not
need to be repeated after that.

The implementation is the Django management command `load_html_files`, which
reads from the legacy HTML legal code files in the
[creativecommons/cc-legal-tools-data][repodata] repository, and populates the
database records and translation files.

`load_html_files` uses [BeautifulSoup4][bs4docs] to parse the legacy HTML legal
code:
1. `import_zero_license_html()` for CC0 Public Domain tool
   - HTML is handled specifically (using tag ids and classes) to populate
     translation strings and to be used with specific HTML formatting when
     displayed via template
2. `import_by_40_license_html()` for 4.0 License tools
   - HTML is handled specifically (using tag ids and classes) to populate
     translation strings and to be used with specific HTML formatting when
     displayed via a template
3. `import_by_30_unported_license_html()` for unported 3.0 License tools
   (English-only)
   - HTML is handled specifically to be used with specific HTML formatting
     when displayed via a template
4. `simple_import_license_html()` for everything else
   - HTML is handled generically; only the title and license body are
     identified. The body is stored in the `html` field of the
     `LegalCode` model

[bs4docs]: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
[repodata]: https://github.com/creativecommons/cc-legal-tools-data

### Import Process

This process will read the HTML files from the specified directory, populate
`LegalCode` and `Tool` models, and create the `.po` portable object Gettext
files in [creativecommons/cc-legal-tools-data][repodata].

1. Ensure the [Data Repository](#data-repository), above, is in place
2. Ensure [Docker Compose Setup](#docker-compose-setup), above, is complete
3. Clear data in the database
    ```shell
    docker compose exec app ./manage.py clear_license_data
    ```
4. Load legacy HTML in the database
    ```shell
    docker compose exec app ./manage.py load_html_files
    ```
5. Optionally (and only as appropriate):
   1. Commit the `.po` portable object Gettext file changes in
      [creativecommons/cc-legal-tools-data][repodata]
   2. [Translation Update Process](#translation-update-process), below
   3. [Generate Static Files](#generate-static-files), below

[repodata]:https://github.com/creativecommons/cc-legal-tools-data

### Import Dependency Documentation

- [Beautiful Soup Documentation — Beautiful Soup 4 documentation][bs4docs]
  - [lxml - Processing XML and HTML with Python][lxml]
- [Quick start guide — polib documentation][polibdocs]

[bs4docs]: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
[lxml]: https://lxml.de/
[polibdocs]: https://polib.readthedocs.io/en/latest/quickstart.html

## Translation

To upload/download translation files to/from Transifex, you'll need an account
there with access to these translations. Then follow the [Authentication -
Transifex API v3][transauth]: to get an API token, and set
`TRANSIFEX["API_TOKEN"]` in your environment with its value.

The [creativecommons/cc-legal-tools-data][repodata] repository must be cloned
next to this `cc-legal-tools-app` repository. (It can be elsewhere, then you
need to set `DATA_REPOSITORY_DIR` to its location.) Be sure to clone using a
URL that starts with `git@github...` and not `https://github...`, or you won't
be able to push to it. Also see [Data Repository](#data-repository), above.

In production, the `check_for_translation_updates` management command should be
run hourly. See [Check for Translation
Updates](#check-for-translation-updates), below.

Also see [Publishing changes to git repo](#publishing-changes-to-git-repo),
below.

[Babel][babel] is used for localization information.

Documentation:
- [Babel — Babel documentation][babel]
- [Translation | Django documentation | Django][djangotranslation]

[babel]: http://babel.pocoo.org/en/latest/index.html
[repodata]:https://github.com/creativecommons/cc-legal-tools-data
[transauth]: https://transifex.github.io/openapi/index.html#section/Authentication

### How the tool translation is implemented

Django Translation uses two sets of Gettext Files in the
[creativecommons/cc-legal-tools-data][repodata] repository (the [Data
Repository](#data-repository), above). See that repository for detailed
information and definitions.

Documentation:
- [Translation | Django documentation | Django][djangotranslation]
- Transifex API
  - [Introduction to API 3.0 | Transifex Documentation][api30intro]
  - [Transifex API v3][api30]
  - Python SDK: [transifex-python/transifex/api][apisdk]

[api30]: https://transifex.github.io/openapi/index.html#section/Introduction
[api30intro]: https://docs.transifex.com/api-3-0/introduction-to-api-3-0
[apisdk]: https://github.com/transifex/transifex-python/tree/devel/transifex/api
[djangotranslation]: https://docs.djangoproject.com/en/4.2/topics/i18n/translation/
[repodata]: https://github.com/creativecommons/cc-legal-tools-data

### Check for Translation Updates

> :warning: **This functionality is currently disabled.**

The hourly run of `check_for_translation_updates` looks to see if any of the
translation files in Transifex have newer last modification times than we know
about. It performs the following process (which can also be done manually:

1. Ensure the [Data Repository](#data-repository), above, is in place
2. Within the [creativecommons/cc-legal-tools-data][repodata] (the [Data
   Repository](#data-repository)):
   1. Checkout or create the appropriate branch.
      - For example, if a French translation file for BY 4.0 has changed, the
        branch name will be `cc4-fr`.
   2. Download the updated `.po` portable object Gettext file from Transifex
   3. Do the [Translation Update Process](#translation-update-process) (below)
      - _This is important and easy to forget,_ but without it, Django will
        keep using the old translations
   4. Commit that change and push it upstream.
3. Within this `cc-legal-tools-app` repository:
   1. For each branch that has been updated, [Generate Static
      Files](#generate-static-files) (below). Use the options to update git and
      push the changes.

[repodata]:https://github.com/creativecommons/cc-legal-tools-data

### Check for Translation Updates Dependency Documentation

- [GitPython Documentation — GitPython documentation][gitpythondocs]
- [Requests: HTTP for Humans™ — Requests documentation][requestsdocs]

[gitpythondocs]: https://gitpython.readthedocs.io/en/stable/index.html
[requestsdocs]: https://docs.python-requests.org/en/master/

### Translation Update Process

This Django Admin command must be run any time the `.po` portable object
Gettext files are created or changed.

1. Ensure the [Data Repository](#data-repository), above, is in place
2. Ensure [Docker Compose Setup](#docker-compose-setup), above, is complete
3. Compile translation messages (update the `.mo` machine object Gettext files)
    ```shell
    docker compose exec app ./manage.py compilemessages
    ```

## Generate Static Files

Generating static files updates the static files in the `docs/` directory of
the [creativecommons/cc-legal-tools-data][repodata] repository (the [Data
Repository](#data-repository), above).

### Static Files Process

This process will write the HTML files in the cc-legal-tools-data clone
directory under `docs/`. It will not commit the changes (`--nogit`) and will
not push any commits (`--nopush` is implied by `--nogit`).

1. Ensure the [Data Repository](#data-repository), above, is in place
2. Ensure [Docker Compose Setup](#docker-compose-setup), above, is complete
3. Delete the contents of the `docs/` directory and then recreate/copy the
   static files it should contain:
    ```shell
    docker compose exec app ./manage.py publish -v2
    ```

### Publishing changes to git repo

When the site is deployed, to enable pushing and pulling the licenses data repo
with GitHub, create an SSH deploy key for the cc-legal-tools-data repo with
write permissions, and put the private key file (not password protected)
somewhere safe (owned by `www-data` if on a server), and readable only by its
owner (0o400). Then in settings, make `TRANSLATION_REPOSITORY_DEPLOY_KEY` be
the full path to that deploy key file.

### Publishing Dependency Documentation

- [Beautiful Soup Documentation — Beautiful Soup 4 documentation][bs4docs]
  - [lxml - Processing XML and HTML with Python][lxml]
- [GitPython Documentation — GitPython documentation][gitpythondocs]

[bs4docs]: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
[gitpythondocs]: https://gitpython.readthedocs.io/en/stable/index.html
[lxml]: https://lxml.de/

## Licenses

### Code

[`LICENSE`](LICENSE): the code within this repository is licensed under the
Expat/[MIT][mit] license.

[mit]: http://www.opensource.org/licenses/MIT "The MIT License | Open Source Initiative"

### Legal Code text

[![CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
button][cc-zero-png]][cc-zero]

The text of the Creative Commons public licenses (legal code) is dedicated to
the public domain under the [CC0 1.0 Universal (CC0 1.0) Public Domain
Dedication][cc-zero].

[cc-zero-png]: https://licensebuttons.net/l/zero/1.0/88x31.png "CC0 1.0 Universal (CC0 1.0) Public Domain Dedication button"
[cc-zero]: https://creativecommons.org/publicdomain/zero/1.0/

### vocabulary-theme

[![CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
button][cc-zero-png]][cc-zero]

[`COPYING`](COPYING): All the code within Vocabulary is dedicated to
the public domain under the [CC0 1.0 Universal (CC0 1.0) Public Domain
Dedication][cc-zero].

[cc-zero-png]: https://licensebuttons.net/l/zero/1.0/88x31.png "CC0 1.0 Universal (CC0 1.0) Public Domain Dedication button"
[cc-zero]: https://creativecommons.org/publicdomain/zero/1.0/ "Creative Commons — CC0 1.0 Universal"

#### Normalize.css

normalize.css is licensed under the Expat/[MIT][mit] License.

[mit]: https://opensource.org/license/mit/

#### Fonts

##### Accidenz Commons

[Accidenz Commons][accidenzcommons] by Archetypo is licensed under the [Creative
Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
License][ccbysa40].

[accidenzcommons]: https://creativecommons.org/2019/10/28/accidenz-commons-open-licensed-font/
[ccbysa40]: https://creativecommons.org/licenses/by-sa/4.0/

##### JetBrains Mono

[JetBrains Mono][jetbrainsmono] is licensed under the [OFL-1.1 License][ofl].

[jetbrainsmono]: https://www.jetbrains.com/lp/mono/
[ofl]: https://github.com/JetBrains/JetBrainsMono/blob/master/OFL.txt

##### Roboto Condensed

[Roboto Condensed][robotocondensed] by Christian Robertson is licensed under
the [Apache License, Version 2.0][apache20].

[robotocondensed]: https://fonts.google.com/specimen/Roboto+Condensed
[apache20]: http://www.apache.org/licenses/LICENSE-2.0

##### Source Sans Pro

[Source Sans Pro][sourcesanspro] by Paul D. Hunt is licensed under the [Open
Font License][oflsil].

[sourcesanspro]: https://fonts.adobe.com/fonts/source-sans
[oflsil]: https://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL

##### Vocabulary Icons

Vocabulary Icons use icons from [Font Awesome][fontawesome] which are licensed
under the [Creative Commons Attribution 4.0 International (CC BY 4.0)
License][ccbysa40].

[fontawesome]: https://fontawesome.com/
[ccby40]: https://creativecommons.org/licenses/by/4.0/


ansible-zuul-jobs
=================

Zuul job definitions for Ansible tenant.


![header ](imgs/of_banner.png)
_Figure: Comparison of OpenFold and AlphaFold2 predictions to the experimental structure of PDB 7KDX, chain B._

# OpenFold

A faithful but trainable PyTorch reproduction of DeepMind's
[AlphaFold 2](https://github.com/deepmind/alphafold).

# Documentation
See our new home for docs at [openfold.readthedocs.io](https://openfold.readthedocs.io/en/latest/), with instructions for installation and model inference/training.

Much of the content from this page may be found [here.](https://github.com/aqlaboratory/openfold/blob/main/docs/source/original_readme.md)

## Copyright Notice

While AlphaFold's and, by extension, OpenFold's source code is licensed under
the permissive Apache Licence, Version 2.0, DeepMind's pretrained parameters
fall under the CC BY 4.0 license, a copy of which is downloaded to
`openfold/resources/params` by the installation script. Note that the latter
replaces the original, more restrictive CC BY-NC 4.0 license as of January 2022.

## Contributing

If you encounter problems using OpenFold, feel free to create an issue! We also
welcome pull requests from the community.

## Citing this Work

Please cite our paper:

```bibtex
@article {Ahdritz2022.11.20.517210,
	author = {Ahdritz, Gustaf and Bouatta, Nazim and Floristean, Christina and Kadyan, Sachin and Xia, Qinghui and Gerecke, William and O{\textquoteright}Donnell, Timothy J and Berenberg, Daniel and Fisk, Ian and Zanichelli, Niccolò and Zhang, Bo and Nowaczynski, Arkadiusz and Wang, Bei and Stepniewska-Dziubinska, Marta M and Zhang, Shang and Ojewole, Adegoke and Guney, Murat Efe and Biderman, Stella and Watkins, Andrew M and Ra, Stephen and Lorenzo, Pablo Ribalta and Nivon, Lucas and Weitzner, Brian and Ban, Yih-En Andrew and Sorger, Peter K and Mostaque, Emad and Zhang, Zhao and Bonneau, Richard and AlQuraishi, Mohammed},
	title = {{O}pen{F}old: {R}etraining {A}lpha{F}old2 yields new insights into its learning mechanisms and capacity for generalization},
	elocation-id = {2022.11.20.517210},
	year = {2022},
	doi = {10.1101/2022.11.20.517210},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/10.1101/2022.11.20.517210},
	eprint = {https://www.biorxiv.org/content/early/2022/11/22/2022.11.20.517210.full.pdf},
	journal = {bioRxiv}
}
```
If you use OpenProteinSet, please also cite:

```bibtex
@misc{ahdritz2023openproteinset,
      title={{O}pen{P}rotein{S}et: {T}raining data for structural biology at scale},
      author={Gustaf Ahdritz and Nazim Bouatta and Sachin Kadyan and Lukas Jarosch and Daniel Berenberg and Ian Fisk and Andrew M. Watkins and Stephen Ra and Richard Bonneau and Mohammed AlQuraishi},
      year={2023},
      eprint={2308.05326},
      archivePrefix={arXiv},
      primaryClass={q-bio.BM}
}
```
Any work that cites OpenFold should also cite [AlphaFold](https://www.nature.com/articles/s41586-021-03819-2) and [AlphaFold-Multimer](https://www.biorxiv.org/content/10.1101/2021.10.04.463034v1) if applicable.


[![Disnake Banner](https://raw.githubusercontent.com/DisnakeDev/disnake/master/assets/banner.png)](https://disnake.dev/)

disnake
=======

A modern, easy to use, feature-rich, and async-ready API wrapper for Discord written in Python.

Key Features
------------

- Proper rate limit handling.
- Type-safety measures.
- [FastAPI](https://fastapi.tiangolo.com/)-like slash command syntax.

The syntax and structure of `discord.py 2.0` is preserved.

Installing
----------

**Python 3.8 or higher is required.**

To install the library without full voice support, you can just run the
following command:

``` sh
# Linux/macOS
python3 -m pip install -U disnake

# Windows
py -3 -m pip install -U disnake
```

Installing `disnake` with full voice support requires you to replace `disnake` here, with `disnake[voice]`. To learn more about voice support (or installing the development version), please visit [this section of our guide](https://guide.disnake.dev/prerequisites/installing-disnake/).

(You can optionally install [PyNaCl](https://pypi.org/project/PyNaCl/) for voice support.)

Note that voice support on Linux requires installation of `libffi-dev` and `python-dev` packages, via your preferred package manager (e.g. `apt`, `dnf`, etc.) before running the following commands.

Versioning
----------

This project does **not** quite follow semantic versioning; for more details, see [version guarantees](https://docs.disnake.dev/en/latest/version_guarantees.html).

To be on the safe side and avoid unexpected breaking changes, pin the dependency to a minor version (e.g. `disnake==a.b.*` or `disnake~=a.b.c`) or an exact version (e.g. `disnake==a.b.c`).

Quick Example
-------------

### Slash Commands Example

``` py
import disnake
from disnake.ext import commands

bot = commands.InteractionBot(test_guilds=[12345])

@bot.slash_command()
async def ping(inter):
    await inter.response.send_message("Pong!")

bot.run("BOT_TOKEN")
```

### Context Menus Example

``` py
import disnake
from disnake.ext import commands

bot = commands.InteractionBot(test_guilds=[12345])

@bot.user_command()
async def avatar(inter, user):
    embed = disnake.Embed(title=str(user))
    embed.set_image(url=user.display_avatar.url)
    await inter.response.send_message(embed=embed)

bot.run("BOT_TOKEN")
```

### Prefix Commands Example

``` py
import disnake
from disnake.ext import commands

bot = commands.Bot(command_prefix=commands.when_mentioned)

@bot.command()
async def ping(ctx):
    await ctx.send("Pong!")

bot.run("BOT_TOKEN")
```

You can find more examples in the [examples directory](https://github.com/DisnakeDev/disnake/tree/master/examples).

    Documentation
    ⁕
    Guide
    ⁕
    Discord Server
    ⁕
    Discord Developers


[<img src="https://img.shields.io/badge/dockerhub-imSim image-orange.svg?logo=Docker">](https://hub.docker.com/r/lsstdesc/imsim-env)
[![Build Status](https://github.com/LSSTDESC/imSim/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/LSSTDESC/imSim/actions/workflows/ci.yml/badge.svg?branch=main)

# imSim
imSim is a software package that simulates the LSST telescope and survey.
It produces simulated images from the 3.25 Gigapixel camera which are suitable
to be processed through the LSST Data Management pipeline. imSim takes as an
input a catalog of astronomical sources along with information about how the light
is distorted on the way to Earth including lensing and extinction information.  The
images which are produced include the systematic effects of the atmosphere,
optics and sensors on the observed PSF.

imSim calls the [GalSim](https://github.com/GalSim-developers/GalSim "GalSim GitHub Page") library for astronomical object rendering and is designed to be used inside of the
LSST Data Management and LSST Simulation Group software environment.  It is not a stand alone program.  It requires a working LSST software stack to build and run.

Communication with the imSim development team should take place through the
[imSim GitHub repository](https://github.com/LSSTDESC/imSim) which is part of the
DESC GitHub organization.  Other questions can be directed to Chris Walter at Duke University.

## imSim models

Details on the models implemented by imSim including their validations can be found in the [imSim Feature Matrix](https://lsstdesc.github.io/imSim/features.html).

## Set up

Please visit the [docs about installation](https://lsstdesc.github.io/imSim/install.html).

## Usage

Basic usage:

Create a configuration `config.yaml`, then run

```sh
galsim config.yaml
```

For more details, visit the [docs about usage](https://lsstdesc.github.io/imSim/usage.html).

## Release Status
imSim is currently still considered beta code. A list of tagged versions and pre-releases of this package can be found [here](https://github.com/LSSTDESC/imSim/releases), with the main branch including the most recent (non-tagged) development.

## Licensing and ownership
This software was developed within the LSST DESC using LSST DESC resources, and
so meets the criteria given in, and is bound by, the LSST DESC Publication Policy
for being a “DESC product”.  imSim is licensed with a BSD 3-Clause "New" or "Revised" License as detailed in the included LICENSE file.


[![Blossom-CI](https://github.com/NVIDIA/nvflare/workflows/Blossom-CI/badge.svg?branch=main)](https://github.com/NVIDIA/nvflare/actions)
[![documentation](https://readthedocs.org/projects/nvflare/badge/?version=main)](https://nvflare.readthedocs.io/en/main/?badge=main)
[![license](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](./LICENSE)
[![pypi](https://badge.fury.io/py/nvflare.svg)](https://badge.fury.io/py/nvflare)
[![pyversion](https://img.shields.io/pypi/pyversions/nvflare.svg)](https://badge.fury.io/py/nvflare)
[![downloads](https://static.pepy.tech/badge/nvflare)](https://pepy.tech/project/nvflare)

# NVIDIA FLARE

[NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) (**NV**IDIA **F**ederated **L**earning **A**pplication **R**untime **E**nvironment)
is a domain-agnostic, open-source, extensible SDK that allows researchers and data scientists to adapt existing ML/DL workflows to a federated paradigm.
It enables platform developers to build a secure, privacy-preserving offering for a distributed multi-party collaboration.

## Features
FLARE is built on a componentized architecture that allows you to take federated learning workloads
from research and simulation to real-world production deployment.

Application Features
* Support both deep learning and traditional machine learning algorithms (eg. PyTorch, TensorFlow, Scikit-learn, XGBoost etc.)
* Support horizontal and vertical federated learning
* Built-in Federated Learning algorithms (e.g., FedAvg, FedProx, FedOpt, Scaffold, Ditto, etc.)
* Support multiple server and client-controlled training workflows (e.g., scatter & gather, cyclic) and validation workflows (global model evaluation, cross-site validation)
* Support both data analytics (federated statistics) and machine learning lifecycle management
* Privacy preservation with differential privacy, homomorphic encryption, private set intersection (PSI)

From Simulation to Real-World
* FLARE Client API to transition seamlessly from ML/DL to FL with minimal code changes
* Simulator and POC mode for rapid development and prototyping
* Fully customizable and extensible components with modular design
* Deployment on cloud and on-premise
* Dashboard for project management and deployment
* Security enforcement through federated authorization and privacy policy
* Built-in support for system resiliency and fault tolerance

> _Take a look at [NVIDIA FLARE Overview](https://nvflare.readthedocs.io/en/main/flare_overview.html) for a complete overview, and [What's New](https://nvflare.readthedocs.io/en/main/whats_new.html) for the lastest changes._

## Installation
To install the [current release](https://pypi.org/project/nvflare/):
```
$ python3 -m pip install nvflare
```
## Getting Started

You can quickly get started using the [FL simulator](https://nvflare.readthedocs.io/en/main/getting_started.html#the-fl-simulator).
A detailed [getting started](https://nvflare.readthedocs.io/en/main/getting_started.html) guide is available in the [documentation](https://nvflare.readthedocs.io/en/main/index.html).

Examples and notebook tutorials are located at [NVFlare/examples](./examples).

## Community

We welcome community contributions! Please refer to the [contributing guidelines](https://github.com/NVIDIA/NVFlare/blob/main/CONTRIBUTING.md) for more details.

Ask and answer questions, share ideas, and engage with other community members at [NVFlare Discussions](https://github.com/NVIDIA/NVFlare/discussions).

## Related Talks and Publications

Take a look at our growing list of [talks, blogs, and publications](https://nvflare.readthedocs.io/en/main/publications_and_talks.html) related to NVIDIA FLARE.

## License

NVIDIA FLARE is released under an [Apache 2.0 license](https://github.com/NVIDIA/NVFlare/blob/main/LICENSE).


<!--
 ~ SPDX-FileCopyrightText: Copyright DB InfraGO AG
 ~ SPDX-License-Identifier: Apache-2.0
 -->

Python-Capellambse
==================

![PyPI - Python Version](https://img.shields.io/pypi/pyversions/capellambse)
![Code QA](https://github.com/DSD-DBS/py-capellambse/actions/workflows/code-qa.yml/badge.svg)
![License: Apache-2.0](https://img.shields.io/github/license/dsd-dbs/py-capellambse)
![REUSE status](https://api.reuse.software/badge/github.com/DSD-DBS/py-capellambse)
![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)
![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)

*A Python 3 headless implementation of the Capella modeling tool*

Intro
-----

`capellambse` allows you reading and writing Capella models from Python without
Java or the [Capella tool](https://www.eclipse.org/capella/) on any
(reasonable) platform. We wanted to "talk" to Capella models from Python, but
without any Java on the way. We thought this project will help individuals and
organisations getting through the MBSE adoption journey with Capella faster and
so we made it public and open-source.

With `capellambse` you can access all (or almost all) Capella model elements,
and render diagrams as SVG and PNG. We made it for automation of Systems
Engineering work, so it integrates nicely into most CI/CD toolchains. We also
found it at the core of our artifact generation pipelines (model to documents,
model to SW interfaces).

The library works with [PVMT](https://www.eclipse.org/capella/addons.html) and
[Requirements](https://github.com/eclipse/capella-requirements-vp) extensions
without any additional efforts.

It started as a basic library somewhere mid 2019. Since then it was
re-architected a few times and now has a full read/write capability for most of
the present Capella ontology. We are continuously improving the API
(introducing shortcuts), increasing the meta-model coverage and have more
engineering automations and improvements in the pipeline to share.

Related projects
----------------

- [`capellambse-context-diagrams`](https://github.com/DSD-DBS/capellambse-context-diagrams)
  — A capellambse extension that visualizes the context of Capella objects, and
  exposes it on element attributes like `.context_diagram`, `.tree_view`, etc.

- [`capella-diff-tools`](https://github.com/DSD-DBS/capella-diff-tools) — A set
  of tools to compare Capella models.

- [`capella-ros-tools`](https://github.com/DSD-DBS/capella-ros-tools) — Import
  and export ROS `*.msg` files to/from Capella models, or transform `*.msg`
  files into a declarative YAML file.

Did you make something cool that is using or extending capellambse? [Tell us
about it](https://github.com/DSD-DBS/py-capellambse/issues), so we can add it
to this list!

Documentation and examples
--------------------------

The library is designed to be easy to use and discover, especially in an
interactive environment such as JupyterLab. Additionally, [API
documentation](https://dsd-dbs.github.io/py-capellambse/) is automatically
generated and published whenever new features and bug fixes are added.

You are encouraged to explore our test models and demo notebooks. Click on the
button below to launch a [Jupyter notebook server] on the public myBinder
service, and get started in seconds:

[![myBinder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/DSD-DBS/py-capellambse/HEAD?labpath=docs%2Fsource%2Fexamples%2F01%20Introduction.ipynb)

*Warning:* [Do not enter confidential
information](https://github.com/alan-turing-institute/the-turing-way/blob/b36c3ac1c78acbbe18441beaa89514544ed12021/workshops/boost-research-reproducibility-binder/workshop-presentations/zero-to-binder-python.md#private-files),
such as passwords for non-public models, into a notebook hosted on myBinder. If
you want to try out `capellambse` with those models, please [install and
run](#installation) it in a local, trusted environment!

The `docs/source/examples` directory contains several hands-on example
notebooks that you can immediately run and start experimenting with. Below is a
short summary of each notebook's goal. If you are in the JupyterLab
environment, you can click the notebook names to directly open them in a new
lab tab. On Github, you will be shown a statically rendered preview of the
notebook.

- [01
  Introduction.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/01%20Introduction.html)
  provides a high-level overview of the library features. It visualizes
  examples like a Component - Function allocation table by leveraging Jupyter's
  and IPython's rich display functionality.

- [02 Intro to Physical
  Architecture.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/02%20Intro%20to%20Physical%20Architecture%20API.html)
  explores some more advanced concepts on the example of the Physical
  Architecture Layer. It shows how to derive tabular data, like a Bill of
  Materials or a Software to Hardware allocation table, by using `pandas`
  dataframes.

- [03 Data
  Values.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/03%20Data%20Values.html)
  shows how the API can be used to explore classes, class instances and other
  objects related to data modeling.

- [04 Intro to Jinja
  templating.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/04%20Intro%20to%20Jinja%20templating.html)
  demonstrates how to effectively combine `capellambse` with the powerful
  [Jinja](https://palletsprojects.com/p/jinja/) templating engine. This enables
  the creation of all sorts of model-derived documents and artifacts, including
  interactive web pages, PDF documents and any other textual representations of
  your models.

- [05 Introduction to
  Libraries.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/05%20Introduction%20to%20Libraries.html)
  shows how to use Capella Library Projects within capellambse. In this example
  you'll learn how the API can be used to open a project that is based on a
  library and find objects in both models.

- [06 Introduction to Requirement access and
  management.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/06%20Introduction%20to%20Requirement%20access%20and%20management.html)
  shows how the API can be used to work with requirements objects, introduced
  by the Capella [Requirements
  Viewpoint](https://www.eclipse.org/capella/addons.html). In this example
  you'll see how to find requirements in the model, see which objects
  requirements are linked / traced to and even export requirements to Excel or
  ReqIF formats.

- [07 Code
  Generation.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/07%20Code%20Generation.html)
  shows how to generate code from class diagrams. In particular, we focus on
  Interface Descriptive Languages with concrete examples for `Class` to [ROS2
  IDL](https://docs.ros.org/en/rolling/Concepts/About-ROS-Interfaces.html) and
  Google [Protocol Buffers](https://developers.google.com/protocol-buffers). We
  also show how simple Python stubs could be generated given a `Class`
  object.

- [08 Property
  Values.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/08%20Property%20Values.html)
  shows how to access property values and property value groups, as well as the
  [Property Value Management (PVMT)](https://eclipse.dev/capella/addons.html)
  extension.

- [09 Context
  Diagrams.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/09%20Context%20Diagrams.html)
  shows the [capellambse-context-diagrams
  extension](https://dsd-dbs.github.io/capellambse-context-diagrams/) that
  visualizes contexts of Capella objects. The extension is external to the
  capellambse library and needs to be installed separately.

- [10 Declarative
  Modeling.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/10%20Declarative%20Modeling.html)
  demonstrates a basic application of the declarative approach to modeling on a
  coffee machine example.

- [11 Complexity
  Assessment.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/11%20Complexity%20Assessment.html)
  quickly demonstrates how to use and view the model complexity badge for a
  Capella model.

We are constantly working on improving everything shown here, as well as adding
even more useful functionality and helpful demos. If you have any new ideas
that were not mentioned yet, [don't hesitate to contribute](CONTRIBUTING.md)!

Installation
------------

In order to use private models that are not publicly available, please install
and use `capellambse` in a local, trusted environment.

You can install the latest released version directly from PyPI.

```bash
pip install capellambse
```

To set up a development environment, clone the project and install it into a
virtual environment.

```bash
git clone https://github.com/DSD-DBS/py-capellambse
cd capellambse
python -m venv .venv

source .venv/bin/activate.sh  # for Linux / Mac
.venv\Scripts\activate  # for Windows

pip install -U pip pre-commit
pip install -e '.[docs,test]'
pre-commit install
```

We recommend developing within a local [Jupyter notebook server] environment.
In order to install and run it in the same virtual environment, execute the
following additional commands:

```bash
pip install jupyter capellambse
cd docs/source/examples
jupyter-notebook
```

If your browser did not open automatically, follow the instructions in the
terminal to start it manually.

Once in the browser, simply click on the [01
Introduction.ipynb](https://dsd-dbs.github.io/py-capellambse/examples/01%20Introduction.html)
notebook to start!

Current limitations
-------------------

We are continuously improving coverage of Capella onthology with our
[high-level API](#TODO) (the current coverage map is available [here](#TODO)),
however it is still incomplete. It covers most of the commonly used paths but
when you need to get to an ontology element that isnt covered yet you may do so
by using the [low-level
API](https://dsd-dbs.github.io/py-capellambse/development/low-level-api.html).

Also, as we started in mid 2019 and there was no such thing as
[Python4Capella](https://github.com/labs4capella/python4capella) yet, we are
not API compatible with that project.

The generated diagrams are currently not persisted in `.aird` files, and
currently there is no plan to implement this. If there is a genuine usecase for
that we may re-consider it - feel free to create an issue or add comments to an
existing one.

Contributing
------------

We'd love to see your bug reports and improvement suggestions! Please take a
look at our [guidelines for contributors](CONTRIBUTING.md) for details.

Licenses
--------

This project is compliant with the [REUSE Specification Version
3.0](https://git.fsfe.org/reuse/docs/src/commit/d173a27231a36e1a2a3af07421f5e557ae0fec46/spec.md).

Copyright DB InfraGO AG, licensed under Apache 2.0 (see full text in
[LICENSES/Apache-2.0.txt](LICENSES/Apache-2.0.txt))

Dot-files are licensed under CC0-1.0 (see full text in
[LICENSES/CC0-1.0.txt](LICENSES/CC0-1.0.txt))

To provide the same look and feel across platforms, we distribute our library
bundled with the OpenSans font (`capellambse/OpenSans-Regular.ttf`). The
OpenSans font is Copyright 2020 [The Open Sans Project
Authors](https://github.com/googlefonts/opensans), licensed under OFL-1.1 (see
full text in [LICENSES/OFL-1.1.txt](LICENSES/OFL-1.1.txt)).

[Jupyter notebook server]: https://jupyter.org/


# Viadot
[![build status](https://github.com/dyvenia/viadot/actions/workflows/build.yml/badge.svg)](https://github.com/dyvenia/viadot/actions/workflows/build.yml)
[![formatting](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![codecov](https://codecov.io/gh/Trymzet/dyvenia/branch/main/graph/badge.svg?token=k40ALkXbNq)](https://codecov.io/gh/Trymzet/dyvenia)
---

**Documentation**: https://dyvenia.github.io/viadot/

**Source Code**: https://github.com/dyvenia/viadot

---

A simple data ingestion library to guide data flows from some places to other places.

## Getting Data from a Source

Viadot supports several API and RDBMS sources, private and public. Currently, we support the UK Carbon Intensity public API and base the examples on it.

```python
from viadot.sources.uk_carbon_intensity import UKCarbonIntensity
ukci = UKCarbonIntensity()
ukci.query("/intensity")
df = ukci.to_df()
df
```

**Output:**
|    | from              | to                |   forecast |   actual | index    |
|---:|:------------------|:------------------|-----------:|---------:|:---------|
|  0 | 2021-08-10T11:00Z | 2021-08-10T11:30Z |        211 |      216 | moderate |

The above `df` is a python pandas `DataFrame` object. The above df contains data downloaded from viadot from the Carbon Intensity UK API.

## Loading Data to a Source
Depending on the source, viadot provides different methods of uploading data. For instance, for SQL sources, this would be bulk inserts. For data lake sources, it would be a file upload. We also provide ready-made pipelines including data validation steps using Great Expectations.

An example of loading data into SQLite from a pandas `DataFrame` using the `SQLiteInsert` Prefect task:

```python
from viadot.tasks import SQLiteInsert

insert_task = SQLiteInsert()
insert_task.run(table_name=TABLE_NAME, dtypes=dtypes, db_path=database_path, df=df, if_exists="replace")
```

## Set up

__Note__: If you're running on Unix, after cloning the repo, you may need to grant executable privileges to the `update.sh` and `run.sh` scripts:
```
sudo chmod +x viadot/docker/update.sh && \
sudo chmod +x viadot/docker/run.sh
```

### a) user
Clone the `main` branch, enter the `docker` folder, and set up the environment:
```
git clone https://github.com/dyvenia/viadot.git && \
cd viadot/docker && \
./update.sh
```

Run the enviroment:
```
./run.sh
```

### b) developer
Clone the `dev` branch, enter the `docker` folder, and set up the environment:
```
git clone -b dev https://github.com/dyvenia/viadot.git && \
cd viadot/docker && \
./update.sh -t dev
```

Run the enviroment:
```
./run.sh -t dev
```

Install the library in development mode (repeat for the `viadot_jupyter_lab` container if needed):
```
docker exec -it viadot_testing pip install -e . --user
```

## Running tests

To run tests, log into the container and run pytest:
```
docker exec -it viadot_testing bash
pytest
```

## Running flows locally

You can run the example flows from the terminal:
```
docker exec -it viadot_testing bash
FLOW_NAME=hello_world; python -m viadot.examples.$FLOW_NAME
```

However, when developing, the easiest way is to use the provided Jupyter Lab container available in the browser at `http://localhost:9000/`.

## Executing Spark jobs
### Setting up
To begin using Spark, you must first declare the environmental variables as follows:
```
DATABRICKS_HOST = os.getenv("DATABRICKS_HOST")
DATABRICKS_API_TOKEN = os.getenv("DATABRICKS_API_TOKEN")
DATABRICKS_ORG_ID = os.getenv("DATABRICKS_ORG_ID")
DATABRICKS_PORT = os.getenv("DATABRICKS_PORT")
DATABRICKS_CLUSTER_ID = os.getenv("DATABRICKS_CLUSTER_ID")
```

Alternatively, you can also create a file called `.databricks-connect` in the root directory of viadot and add the required variables there. It should follow the following format:
```
{
  "host": "",
  "token": "",
  "cluster_id": "",
  "org_id": "",
  "port": ""
}
```
To retrieve the values, follow step 2 in this [link](https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect)

### Executing Spark functions
To begin using Spark, you must first create a Spark Session: `spark = SparkSession.builder.appName('session_name').getOrCreate()`. `spark` will be used to access all the Spark methods. Here is a list of commonly used Spark methods (WIP):
* `spark.createDataFrame(df)`: Create a Spark DataFrame from a Pandas DataFrame
* `sparkdf.write.saveAsTable("schema.table")`: Takes a Spark DataFrame and saves it as a table in Databricks.
* Ensure to use the correct schema, as it should be created and specified by the administrator
* `table = spark.sql("select * from schema.table")`: example of a simple query ran through Python

## How to contribute

1. Fork repository if you do not have write access
2. Set up locally
3. Test your changes with `pytest`
4. Submit a PR. The PR should contain the following:
    - new/changed functionality
    - tests for the changes
    - changes added to `CHANGELOG.md`
    - any other relevant resources updated (esp. `viadot/docs`)

The general flow of working for this repository in case of forking:
1. Pull before making any changes
2. Create a new branch with
```
git checkout -b <name>
```
3. Make some work on repository
4. Stage changes with
```
git add <files>
```
5. Commit the changes with
```
git commit -m <message>
```
__Note__: See out Style Guidelines for more information about commit messages and PR names

6. Fetch and pull the changes that could happen while working with
```
git fetch <remote> <branch>
git checkout <remote>/<branch>
```
7. Push your changes on repostory using
```
git push origin <name>
```
8. Use merge to finish your push to repository
```
git checkout <where_merging_to>
git merge <branch_to_merge>
```

Please follow the standards and best practices used within the library (eg. when adding tasks, see how other tasks are constructed, etc.). For any questions, please reach out to us here on GitHub.

### Style guidelines
- the code should be formatted with Black using default settings (easiest way is to use the VSCode extension)
- commit messages should:
    - begin with an emoji
    - start with one of the following verbs, capitalized, immediately after the summary emoji: "Added", "Updated", "Removed", "Fixed", "Renamed", and, sporadically, other ones, such as "Upgraded", "Downgraded", or whatever you find relevant for your particular situation
    - contain a useful description of what the commit is doing

## Set up Black for development in VSCode
Your code should be formatted with Black when you want to contribute. To set up Black in Visual Studio Code follow instructions below.
1. Install `black` in your environment by writing in the terminal:
```
pip install black
```
2. Go to the settings - gear icon in the bottom left corner and select `Settings` or type "Ctrl" + ",".
3. Find the `Format On Save` setting - check the box.
4. Find the `Python Formatting Provider` and select "black" in the drop-down list.
5. Your code should auto format on save now.


**Flit** is a simple way to put Python packages and modules on PyPI.
It tries to require less thought about packaging and help you avoid common
mistakes.
See `Why use Flit? <https://flit.readthedocs.io/en/latest/rationale.html>`_ for
more about how it compares to other Python packaging tools.

Install
-------

::

    $ python3 -m pip install flit

Flit requires Python 3 and therefore needs to be installed using the Python 3
version of pip.

Python 2 modules can be distributed using Flit, but need to be importable on
Python 3 without errors.

Usage
-----

Say you're writing a module ``foobar`` — either as a single file ``foobar.py``,
or as a directory — and you want to distribute it.

1. Make sure that foobar's docstring starts with a one-line summary of what
   the module is, and that it has a ``__version__``:

   .. code-block:: python

       """An amazing sample package!"""

       __version__ = "0.1"

2. Install flit if you don't already have it::

       python3 -m pip install flit

3. Run ``flit init`` in the directory containing the module to create a
   ``pyproject.toml`` file. It will look something like this:

   .. code-block:: ini

       [build-system]
       requires = ["flit_core >=3.2,<4"]
       build-backend = "flit_core.buildapi"

       [project]
       name = "foobar"
       authors = [{name = "Sir Robin", email = "robin@camelot.uk"}]
       dynamic = ["version", "description"]

       [project.urls]
       Home = "https://github.com/sirrobin/foobar"

   You can edit this file to add other metadata, for example to set up
   command line scripts. See the
   `pyproject.toml page <https://flit.readthedocs.io/en/latest/pyproject_toml.html#scripts-section>`_
   of the documentation.

   If you have already got a ``flit.ini`` file to use with older versions of
   Flit, convert it to ``pyproject.toml`` by running ``python3 -m flit.tomlify``.

4. Run this command to upload your code to PyPI::

       flit publish

Once your package is published, people can install it using *pip* just like
any other package. In most cases, pip will download a 'wheel' package, a
standard format it knows how to install. If you specifically ask pip to install
an 'sdist' package, it will install and use Flit in a temporary environment.

To install a package locally for development, run::

    flit install [--symlink] [--python path/to/python]

Flit packages a single importable module or package at a time, using the import
name as the name on PyPI. All subpackages and data files within a package are
included automatically.


# straxen
Streaming analysis for XENON(nT)

[![Test package](https://github.com/XENONnT/straxen/actions/workflows/pytest.yml/badge.svg?branch=master)](https://github.com/XENONnT/straxen/actions/workflows/pytest.yml)
[![Coverage Status](https://coveralls.io/repos/github/XENONnT/straxen/badge.svg)](https://coveralls.io/github/XENONnT/straxen)
[![PyPI version shields.io](https://img.shields.io/pypi/v/straxen.svg)](https://pypi.python.org/pypi/straxen/)
[![Readthedocs Badge](https://readthedocs.org/projects/straxen/badge/?version=latest)](https://straxen.readthedocs.io/en/latest/?badge=latest)
[![CodeFactor](https://www.codefactor.io/repository/github/xenonnt/straxen/badge)](https://www.codefactor.io/repository/github/xenonnt/straxen)
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/XENONnT/straxen/master.svg)](https://results.pre-commit.ci/latest/github/XENONnT/straxen/master)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5576262.svg)](https://doi.org/10.5281/zenodo.5576262)

[Straxen](https://straxen.readthedocs.io) is the analysis framework for XENONnT, built on top of the generic [strax framework](https://github.com/AxFoundation/strax). Currently it is configured for analyzing XENONnT and XENON1T data.

For installation instructions and usage information, please see the [straxen documentation](https://straxen.readthedocs.io/en/latest/setup.html).

## Further status
[![Python Versions](https://img.shields.io/pypi/pyversions/straxen.svg)](https://pypi.python.org/pypi/straxen)
[![PyPI downloads](https://img.shields.io/pypi/dm/straxen.svg)](https://pypistats.org/packages/straxen)


![SiliconCompiler](https://raw.githubusercontent.com/siliconcompiler/siliconcompiler/main/docs/_images/sc_logo_with_text.png)

[![Python CI Tests](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/python_ci.yml/badge.svg?branch=main)](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/python_ci.yml)
[![Tools CI Tests](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/tools_ci.yml/badge.svg?branch=main)](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/tools_ci.yml)
[![Daily CI Tests](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/daily_ci.yml/badge.svg?branch=main)](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/daily_ci.yml)
[![Wheels](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/wheels.yml/badge.svg?branch=main)](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/wheels.yml)
[![Lint](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/lint.yml/badge.svg?branch=main)](https://github.com/siliconcompiler/siliconcompiler/actions/workflows/lint.yml)
[![Documentation Status](https://readthedocs.org/projects/siliconcompiler/badge/?version=latest)](https://docs.siliconcompiler.com/en/latest/?badge=latest)
[![codecov](https://codecov.io/github/siliconcompiler/siliconcompiler/branch/main/graph/badge.svg?token=V5BQR42Q8C)](https://codecov.io/github/siliconcompiler/siliconcompiler)
[![Downloads](https://static.pepy.tech/personalized-badge/siliconcompiler?period=total&units=international_system&left_color=grey&right_color=blue&left_text=Downloads)](https://pepy.tech/project/siliconcompiler)

# Introduction

SiliconCompiler is a modular hardware build system ("make for silicon"). The project philosophy is to "make the complex possible while keeping the simple simple".

# Supported Technologies

| Type | Supported|
|------|----------|
|**Design Languages**| C, Verilog, SV, VHDL, Chisel, Migen/Amaranth, Bluespec
|**Simulation Tools**| Verilator, Icarus, GHDL, Xyce
|**Synthesis**| Yosys, Vivado, Synopsys, Cadence
|**ASIC APR**| OpenRoad, Synopsys, Cadence
|**FPGA APR**| VPR, nextpnr, Vivado
|**Layout Viewer**| Klayout, OpenRoad, Cadence, Synopsys
|**DRC/LVS**| Magic, Synopsys, Siemens
|**PDKs**| sky130, gf180, asap7, freepdk45, gf12lp, gf22fdx, intel16

# Getting Started

SiliconCompiler is available as wheel packages on PyPI for macOS, Windows and
Linux platforms. For working Python 3.8-3.12 environment, just use pip.

```sh
python -m pip install --upgrade siliconcompiler
```

Converting RTL into DRC clean GDS takes less than 10 lines of simple Python code.

```python
import siliconcompiler                             # import python package
chip = siliconcompiler.Chip('heartbeat')           # create chip object
chip.load_target('skywater130_demo')               # load a pre-defined target
chip.input('heartbeat.v')                          # set input sources
chip.clock('clk', period=10)                       # set constraints
chip.set('option','remote', True)                  # enable remote execution
chip.run()                                         # run compilation
chip.summary()                                     # print summary
chip.show()                                        # show layout
```

Simple designs can be compiled in a single line using the built in command line 'sc' app:

```sh
sc -remote -target "asic_demo"
```

# Why SiliconCompiler?

* **Ease-of-use**: Programmable with a simple [Python API](https://docs.siliconcompiler.com/en/stable/reference_manual/core_api.html)
* **Portability:** Powerful dynamic JSON [schema](https://docs.siliconcompiler.com/en/stable/reference_manual/schema.html) supports ASIC and FPGA design and simulation
* **Speed:** Flowgraph [execution model](https://docs.siliconcompiler.com/en/stable/user_guide/execution_model.html) enables cloud scale execution.
* **Friction-less:** [Remote execution model](https://docs.siliconcompiler.com/en/stable/development_guide/remote_processing.html) enables "zero install" compilation
* **Modularity:** [Tool abstraction layer](https://docs.siliconcompiler.com/en/latest/development_guide/tools.html) makes it easy to add/port new tools to the project.
* **Provenance:** [Compilation manifests](https://docs.siliconcompiler.com/en/stable/user_guide/data_model.html) created automatically during execution.
* **Documented:** An extensive set of auto-generated high quality [reference documents](https://docs.siliconcompiler.com/).
* **In-use:** Actively used by Zero ASIC for commercial tapeouts at advanced process nodes.

# Documentation

The full reference manual and tutorials can be found [HERE](https://docs.siliconcompiler.com/).

# License

[Apache License 2.0](LICENSE)

# How to Cite

If you want to cite our work, please use the following paper:

A. Olofsson, W. Ransohoff, N. Moroze, "[Invited: A Distributed Approach to Silicon Compilation](docs/papers/sc_dac2022.pdf)", 59th Design Automation Conference (DAC), 10-14 July 2022, San Francisco, CA, USA. Published, 7/2022.

Bibtex:
```
@inproceedings{10.1145/3489517.3530673,
author = {Olofsson, Andreas and Ransohoff, William and Moroze, Noah},
title = {A Distributed Approach to Silicon Compilation: Invited},
year = {2022},
booktitle = {Proceedings of the 59th ACM/IEEE Design Automation Conference},
pages = {1343–1346},
location = {San Francisco, California}
}
```

# Installation

Complete installation instructions are available in the [Installation Guide](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html).

To install the project from source (recommended for developers only).

```bash
git clone https://github.com/siliconcompiler/siliconcompiler
cd siliconcompiler
python -m pip install -e .             # Required install step
python -m pip install -e .[docs,test]  # Optional install step for generating docs and running tests
```

# EDA Tool Installation

Installation instructions for all external tools can be found in the
[External Tools](https://docs.siliconcompiler.com/en/stable/user_guide/installation.html#external-tools) section
of the user guide. We have included shell setup scripts (Ubuntu) for most of the supported tools.
See the [./setup](./setup) directory for a complete set of scripts and [./setup/_tools.json](./setup/_tools.json) for the currently recommended tool versions.

# Contributing

SiliconCompiler is an open-source project and welcomes contributions. To find out
how to contribute to the project, see our
[Contributing Guidelines.](./CONTRIBUTING.md)

# Issues / Bugs

We use [GitHub Issues](https://github.com/siliconcompiler/siliconcompiler/issues)
for tracking requests and bugs.

# More information

| Resources | Link|
|-----------|-----|
| **Website**|  https://www.siliconcompiler.com
| **Documentation**|  https://docs.siliconcompiler.com
| **Sources**|  https://github.com/siliconcompiler/siliconcompiler
| **Issues**|  https://github.com/siliconcompiler/siliconcompiler/issues
| **RFCs**|  https://github.com/siliconcompiler/rfcs
| **Discussion**| https://github.com/siliconcompiler/siliconcompiler/discussions


# Papyri

**Papyri** is a set of tools to build, publish (future functionality - to be done), install and render
documentation within IPython and Jupyter.

---

| Information | Links |
| :---------- | :-----|
|   Project   | [![License](https://img.shields.io/badge/License-MIT-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://opensource.org/license/mit/) [![Rendered documentation](https://img.shields.io/badge/%F0%9F%93%96%20Rendered%20docs-gray.svg?colorA=2D2A56&colorB=5936D9&style=flat.svg)](https://papyri.vercel.app/)  |
|     CI      | [![Python Package](https://github.com/jupyter/papyri/actions/workflows/python-package.yml/badge.svg)](https://github.com/jupyter/papyri/actions/workflows/python-package.yml) [![Linting](https://github.com/jupyter/papyri/actions/workflows/lint.yml/badge.svg)](https://github.com/jupyter/papyri/actions/workflows/lint.yml) |

Papyri allows:
- bidirectional crosslinking across libraries,
- navigation,
- proper reflow of user docstrings text,
- proper reflow of inline images (when rendered to html),
- proper math rendering (both in terminal and html),
- and more.

## Motivation

See some of the reasons behind the project on [this blog post](https://labs.quansight.org/blog/2021/05/rethinking-jupyter-documentation/).

Key motivation is building a set of tools to build better documentation for Python projects.
  - Uses an opinionated implementation to enable better understanding about the structure of your project.
  - Allow automatic cross-links (back and forth) between documentation across Python packages.
  - Use a documentation IR (intermediate representation) to separate building the docs from rendering the docs in many contexts.

This approach should hopefully allow a conda-forge-like model, where projects upload their IR to a given repo, a _single
website_ that contains documentation for multiple projects (without sub domains). The documentation pages can then be built with better cross-links between
projects, and _efficient_ page rebuild.

This should also allow displaying user-facing documentation on _non html_ backends (think terminal), or provide documentation in an
IDE (Spyder/Jupyterlab), without having to iframe it.

## Overview Presentation

And this [small presentation at CZI EOSS4 meeting in early november 2021](https://docs.google.com/presentation/d/1sSh44smooCiOlj0-Zrac9n5KX0K_ABBFznDmMIwUnbM/edit?usp=sharing).

## Screenshots

<detail>
<summary>Click to expand</summary>
Navigating astropy's documentation from within IPython. Note that this includes
forward refs but also backward references (i.e. which pages link to the current page.)

![](assets/astropy.gif)

Type inference and keyboard navigation in terminal: Directives are properly
rendered in terminal, examples are type inferred, clicking (or pressing enter)
on highlighted tokens would open said page (backspace navigates back).

![](assets/papyri-nav.gif)

Since Jupyter Notebook and Lab pages can render HTML, it should be possible to have
inline graphs and images when using Jupyter inline help (to be implemented). In
terminals, we replace inline images with a button/link to open images in an
external viewer (quicklook, evince, paint...)

![](assets/inline_graph.png)

Papyri has complete information about which pages link to other pages; this allows
us to create a local graph of which pages mention each other to find related topics.

Below, you can see the local connectivity graph for `numpy.zeros` (d3js, draggable, clickable).
`numpy.zeroes` links to (or is linked from) all dots present there. In green, we show other
numpy functions; in blue, we show skimage functions; in orange, we show scipy functions; in red, we show xarray functions. Arrows
between dots indicate pages which link to each other (for example ndarray is linked
from `xarray.cos`), and dot size represents the _popularity_ of a page.

![](assets/local_graph_zeroes.png)

Math expressions are properly rendered even in the terminal: here, `polyfit` is shown in IPyhton with
papyri enabled (left) and disabled (right).

![](assets/vs_math.png)
</detail>

---

## Table of contents

- [Installation](#installation)
- [Usage](#usage)
- [Rendering](#rendering)
- [Architecture](#architecture)

## Installation (not fully functional):

Some functionality is not yet available when installing from PyPI. For now you
need a [Development installation](#development-installation) to access all
features.

You'll need Python 3.8 or newer, otherwise pip will tell you it can't find any matching distribution.

Install from PyPI:

```bash
$ pip install papyri
````

Install given package documentation:

```bash
$ papyri install package_name [package_name [package_name [...]]]
```

Only numpy 1.20.0, scipy 1.5.0 and xarray 0.17.0 are currently installable and published.
For other packages you will need to build locally which is a much more involved
process.

Run IPython terminal with Papyri as an extension:

```
$ ipython --ext papyri.ipython
```

This will augment the `?` operator to show better documentation (when installed with `papyri install ...`

*Papyri does not completely build its own docs yet, but you might be able to view a static rendering of it
[here](https://pydocs.github.io/). It is not yet automatically built, so might be out of date.*

### Development installation

You may need to get a modified version of numpydoc depending on the stage of development. You will need [pip >
21.3](https://pip.pypa.io/en/stable/news/#v21-3-1) if you want to make editable installs.

```
# clone this repo
# cd this repo
pip install -e .
```

Some functionality requires ``tree_sitter_rst``. To build the TreeSitter rst parser:

```bash
$ git submodule update --init
$ papyri build-parser
```

[Look at CI file](https://github.com/jupyter/papyri/blob/main/.github/workflows/python-package.yml) if those instructions are not up to date.

Note that papyri still uses a custom parser which will be removed in the future to rely mostly on TreeSitter.

### Testing

Install extra development dependencies by running:

```bash
$ pip install -r requirements-dev.txt
```

Run tests using

```bash
$ pytest
```

## Usage

Papyri relies on three steps:

 - IR generation (executed by package maintainers);
 - IR installation (executed by end users or via pip/conda);
 - IR rendering (usually executed by the IDE, CLI/webserver).

### IR Generation (`papyri gen`)

This is the step you want to trigger if you are building documentation using Papyri for a library you maintain. Most
likely as an end user you will not have to issue this step and can install pre-published documentation bundles.
This step is likely to occur only once per new release of a project.

The Toml files in `examples` will give you example configurations from some existing libraries.

```
$ ls -1 examples/*.toml
examples/IPython.toml
examples/astropy.toml
examples/dask.toml
examples/matplotlib.toml
examples/numpy.toml
examples/papyri.toml
examples/scipy.toml
examples/skimage.toml
```

Right now these files lives in papyri but would likely be in relevant repositories under `docs/papyri.toml` later on.

> [!NOTE]
> It is _slow_ on full numpy/scipy; use `--no-infer` (see below) for a subpar but faster experience.

Use `papyri gen <path to example file>`

for example:

```
$ papyri gen examples/numpy.toml
```

```
$ papyri gen examples/scipy.toml
```

This will create intermediate docs files in in `~/.papyri/data/<library name>_<library_version>`. See [Generation](#generation-papyri-gen) for more details.

You can also generate intermediate docs files for a subset of objects using the `--only` flag. For example:

```
$ papyri gen examples/numpy.toml --only numpy:einsum
```

> [!IMPORTANT]
> To avoid ambiguity, papyri uses [fully qualified names](#qualified-names) to refer to objects. This means that you need to use `numpy:einsum` instead of `einsum` or `numpy.einsum` to refer to the `einsum` function in the `numpy` module, for example.

### Installation/ingestion

The installation/ingestion of documentation bundles is the step that makes all
bundles "aware" of each other, and allows crosslinking/indexing to work.

We'll reserve the term "install" and "installation" for when you download pre-build
documentation bundle from an external source and give only the package name
– which is not completely implemented yet.

You can ingest local folders with the following command:

```
$ papyri ingest ~/.papyri/data/<path to folder generated at previous step>
```

This will crosslink the newly generated folder with the existing ones.
Ingested data can be found in  `~/.papyri/ingest/` but you are not supposed to
interact with this folder with tools external to papyri.

There are currently a couple of pre-built documentation bundles that can be
pre-installed, but are likely to break with each new version of papyri. We
suggest you use the developer installation and ingestion procedure for now.

## Rendering

The last step of the papyri pipeline is to render the docs, or the subset that
is of interest to you. This will likely be done by your favorite IDE, probably
just in time when you explore documentation. Nonetheless, we've
implemented a couple of external renderers to help debug issues.

> [!WARNING]
> Many rendering methods currently require papyri's own docs to be built and ingested first.

```
$ papyri gen examples/papyri.toml
$ papyri ingest ~/.papyri/data/papyri_0.0.7  # or any current version
```

Or you can try to pre-install an old papyri doc bundle:

```
$ papyri install papyri
```

### Standalone HTML rendering

To see the rendered documentation for all packages previously ingested, run

```bash
$ papyri serve
```

This will start a live server that will render the pages on the fly.

If you need to render static versions of the pages, use either of the following
commands:

```bash
$ papyri render  # render all the html pages statically in ~/.papyri/html
$ papyri serve-static # start a http.server with the proper root to serve above files.
```

### Rich terminal rendering

To render the documentation for a single object on a terminal, use

```
$ papyri rich <fully qualified name>
```

For example:

```
$ papyri rich numpy:einsum  # note the colon for the fully qualified name.
```

To use the experimental interactive Textual interface in the terminal, use

```
$ papyri textual <fully qualified name>
```

### IPython extension

To run `papyri` as an IPython extension, run:

```
$ ipython --ext papyri.ipython
```

This will start an IPython session with an augmented `?` operator.

### Jupyter extension

In progress.

### More commands

You can run `papyri` without a command to see all currently available commands.

## Papyri - Name's meaning

See the legendary [Villa of Papyri](https://en.wikipedia.org/wiki/Villa_of_the_Papyri), which get its name from its
collection of many papyrus scrolls.

## Architecture

### Generation (`papyri gen`)

Collects the documentation of a project into a *DocBundle* -- a number of
*DocBlobs* (currently json files), with a defined semantic structure, and
some metadata (version of the project this documentation refers to, and
potentially some other blobs).

During the generation a number of normalisation and inference steps can and
should happen. For example:

  - Using type inference into the `Examples` sections of docstrings and storing
    those as pairs (token, reference), so that you can later decide that
    clicking on `np.array` in an example brings you to numpy array
    documentation; whether or not we are currently in the numpy documentation;
  - Parsing "See Also" into a well defined structure;
  - Running examples to generate images for docs with images (partially
    implemented);
  - Resolve local references. For example, when building the NumPy docs,
    `zeroes_like` is non-ambiguous and should be normalized to
    `numpy.zeroes_like`. Similarly, `~.pyplot.histogram`, should be normalized
    to `matplotlib.pyplot.histogram` as the **target** and `histogram` as the
    text.

The Generation step is likely project specific, as there might be import
conventions that are defined per-project and should not need to be repeated
(`import pandas as pd`, for example.)

The generation step is likely to be the most time consuming, and for each
project, results in the following outputs:

- A `papyri.json` file, which is a list of unique qualified names corresponding
  to the documented objects and some metadata;
- A `toc.json` file, ?
- An `assets` folder, containing all the images generated during the
  generation;
- A `docs` folder, ?
- An `examples` folder, ?
- A `module` folder, containing one json file per documented object.

After the generation step, *what should have been processed*?

### Ingestion (`papyri ingest`)

The ingestion step takes a DocBundle and/or DocBlobs and adds them into a graph
of known items; the ingestion is critical to efficiently build the collection
graph metadata and understand which items refers to which. This allows the
following:

 - Update the list of backreferences to a *DocBundle*;
 - Update forward references metadata to know whether links are valid.

Currently the ingestion loads all in memory and updates all the bundle in place
but this can likely be done more efficiently.

A lot more can likely be done at larger scale, like detecting if documentation
has changed in previous versions to infer for which versions of a library this
documentation is valid.

There is also likely some curating that might need to be done at that point, as
objects such as `numpy.array` have an extremely large number of back-references.

### Qualified names

To avoid ambiguity when referring to objects, papyri uses the
*fully qualified name* of the object for its operations. This means that instead
of a dot (`.`), we use a colon (`:`) to separate the module part from the
object's name and sub attributes.

To understand why we need this, assume the following situation: a top level
`__init__` imports a function from a submodule that has the same name as the
submodule:

```
# project/__init__.py
from .sub import sub
```

This submodule defines a class (here we use lowercase for the example):

```
# project/sub.py
class sub:
    attribute:str
attribute = 'hello'
```

and a second submodule is defined:
```
# project/attribute.py
None
```

Using qualified names only with dots (`.`) can make it difficult to find out
which object we are referring to, or implement the logic to find the object.
For example, to get the object `project.sub.attribute`, one would do:

```
import project
x = getattr(project, 'sub')
getattr(x, 'attribute')
```

But here, because of the `from .sub import sub`, we end up getting the class
attribute instead of the module. This ambiguity is lifted with a `:` as we now
explicitly know the module part, and `package.sub.attribute` is distinct from
`package.sub:attribute`. Note that `package:sub.attribute` is also
non-ambiguous, even if not the right fully qualified name for an object.

Moreover, using `:` as a separator makes the implementation much easier, as
in the case of `package.sub:attribute` it is possible to directly execute
`importlib.import_module('package.sub')` to obtain a reference to the `sub`
submodule, without try/except or recursive `getattr` checking for the type of an
object.

### Tree sitter information

See https://tree-sitter.github.io/tree-sitter/creating-parsers

### When things don't work !

#### `SqlOperationalError`:

- The DB schema likely have changed, try: `rm -rf ~/.papyri/ingest/`.


![FlexMeasures Logo Light](https://github.com/FlexMeasures/screenshots/blob/main/logo/flexmeasures-horizontal-color.svg#gh-light-mode-only)
![FlexMeasures Logo Dark](https://github.com/FlexMeasures/screenshots/blob/main/logo/flexmeasures-horizontal-dark.svg#gh-dark-mode-only)

[![License](https://img.shields.io/github/license/seitabv/flexmeasures?color=blue)](https://github.com/FlexMeasures/flexmeasures/blob/main/LICENSE)
![lint-and-test](https://github.com/FlexMeasures/flexmeasures/workflows/lint-and-test/badge.svg)
[![Pypi Version](https://img.shields.io/pypi/v/flexmeasures.svg)](https://pypi.python.org/pypi/flexmeasures)
[![](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Documentation Status](https://readthedocs.org/projects/flexmeasures/badge/?version=latest)](https://flexmeasures.readthedocs.io/en/latest/?badge=latest)
[![Coverage](https://coveralls.io/repos/github/FlexMeasures/flexmeasures/badge.svg)](https://coveralls.io/github/FlexMeasures/flexmeasures)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6095/badge)](https://bestpractices.coreinfrastructure.org/projects/6095)

*FlexMeasures* is an intelligent EMS (energy management system) to optimize behind-the-meter energy flexibility.
Build your smart energy apps & services with FlexMeasures as backend for real-time orchestration!

In a nutshell, FlexMeasures turns data into optimized schedules for flexible assets like batteries and heat pumps, or for flexible industry processes:

![The most simple view of FlexMeasures, turning data into schedules](https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/simple-flexEMS.png)

Here is why using FlexMeasures is a great idea:

- Developing energy flexibility apps & services (e.g. to enable demand response) is crucial, but expensive.
- FlexMeasures reduces development costs with real-time data intelligence & integrations, uncertainty models and developer support such as API/UI and plugins.

![High-level overview of FlexMeasures as an EMS for energy flexibility apps, using plugins to fit a given use case](https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/overview-flexEMS.png)

So why optimise the schedules of flexible assets? Because planning ahead allows flexible assets to serve the whole system with their flexibility, e.g. by shifting energy consumption to other times.
For the asset owners, this creates CO₂ savings but also monetary value (e.g. through self-consumption, dynamic tariffs and grid incentives). FlexMeasures thrives to be applicable in cases with multiple sources of value ("value stacking") and multiple types of assets (e.g. home/office/factory).

As possible users, we see energy service companies (ESCOs) who want to build real-time apps & services around energy flexibility for their customers, or medium/large industrials who are looking for support in their internal digital tooling. However, even small companies and hobby projects might find FlexMeasures useful!

## How does FlexMeasures enable rapid development of energy flexibility apps?

FlexMeasures is designed to help with three basic needs of developers in the energy flexibility domain:

### I need help with integrating real-time data and continuously computing new data

FlexMeasures is designed to make decisions based on data in an automated way. Data pipelining and dedicated machine learning tooling is crucial.

- API/CLI functionality to read in time series data
- Extensions for integrating 3rd party data, e.g. from [ENTSO-E](https://github.com/SeitaBV/flexmeasures-entsoe) or [OpenWeatherMap](https://github.com/SeitaBV/flexmeasures-openweathermap)
- Forecasting for the upcoming hours
- Schedule optimization for flexible assets

### It's hard to correctly model data with different sources, resolutions, horizons and even uncertainties

Much developer time is spent correcting data and treating it correctly, so that you know you are computing on the right knowledge.

FlexMeasures is built on the [timely-beliefs framework](https://github.com/SeitaBV/timely-beliefs), so we model this real-world aspect accurately:

- Expected data properties are explicit (e.g. unit, time resolution)
- Incoming data is converted to fitting unit and time resolution automatically
- FlexMeasures also stores who thought that something happened (or that it will happen), and when they thought so
- Uncertainty can be modelled (useful for forecasting)

### I want to build new features quickly, not spend days solving basic problems

Building customer-facing apps & services is where developers make impact. We make their work easy.

- FlexMeasures has well-documented API endpoints and CLI commands to interact with its model and data
- You can extend it easily with your own logic by writing plugins
- A backend UI shows you your assets in maps and your data in plots. There is also support for plots to be available per API, for integration in your own frontend
- Multi-tenancy ― model multiple accounts on one server. Data is only seen/editable by authorized users in the right account

## Getting started

Head over to our [documentation](https://flexmeasures.readthedocs.io), e.g. the [getting started guide](https://flexmeasures.readthedocs.io/en/latest/getting-started.html) or the [5-minute tutorial](https://flexmeasures.readthedocs.io/en/latest/tut/toy-example-from-scratch.html). Or find more information on [FlexMeasures.io](https://flexmeasures.io).

See also [Seita's Github profile](https://github.com/SeitaBV), e.g. for FlexMeasures plugin examples.

## Development & community

FlexMeasures was initiated by [Seita BV](https://www.seita.nl) in The Netherlands in order to make sure that smart backend software is available to all parties working with energy flexibility, no matter where they are working on their local energy transition.

We made FlexMeasures freely available under the Apache2.0 licence and it is now [an incubation project at the Linux Energy Foundation](https://www.lfenergy.org/projects/flexmeasures/).

Within the FlexMeasures project, [we welcome contributions](https://github.com/FlexMeasures/tsc/blob/main/CONTRIBUTING.md). You can also [learn more about our governance](https://github.com/Flexmeasures/tsc/blob/main/GOVERNANCE.md).

You can connect with the community here on GitHub (e.g. by creating an issue), on [the mailing list](https://lists.lfenergy.org/g/flexmeasures), on [the FlexMeasures channel within the LF Energy Slack](https://slack.lfenergy.org/) or [by contacting the current maintainers](https://seita.nl/who-we-are/#contact).


#  Ikabot - (Ikariam Bot)
[![Downloads](https://static.pepy.tech/badge/ikabot)](https://pepy.tech/project/ikabot) ![GitHub Repo stars](https://img.shields.io/github/stars/physics-sec/ikabot?style=flat) ![licence](https://img.shields.io/github/license/physics-sec/ikabot?style=flat)

Ikabot, written in Python, is a cross-platform program that brings you the power of a premium Ikariam account without the need to spend ambrosia. Our toolset is designed to simplify every aspect of your gaming experience, from efficient city management, strategic warfare, and simplified alerts.

Discover its features [here](https://github.com/physics-sec/ikabot/wiki/Features) and enhance your Ikariam adventure with Ikabot.

#  Getting Started
- Checkout the [Getting Started guide](https://github.com/physics-sec/ikabot/wiki/Getting-Started)

#  Contribution Guidelines

- Before contributing code, read the [Contribution Guidelines](.github/CONTRIBUTING.md)
  - GitHub issues are for bugs and feature requests.
  - For questions, help and chat in general, join our [Discord community](https://discord.gg/3hyxPRj).
- Want to say thanks?
  - Hit the :star: Star :star: button

#  Contributors

This project exists thanks to all the people who contribute.


[![Unit Tests](https://img.shields.io/github/actions/workflow/status/icecube/pisa/.github/workflows/pythonpackage.yml?label=unit%20tests)](https://github.com/icecube/pisa/actions/workflows/pythonpackage.yml)
[![Pull Requests](https://img.shields.io/github/issues-pr/icecube/pisa)](https://github.com/icecube/pisa/pulls)
[![Activity](https://img.shields.io/github/commit-activity/m/icecube/pisa)](https://github.com/icecube/pisa/pulse)
[![Contributors](https://img.shields.io/github/contributors/icecube/pisa)](https://github.com/icecube/pisa/graphs/contributors)
[![Repo Stars](https://img.shields.io/github/stars/icecube/pisa?style=social)](https://github.com/icecube/pisa/stargazers)

[Introduction](pisa/README.md) |
[Installation](INSTALL.md) |
[Documentation](https://icecube.github.io/pisa/) |
[Terminology](pisa/glossary.md) |
[License](LICENSE)

PISA is a software written to analyze the results (or expected results) of an experiment based on Monte Carlo simulation.

In particular, PISA was written by and for the IceCube Collaboration for analyses employing the [IceCube Neutrino Observatory](https://icecube.wisc.edu/), including the [DeepCore](https://arxiv.org/abs/1109.6096) and the planned [Upgrade]([https://arxiv.org/abs/2307.15295](https://arxiv.org/pdf/1908.09441.pdf)) low-energy in-fill arrays.

> [!NOTE]
> However, any experiment can make use of PISA for analyzing expected and actual results.

PISA was originally developed to cope with low-statistics Monte Carlo (MC) by using parameterizations of the MC and operate on histograms of the data rather than directly reweighting the MC. However, PISA's methods apply equally well to high-MC situations, and PISA also performs traditional reweighted-MC analysis as well.

If you use PISA, please cite our publication ([e-Print available here](https://arxiv.org/abs/1803.05390)):
```
Computational Techniques for the Analysis of Small Signals
in High-Statistics Neutrino Oscillation Experiments
IceCube Collaboration - M.G. Aartsen et al.
Mar 14, 2018
Published in: Nucl.Instrum.Meth.A 977 (2020) 164332
```

# Quick start

## Installation

```shell
git clone git@github.com:icecube/pisa.git
cd pisa
pip install -e .
```

For detailed installation instructions and common issues see [Installation](INSTALL.md)

## Minimal Example

Producing some oscillograms

```python
from pisa.core import Pipeline
import matplotlib.pyplot as plt
```

    << PISA is running in single precision (FP32) mode; numba is running on CPU (single core) >>

Instantiate a `Pipeline` or multiple pipelines in a `DistributionMaker` using PISA config files

```python
template_maker = Pipeline("settings/pipeline/osc_example.cfg")
```

Run the pipleine with nominal settings

```python
template_maker.run()
```

Get the oscillation probabilities

```python
outputs = template_maker.data.get_mapset('prob_mu')
```

Plot some results

```python
fig, axes = plt.subplots(figsize=(18, 5), ncols=3)
outputs['nue_cc'].plot(ax=axes[0], cmap='RdYlBu_r', vmin=0, vmax=1);
outputs['numu_cc'].plot(ax=axes[1], cmap='RdYlBu_r', vmin=0, vmax=1);
outputs['nutau_cc'].plot(ax=axes[2], cmap='RdYlBu_r', vmin=0, vmax=1);
```

![png](README_files/README_10_0.png)

# Contributions

Contributors are listed specifically [here](CONTRIBUTORS.md), while the used external software is summarized [here](EXTERNAL_ATTRIBUTION.md).


# PowerDNS-Admin

A PowerDNS web interface with advanced features.

[![CodeQL](https://github.com/PowerDNS-Admin/PowerDNS-Admin/actions/workflows/codeql-analysis.yml/badge.svg?branch=master)](https://github.com/PowerDNS-Admin/PowerDNS-Admin/actions/workflows/codeql-analysis.yml)
[![Docker Image](https://github.com/PowerDNS-Admin/PowerDNS-Admin/actions/workflows/build-and-publish.yml/badge.svg?branch=master)](https://github.com/PowerDNS-Admin/PowerDNS-Admin/actions/workflows/build-and-publish.yml)

#### Features:

- Provides forward and reverse zone management
- Provides zone templating features
- Provides user management with role based access control
- Provides zone specific access control
- Provides activity logging
- Authentication:
  - Local User Support
  - SAML Support
  - LDAP Support: OpenLDAP / Active Directory
  - OAuth Support: Google / GitHub / Azure / OpenID
- Two-factor authentication support (TOTP)
- PDNS Service Configuration & Statistics Monitoring
- DynDNS 2 protocol support
- Easy IPv6 PTR record editing
- Provides an API for zone and record management among other features
- Provides full IDN/Punycode support

## [Project Update - PLEASE READ!!!](https://github.com/PowerDNS-Admin/PowerDNS-Admin/discussions/1708)

## Running PowerDNS-Admin

There are several ways to run PowerDNS-Admin. The quickest way is to use Docker.
If you are looking to install and run PowerDNS-Admin directly onto your system, check out
the [wiki](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/docs/wiki/) for ways to do that.

### Docker

Here are two options to run PowerDNS-Admin using Docker.
To get started as quickly as possible, try option 1. If you want to make modifications to the configuration option 2 may
be cleaner.

#### Option 1: From Docker Hub

To run the application using the latest stable release on Docker Hub, run the following command:

```
$ docker run -d \
    -e SECRET_KEY='a-very-secret-key' \
    -v pda-data:/data \
    -p 9191:80 \
    powerdnsadmin/pda-legacy:latest
```

This creates a volume named `pda-data` to persist the default SQLite database with app configuration.

#### Option 2: Using docker-compose

1. Update the configuration
   Edit the `docker-compose.yml` file to update the database connection string in `SQLALCHEMY_DATABASE_URI`.
   Other environment variables are mentioned in
   the [AppSettings.defaults](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/powerdnsadmin/lib/settings.py) dictionary.
   To use a Docker-style secrets convention, one may append `_FILE` to the environment variables with a path to a file
   containing the intended value of the variable (e.g. `SQLALCHEMY_DATABASE_URI_FILE=/run/secrets/db_uri`).
   Make sure to set the environment variable `SECRET_KEY` to a long, random
   string (https://flask.palletsprojects.com/en/1.1.x/config/#SECRET_KEY)

2. Start docker container
   ```
   $ docker-compose up
   ```

You can then access PowerDNS-Admin by pointing your browser to http://localhost:9191.

## Screenshots

![dashboard](docs/screenshots/dashboard.png)

## Support

**Looking for help?** Try taking a look at the project's
[Support Guide](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/.github/SUPPORT.md) or joining
our [Discord Server](https://discord.powerdnsadmin.org).

## Security Policy

Please see our [Security Policy](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/SECURITY.md).

## Contributing

Please see our [Contribution Guide](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/docs/CONTRIBUTING.md).

## Code of Conduct

Please see our [Code of Conduct Policy](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/docs/CODE_OF_CONDUCT.md).

## License

This project is released under the MIT license. For additional
information, [see the full license](https://github.com/PowerDNS-Admin/PowerDNS-Admin/blob/master/LICENSE).