# [pysal/spaghetti](http://pysal.org/spaghetti/)

# *SPA*tial *G*rap*H*s: n*ET*works, *T*opology, & *I*nference

Spaghetti is an open-source Python library for the analysis of network-based spatial data. Originating from the `network` module in [PySAL (Python Spatial Analysis Library)](http://pysal.org), it is under active development for the inclusion of newly proposed methods for building graph-theoretic networks and the analysis of network events.

*An example of a network's [minimum spanning tree](https://pysal.org/spaghetti/generated/spaghetti.spanning_tree.html#spaghetti.spanning_tree):*

|[![PyPI version](https://badge.fury.io/py/spaghetti.svg)](https://badge.fury.io/py/spaghetti)| [![Conda Version](https://img.shields.io/conda/vn/conda-forge/spaghetti.svg)](https://anaconda.org/conda-forge/spaghetti) | ![tag](https://img.shields.io/github/v/release/pysal/spaghetti?include_prereleases&sort=semver) | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pysal/spaghetti/main)
|:---:|:---:|:---:|:---:|
|[![Downloads](https://pepy.tech/badge/spaghetti)](https://pepy.tech/project/spaghetti) | [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/spaghetti.svg)](https://anaconda.org/conda-forge/spaghetti) | [![Documentation](https://img.shields.io/static/v1.svg?label=docs&message=current&color=9cf)](http://pysal.org/spaghetti/) | [![Discord](https://img.shields.io/badge/Discord-join%20chat-7289da?style=flat&logo=discord&logoColor=cccccc&link=https://discord.gg/BxFTEPFFZn)](https://discord.gg/BxFTEPFFZn)
| ![Pypi python versions](https://img.shields.io/pypi/pyversions/spaghetti.svg) | [![Conda Recipe](https://img.shields.io/badge/recipe-spaghetti-red.svg)](https://github.com/conda-forge/spaghetti-feedstock) | [![codecov](https://codecov.io/gh/pysal/spaghetti/branch/main/graph/badge.svg)](https://codecov.io/gh/pysal/spaghetti) | [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
| [![Continuous Integration](https://github.com/pysal/spaghetti/actions/workflows/testing.yml/badge.svg)](https://github.com/pysal/spaghetti/actions/workflows/testing.yml) | [![status](https://joss.theoj.org/papers/52b8d0c81bbf311465b45bfc26379e74/status.svg)](https://joss.theoj.org/papers/52b8d0c81bbf311465b45bfc26379e74) | [![DOI](https://zenodo.org/badge/88305306.svg)](https://zenodo.org/badge/latestdoi/88305306) | [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

## Examples

The following are a selection of some examples that can be launched individually as interactive binders from the links on their respective pages. Additional examples can be found in the [Tutorials](https://pysal.org/spaghetti/tutorials.html) section of the documentation. See the [`pysal/notebooks`](http://pysal.org/notebooks) project for a [`jupyter-book`](https://github.com/choldgraf/jupyter-book) version of this repository.
* [Quickstart](https://pysal.org/spaghetti/notebooks/quickstart.html)
* [Shortest Path Visualization](https://pysal.org/spaghetti/notebooks/shortest-path-visualization.html)
* [Caveats](https://pysal.org/spaghetti/notebooks/caveats.html)

## Installation

Python >= [3.10](https://docs.python.org/3.10/) is tested for support by `spaghetti`. Please make sure that you are operating in a Python >= 3.10 environment.

**Installing with `conda` via [`conda-forge`](https://github.com/conda-forge/spaghetti-feedstock) (highly recommended)**

To install `spaghetti` and all its dependencies, we recommend using the [`conda`](https://docs.conda.io/en/latest/)
manager, specifically with the [`conda-forge`](https://conda-forge.org) channel. This can be obtained by installing the [`Anaconda Distribution`](https://docs.continuum.io/anaconda/) (a free Python distribution for data science), or through [`miniconda`](https://docs.conda.io/en/latest/miniconda.html) (minimal distribution only containing Python and the conda package manager).

Using `conda`, `spaghetti` can be installed as follows:
```
$ conda config --set channel_priority strict
$ conda install --channel conda-forge spaghetti
```
Also, `geopandas` provides [a nice example](https://geopandas.readthedocs.io/en/latest/getting_started/install.html#creating-a-new-environment) to create a fresh environment for working with spatial data.

**Installing with [`PyPI`](https://pypi.org/project/spaghetti/)**
```
$ pip install spaghetti
```
*or* download the source distribution (`.tar.gz`) and decompress it to your selected destination. Open a command shell and navigate to the decompressed folder.
```
$ pip install .
```

***Warning***

When installing via `pip`, you have to ensure that the required dependencies for `spaghetti` are installed on your operating system. Details on how to install these packages are linked below. Using `conda` (above) avoids having to install the dependencies separately.

Install the most current development version of `spaghetti` by running:

```
$ pip install git+https://github.com/pysal/spaghetti
```

## Requirements

- [`esda`](https://pysal.org/esda/)
- [`geopandas`](https://geopandas.org/en/stable/)
- [`libpysal`](https://pysal.org/libpysal/)
- [`libspatialindex`](https://libspatialindex.org/en/stable/)
- [`numpy`](https://numpy.org/devdocs/)
- [`rtree`](https://rtree.readthedocs.io/en/stable/)
- [`scipy`](http://scipy.github.io/devdocs/)
- [`shapely`](https://shapely.readthedocs.io/en/stable/)

## History

`spaghetti` was
created and has evolved in line with the Python Spatial Analysis Library ecosystem for
the specific purpose of utilizing the functionality of spatial weights in
[`libpysal`](https://pysal.org/libpysal/) for generating network segment contiguity objects.
The PySAL project was started in the mid-2000s when installation was difficult to maintain.
Due to the non-triviality of relying on dependencies to secondary packages, a conscious
decision was made to limit dependencies and build native PySAL data structures in cases
where at all possible. Therefore, the original `pysal.network` submodule was developed to
address the need for integrating support for network data structures with PySAL weights
data structures, with the target audience being spatial data scientists and anyone
interested in investigating network-centric phenomena within PySAL. Owing to the
co-development of network functionality found within `spaghetti` and the evolution of
the wider PySAL ecosystem, today, the package provides specialized network functionality
that easily integrates with the rest of PySAL. This allows users of `spaghetti`’s network
functionality to access spatial analysis functionality that complements network analysis,
such as spatial statistical tools with `esda` and integration with core components of
`libpysal`: `libpysal.weights` (mentioned above),
`libpysal.cg` (computational geometry and data structures),
`libpysal.io` (input-output), and `libpysal.examples` (built-in example data).

##  Contribute

PySAL-spaghetti is under active development and contributors are welcome.

If you have any suggestions, feature requests, or bug reports, please open new [issues](https://github.com/pysal/spaghetti/issues) on GitHub. To submit patches, please review [PySAL's documentation for developers](https://pysal.org/docs/devs/), the PySAL [development guidelines](https://github.com/pysal/pysal/wiki), the `spaghetti` [contributing guidelines](https://github.com/pysal/spaghetti/blob/main/.github/CONTRIBUTING.md) before  opening a [pull request](https://github.com/pysal/spaghetti/pulls). Once your changes get merged, you’ll automatically be added to the [Contributors List](https://github.com/pysal/spaghetti/graphs/contributors).

## Support

If you are having issues, please [create an issue](https://github.com/pysal/spaghetti/issues), start a [discussion](https://github.com/pysal/spaghetti/discussions), or talk to us in [PySAL's Discord channel](https://discord.gg/BxFTEPFFZn). All questions, comments, & discussions should happen in a public forum, where possible. Private messages and emails will not be answered in a substantive manner.

## Code of Conduct

As a PySAL-federated project, `spaghetti` follows the [Code of Conduct](https://github.com/pysal/governance/blob/main/conduct/code_of_conduct.rst) under the [PySAL governance model](https://github.com/pysal/governance).

## License

The project is licensed under the [BSD 3-Clause license](https://github.com/pysal/spaghetti/blob/main/LICENSE.txt).

## BibTeX Citation

If you use PySAL-spaghetti in a scientific publication, we would appreciate using the following citations:

```
@article{Gaboardi2021,
    doi       = {10.21105/joss.02826},
    url       = {https://doi.org/10.21105/joss.02826},
    year      = {2021},
    publisher = {The Open Journal},
    volume    = {6},
    number    = {62},
    pages     = {2826},
    author    = {James D. Gaboardi and Sergio Rey and Stefanie Lumnitz},
    title     = {spaghetti: spatial network analysis in PySAL},
    journal   = {Journal of Open Source Software}
}

@misc{Gaboardi2018,
    author    = {Gaboardi, James D. and Laura, Jay and Rey, Sergio and
                 Wolf, Levi John and Folch, David C. and Kang, Wei and
                 Stephens, Philip and Schmidt, Charles},
    month     = {oct},
    year      = {2018},
    title     = {pysal/spaghetti},
    url       = {https://github.com/pysal/spaghetti},
    doi       = {10.5281/zenodo.1343650},
    keywords  = {graph-theory,network-analysis,python,spatial-networks,topology}
}
```

## Funding
This project is/was partially funded through:

[](https://atlantardc.wordpress.com) Atlanta Research Data Center: [A Polygon-Based Approach to Spatial Network Allocation](https://atlantardc.files.wordpress.com/2018/05/ardc-newsletter_2018_2.pdf)

[](https://www.nsf.gov/index.jsp) National Science Foundation Award #1825768: [National Historical Geographic Information System](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1825768&HistoricalAwards=false)


# Representational Similarity Analysis 3.0

[![Documentation Status](https://readthedocs.org/projects/rsatoolbox/badge/?version=latest)](https://rsatoolbox.readthedocs.io/en/latest/?badge=latest)
[![PyPI version](https://badge.fury.io/py/rsatoolbox.svg)](https://badge.fury.io/py/rsatoolbox)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/rsatoolbox/badges/version.svg)](https://anaconda.org/conda-forge/rsatoolbox)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/626ca9ec9f75485a9f73783c02710b1f)](https://www.codacy.com/gh/rsagroup/rsatoolbox?utm_source=github.com&utm_medium=referral&utm_content=rsagroup/rsatoolbox&utm_campaign=Badge_Grade)
[![CodeFactor](https://www.codefactor.io/repository/github/rsagroup/rsatoolbox/badge)](https://www.codefactor.io/repository/github/rsagroup/rsatoolbox)

Conceived during the RSA retreat 2019 in Blue Mountains.

[Documentation](https://rsatoolbox.readthedocs.io/)

#### Getting Started

To install the latest stable version of rsatoolbox with pip:

```sh
pip install rsatoolbox
```

or with conda:

```sh
conda install -c conda-forge rsatoolbox
```

here is a simple code sample:

```python
import numpy, rsatoolbox
data = rsatoolbox.data.Dataset(numpy.random.rand(10, 5))
rdms = rsatoolbox.rdm.calc_rdm(data)
rsatoolbox.vis.show_rdm(rdms)
```


[![Jazzband](https://jazzband.co/static/img/badge.svg)](https://jazzband.co/)
[![Test & Deploy](https://github.com/jazzband/website/actions/workflows/test-and-deploy.yml/badge.svg)](https://github.com/jazzband/website/actions/workflows/test-and-deploy.yml)
[![Calver](https://img.shields.io/badge/calver-YY.MM.PATCH-22bfda.svg)](https://calver.org/)

The code that renders https://jazzband.co.

## Installation

Copy `.end-dist` to `.env`.

Install Docker, docker-compose and NPM.

Run `make build`. This will create a set of Docker containers with all backends
and dependencies.

The Jazzband site uses a self-signed TLS certificate for development to be able
to reproduce the production environment as close as possible. To that effect
it's required to install [`mkcert`](https://github.com/FiloSottile/mkcert)
in your system's certificate trust store (once). To do that install `mkcert`
by following the installation instructions and then run `make trust`.

In case the embedded self-signed certificates are outdated you can recreate
them by running `make cert`.

## Running

Run `make run` to run the development server and worker. The website will be available
at https://localhost:5000.

## License

The content of this project is licensed under the
[Attribution-NonCommercial-ShareAlike 4.0 International] license, and
the underlying source code used to format and display that content is licensed
under the MIT license.

[add-to-org]: https://github.com/benbalter/add-to-org
[Attribution-NonCommercial-ShareAlike 4.0 International]: https://creativecommons.org/licenses/by-nc-sa/4.0/


# Arenadata Cluster Manager

That is Arenadata Cluster Manager Project aka Chapelnik

# Documentation

[ArenaData ADCM Documentation](http://docs.arenadata.io/adcm/)

# Develop

All standard Django commands are available.

Run dev server for the first time with these commands:
1. `manage.py migrate`
2. `init_db.py`
3. `manage.py upgraderole`
4. `manage.py runserver --insecure`

Re-run them when needed/applicable.

# Sources

## Dirs

* assemble - information about the way we build product
* python - core django modules and python functions
* docs
* go - golang part of application. There is a status server here now.
* test
* spec - specification in form of Sphinx RST
* adcm-web - UI source

# Build logic

There is a Makefile in repo. It could be used for building application.

## Fast start with make

You have to have GNU Make on your host and Docker daemon accessible for a user. Also, you have to have access to ci.arenadata.io

```sh
# Clone repo
git clone https://github.com/arenadata/adcm

cd adcm

# Run build process
make build
```

That will be an image hub.adsw.io/adcm/adcm:<branch_name> as a result of the operation above.

## Makefile description

Makefile has self-documented help message. Just type.

```sh
$ make
buildbaseimage                 Build base image for ADCM's container. That is alpine with all packages.
build                          Build final docker image and all depended targets except baseimage.
buildjs                        Build client side js/html/css in directory wwwroot
buildss                        Build status server
clean                          Cleanup. Just a cleanup.
describe                       Create .version file with output of describe
help                           Shows that help
build2js                       For new design and api v2: Build client side js/html/css in directory wwwroot
build2                         For new design and api v2: Build final docker image and all depended targets except baseimage.
```

And check out the description for every operation available.

## Pre-commit hook

We are using black, pylint and pre-commit to care about code formatting and linting.

So you have to install pre-commit hook before you do something with code.

``` sh
pip install pre-commit # Or do it with your preffered way to install pip packages
pre-commit install
```

After this you will see invocation of black and pylint on every commit.

## Running ADCM using SQLite

1. Start container:

    ```shell
    docker run -d --restart=always -p 8000:8000 -v /opt/adcm:/adcm/data --name adcm hub.arenadata.io/adcm/adcm:latest
    ```

    Use `-v /opt/adcm:/adcm/data:Z` for SELinux

## Running ADCM using client PostgreSQL DB
_PostgreSQL must be version 11 or newer - JSONB field used_

1. Start container:
   ```shell
   docker run -d --restart=always -p 8000:8000 -v /opt/adcm:/adcm/data
   -e DB_HOST="DATABASE_HOSTNAME_OR_IP_ADDRESS" -e DB_PORT="DATABASE_TCP_PORT"
   -e DB_USER="DATABASE_USERNAME" -e DB_NAME="DATABASE_NAME"
   -e DB_PASS="DATABASE_USER_PASSWORD" --name adcm hub.arenadata.io/adcm/adcm:latest
   ```
   Use `-v /opt/adcm:/adcm/data:Z` for SELinux
   Target PostgreSQL DB must not have DB with name `DATABASE_NAME`

## Migrate SQLite -> client PostgreSQL
>__NOTE__: `adcm` is the ADCM's container name.
1. Dump SQLite DB to file:
   ```shell
   docker exec -it adcm /adcm/python/manage.py dumpdata --natural-foreign --natural-primary -o /adcm/data/var/data.json
   ```
2. Stop container:
   ```shell
   docker stop adcm
   docker rm adcm
   ```
3. Start container in `MIGRATION_MODE`:
   ```shell
   docker run -d --restart=always -p 8000:8000 -v /opt/adcm:/adcm/data
   -e DB_HOST="DATABASE_HOSTNAME_OR_IP_ADDRESS" -e DB_PORT="DATABASE_TCP_PORT"
   -e DB_USER="DATABASE_USERNAME" -e DB_NAME="DATABASE_NAME"
   -e DB_PASS="DATABASE_USER_PASSWORD" -e MIGRATION_MODE=1
   --name adcm hub.arenadata.io/adcm/adcm:latest
   ```
   Use `-v /opt/adcm:/adcm/data:Z` for SELinux
   Target PostgreSQL DB must not have DB with name `DATABASE_NAME`
4. Load dumped SQLite DB data to PostgreSQL
   ```shell
   docker exec -it adcm /adcm/python/manage.py loaddata /adcm/data/var/data.json
   ```
5. Stop container:
   ```shell
   docker stop adcm
   docker rm adcm
   ```
6. Start container:
   ```shell
   docker run -d --restart=always -p 8000:8000 -v /opt/adcm:/adcm/data
   -e DB_HOST="DATABASE_HOSTNAME_OR_IP_ADDRESS" -e DB_PORT="DATABASE_TCP_PORT"
   -e DB_USER="DATABASE_USERNAME" -e DB_NAME="DATABASE_NAME"
   -e DB_PASS="DATABASE_USER_PASSWORD" -e MIGRATION_MODE=0
   --name adcm hub.arenadata.io/adcm/adcm:latest
   ```

## Set log level
1. add `-e` option to `docker run` command:
   ```shell
   docker run ... -e LOG_LEVEL="INFO"
   ```

   valid choices are: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`

   defaults to `ERROR`


<!--
Copyright 2021-2022 NVIDIA Corporation

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->

# Legate

The Legate project endeavors to democratize computing by making it possible
for all programmers to leverage the power of large clusters of CPUs and GPUs
by running the same code that runs on a desktop or a laptop at scale.
Using this technology, computational and data scientists can develop and test
programs on moderately sized data sets on local machines and then immediately
scale up to larger data sets deployed on many nodes in the cloud or on a
supercomputer without any code modifications.

The Legate project is built upon two foundational principles:

1. For end users, such as computational and data scientists, the programming
   model must be identical to programming a single sequential CPU on their
   laptop or desktop. All concerns relating to parallelism, data
   distribution, and synchronization must be implicit. The cloud or a
   supercomputer should appear as nothing more than a super-powerful CPU core.
2. Software must be compositional and not just interoperable (i.e. functionally
   correct). Libraries developed in the Legate ecosystem must be able to exchange
   partitioned and distributed data without requiring "shuffles" or unnecessary
   blocking synchronization. Computations from different libraries should be
   able to use arbitrary data and still be reordered across abstraction boundaries
   to hide communication and synchronization latencies where the original sequential
   semantics of the program allow. This is essential for achieving speed-of-light
   performance on large scale machines.

The Legate project is still in its nascent stages of development, but much of
the fundamental architecture is in place. We encourage development and contributions
to existing Legate libraries, as well as the development of new Legate libraries.
Pull requests are welcomed.

If you have questions, please contact us at legate(at)nvidia.com.

- [Legate](#legate)
  - [Why Legate?](#why-legate)
  - [What is the Legate Core?](#what-is-the-legate-core)
  - [How Does Legate Work?](#how-does-legate-work)
  - [How Do I Install Legate?](#how-do-i-install-legate)
  - [How Do I Use Legate?](#how-do-i-use-legate)
    - [Distributed Launch](#distributed-launch)
    - [Debugging and Profiling](#debugging-and-profiling)
  - [Running Legate programs with Jupyter Notebook](#running-legate-programs-with-jupyter-notebook)
    - [Installation of the Legate IPython Kernel](#installation-of-the-legate-ipython-kernel)
    - [Running with Jupyter Notebook](#running-with-jupyter-notebook)
    - [Configuring the Jupyter Notebook](#configuring-the-jupyter-notebook)
    - [Magic Command](#magic-command)
  - [Other FAQs](#other-faqs)
  - [Contributing](#contributing)
  - [Documentation](#documentation)
  - [Next Steps](#next-steps)

## Why Legate?

Computational problems today continue to grow both in their complexity as well
as the scale of the data that they consume and generate. This is true both in
traditional HPC domains as well as enterprise data analytics cases. Consequently,
more and more users truly need the power of large clusters of both CPUs and
GPUs to address their computational problems. Not everyone has the time or
resources required to learn and deploy the advanced programming models and tools
needed to target this class of hardware today. Legate aims to bridge this gap
so that any programmer can run code on any scale machine without needing to be
an expert in parallel programming and distributed systems, thereby allowing
developers to bring the problem-solving power of large machines to bear on
more kinds of challenging problems than ever before.

## What is the Legate Core?

The Legate Core is our version of [Apache Arrow](https://arrow.apache.org/). Apache
Arrow has significantly improved composability of software libraries by making it
possible for different libraries to share in-memory buffers of data without
unnecessary copying. However, it falls short when it comes to meeting two
of our primary requirements for Legate:

1. Arrow only provides an API for describing a physical representation
   of data as a single memory allocation. There is no interface for describing
   cases where data has been partitioned and then capturing the logical
   relationships of those partitioned subsets of data.
2. Arrow is mute on the subject of synchronization. Accelerators such as GPUs
   achieve significantly higher performance when computations are performed
   asynchronously with respect to other components of the system. When data is
   passed between libraries today, accelerators must be pessimistically
   synchronized to ensure that data dependences are satisfied across abstraction
   boundaries. This might result in tolerable overheads for single GPU systems,
   but can result in catastrophically poor performance when hundreds of GPUs are involved.

The Legate Core provides an API very similar to Arrow's interface with several
important distinctions that provide stronger guarantees about data coherence and
synchronization to aid library developers when building Legate libraries. These
guarantees are the crux of how libraries in the Legate ecosystem are able to
provide excellent composability.

The Legate Core API imports several important concepts from Arrow such that
users that are familiar with Arrow already will find it unsurprising. We use
the same type system representation as Arrow so libraries that have already
adopted it do not need to learn or adapt to a new type system. We also reuse
the concept of an [Array](https://arrow.apache.org/docs/cpp/api/array.html)
from Arrow. The `LegateArray` class supports many of the same methods as
the Arrow Array interface (we'll continue to add methods to improve
compatibility). The main difference is that instead of obtaining
[Buffer](https://arrow.apache.org/docs/cpp/api/memory.html#buffers)
objects from arrays to describe allocations of data that back the array, the
Legate Core API introduces a new primitive called a `LegateStore` which
provides a new interface for reasoning about partitioned and distributed
data in asynchronous execution environments.

Any implementation of a `LegateStore` must maintain the following guarantees
to clients of the Legate Core API (i.e. Legate library developers):

1. The coherence of data contained in a `LegateStore` must be implicitly
   managed by the implementation of the Legate Core API. This means that
   no matter where data is requested to perform a computation in a machine,
   the most recent modifications to that data in program order must be
   reflected. It should never be clients responsibility to maintain this
   coherence.
2. It should be possible to create arbitrary views onto `LegateStore` objects
   such that library developers can precisely describe the working sets of
   their computations. Modifications to views must be reflected onto all
   aliasing views data. This property must be maintained by the Legate Core
   API implementation such that it is never the concern of clients.
3. Dependence management between uses of the `LegateStore` objects and their
   views is the responsibility of Legate Core API regardless of what
   (asynchronous) computations are performed on `LegateStore` objects or their
   views. This dependence analysis must be both sound and precise. It is
   illegal to over-approximate dependences. This dependence analysis must also
   be performed globally in scope. Any use of the `LegateStore` on any
   processor/node in the system must abide by the original sequential
   semantics of the program

Note that we do not specify exactly what the abstractions are that are needed
for implementing `LegateStore` objects. Our goal is not prescribe what these
abstractions are as they may be implementation dependent. Our only requirements
are that they have these properties to ensure that incentives are aligned in
such a way for Legate libraries to achieve a high degree of composability
at any scale of machine. Indeed, these requirements shift many of the burdens
that make implementing distributed and accelerated libraries hard off of the
library developers and onto the implementation of the Legate Core API. This
is by design as it allows the costs to be amortized across all libraries in
the ecosystem and ensures that Legate library developers are more productive.

## How Does Legate Work?

Our implementation of the Legate Core API is built on top of the
[Legion](https://legion.stanford.edu/) programming model and runtime system.
Legion was originally designed for large HPC applications that target
supercomputers and consequently applications written in the Legion programming
model tend to both perform and scale well on large clusters of both CPUs and
GPUs. Legion programs are also easy to port to new machines as they inherently
decouple the machine-independent specification of computations from decisions
about how that application is mapped to the target machine. Due to this
abstract nature, many programmers find writing Legion programs challenging.
By implementing the Legate Core API on top of Legion, we've made it easier
to use Legion such that developers can still get access to the benefits of
Legion without needing to learn all of the lowest-level interfaces.

The [Legion programming model](https://legion.stanford.edu/pdfs/sc2012.pdf)
greatly aids in implementing the Legate Core API. Data types from libraries,
such as arrays in cuNumeric are mapped down onto `LegateStore` objects
that wrap Legion data types such as logical regions or futures.
In the case of regions, Legate application libraries rely heavily on
Legion's [support for partitioning of logical regions into arbitrary
subregion views](https://legion.stanford.edu/pdfs/oopsla2013.pdf).
Each library has its own heuristics for computing such partitions that
take into consideration the computations that will access the data, the
ideal sizes of data to be consumed by different processor kinds, and
the available number of processors. Legion automatically manages the coherence
of subregion views regardless of the scale of the machine.

Computations in Legate application libraries are described by Legion tasks.
Tasks describe their data usage in terms of `LegateStore` objects, thereby
allowing Legion to infer where dependences exist. Legion uses distributed
bounding volume hierarchies, similar to a high performance ray-tracer,
to soundly and precisely perform dependence analysis on logical regions
and insert the necessary synchronization between tasks to maintain the
original sequential semantics of a Legate program.

Each Legate application library also comes with its own custom Legion
mapper that uses heuristics to determine the best choice of mapping for
tasks (e.g. are they best run on a CPU or a GPU). All
Legate tasks are currently implemented in native C or CUDA in order to
achieve excellent performance on the target processor kind, but Legion
has bindings in other languages such as Python, Fortran, and Lua for
users that would prefer to use them. Importantly, by using Legion,
Legate is able to control the placement of data in order to leave it
in-place in fast memories like GPU framebuffers across tasks.

When running on large clusters, Legate leverages a novel technology provided
by Legion called "[control replication](https://research.nvidia.com/sites/default/files/pubs/2021-02_Scaling-Implicit-Parallelism//ppopp.pdf)" to avoid the sequential bottleneck
of having one node farm out work to all the nodes in the cluster. With
control replication, Legate will actually replicate the Legate program and
run it across all the nodes of the machine at the same time. These copies
of the program all cooperate logically to appear to execute as one
program. When communication is necessary between
different computations, the Legion runtime's program analysis will automatically
detect it and insert the necessary data movement and synchronization
across nodes (or GPU framebuffers). This is the transformation that allows
sequential programs to run efficiently at scale across large clusters
as though they are running on a single processor.

## How Do I Install Legate?

Legate Core is available from [conda](https://docs.conda.io/projects/conda/en/latest/index.html)
on the [legate channel](https://anaconda.org/legate/legate-core).
Please make sure you have at least conda version 24.1 installed, then create
a new environment containing Legate Core:

```
conda create -n myenv -c nvidia -c conda-forge -c legate legate-core
```

or install it into an existing environment:

```
conda install -c nvidia -c conda-forge -c legate legate-core
```

Only linux-64 packages are available at the moment.

The default package contains GPU support, and is compatible with CUDA >= 11.8
(driver >= 520), and Volta or later GPU architectures. There are also CPU-only
packages available, which will be automatically selected when installing on a
machine without GPUs available. See [BUILD.md](BUILD.md) for instructions on
building Legate Core from source.

## How Do I Use Legate?

After installing the Legate Core library, the next step is to install a Legate
application library such as cuNumeric. The installation process for a
Legate application library will require you to provide a pointer to the location
of your Legate Core library installation as this will be used to configure the
installation of the Legate application library. After you finish installing any
Legate application libraries, you can then simply replace their `import` statements
with the equivalent ones from any Legate application libraries you have installed.
For example, you can change this:
```python
import numpy as np
```
to this:
```python
import cunumeric as np
```
After this, you can use the `legate` driver script in the `bin` directory of
your installation to run any Python program.

You can also use the standard Python interpreter, but in that case configuration
options can only be passed through the environment (see below), and some options
are not available (check the output of legate --help for more details).

For example, to run your script in the default configuration (4 CPUs cores and
4 GB of memory) just run:
```
$ legate my_python_program.py [other args]
```
The `legate` script also allows you to control the amount of resources that
Legate consumes when running on the machine. The `--cpus` and `--gpus` flags
are used to specify how many CPU and GPU processors should be used on a node.
The `--sysmem` flag can be used to specify how many MBs of DRAM Legate is allowed
to use per node, while the `--fbmem` flag controls how many MBs of framebuffer
memory Legate is allowed to use per GPU. For example, when running on a DGX
station, you might run your application as follows:
```
$ legate --cpus 16 --gpus 4 --sysmem 100000 --fbmem 15000 my_python_program.py
```
This will make 16 CPU processors and all 4 GPUs available for use by Legate.
It will also allow Legate to consume up to 100 GB of DRAM memory and 15 GB of
framebuffer memory per GPU for a total of 60 GB of GPU framebuffer memory. Note
that you probably will not be able to make all the resources of the machine
available for Legate as some will be used by the system or Legate itself for
meta-work. Currently if you try to exceed these resources during execution then
Legate will inform you that it had insufficient resources to complete the job
given its current mapping heuristics. If you believe the job should fit within
the assigned resources please let us know so we can improve our mapping heuristics.
There are many other flags available for use in the `legate` driver script that you
can use to communicate how Legate should view the available machine resources.
You can see a list of them by running:
```
$ legate --help
```
In addition to running Legate programs, you can also use Legate in an interactive
mode by simply not passing any `*py` files on the command line. You can still
request resources just as you would though with a normal file. Legate will
still use all the resources available to it, including doing multi-node execution.
```
$ legate --cpus 16 --gpus 4 --sysmem 100000 --fbmem 15000
Welcome to Legion Python interactive console
>>>
```
Note that Legate does not currently support multi-tenancy cases where different
users are attempting to use the same hardware concurrently.

As a convenience, several command-line options can have their default values set
via environment variables. These environment variables, their corresponding command-
line options, and their default values are as follows.

| CLI Option               | Env. Variable                    | Default Value |
|--------------------------|----------------------------------|---------------|
| --omps                   | LEGATE\_OMP\_PROCS               | 0             |
| --ompthreads             | LEGATE\_OMP\_THREADS             | 4             |
| --utility                | LEGATE\_UTILITY\_CORES           | 2             |
| --sysmem                 | LEGATE\_SYSMEM                   | 4000          |
| --numamem                | LEGATE\_NUMAMEM                  | 0             |
| --fbmem                  | LEGATE\_FBMEM                    | 4000          |
| --zcmem                  | LEGATE\_ZCMEM                    | 32            |
| --regmem                 | LEGATE\_REGMEM                   | 0             |
| --eager-alloc-percentage | LEGATE\_EAGER\_ALLOC\_PERCENTAGE | 50            |

### Distributed Launch

If Legate is compiled with networking support (see the
[installation section](#how-do-i-install-legate)),
it can be run in parallel by using the `--nodes` option followed by the number of nodes
to be used.  Whenever the `--nodes` option is used, Legate will be launched
using `mpirun`, even with `--nodes 1`.  Without the `--nodes` option, no launcher will
be used.  Legate currently supports `mpirun`, `srun`, and `jsrun` as launchers and we
are open to adding additional launcher kinds. You can select the
target kind of launcher with `--launcher`.

### Debugging and Profiling

Legate also comes with several tools that you can use to better understand
your program both from a correctness and a performance standpoint. For
correctness, Legate has facilities for constructing both dataflow
and event graphs for the actual run of an application. These graphs require
that you have an installation of [GraphViz](https://www.graphviz.org/)
available on your machine. To generate a dataflow graph for your Legate
program simply pass the `--dataflow` flag to the `legate.py` script and after
your run is complete we will generate a `dataflow_legate.pdf` file containing
the dataflow graph of your program. To generate a corresponding event graph
you simply need to pass the `--event` flag to the `legate.py` script to generate
a `event_graph_legate.pdf` file. These files can grow to be fairly large for non-trivial
programs so we encourage you to keep your programs small when using these
visualizations or invest in a [robust PDF viewer](https://get.adobe.com/reader/).

For profiling all you need to do is pass the `--profile` flag to Legate and
afterwards you will have a `legate_prof` directory containing a web page that
can be viewed in any web browser that displays a timeline of your program's
execution. You simply need to load the `index.html` page from a browser. You
may have to enable local JavaScript execution if you are viewing the page from
your local machine (depending on your browser).

We recommend that you do not mix debugging and profiling in the same run as
some of the logging for the debugging features requires significant file I/O
that can adversely effect the performance of the application.

## Running Legate programs with Jupyter Notebook

Same as normal Python programs, Legate programs can be run
using Jupyter Notebook. Currently we support single node execution with
multiple CPUs and GPUs, and plan to support multi-node execution in the future.
We leverage Legion's Jupyter support, so you may want to refer to the
[relevant section in Legion's README](https://github.com/StanfordLegion/legion/blob/master/jupyter_notebook/README.md).
To simplify the installation, we provide a script specifically for Legate libraries.

### Installation of the Legate IPython Kernel

Please install Legate, then run the following command to install a default
Jupyter kernel:
```
legate-jupyter
```
If installation is successful, you will see some output like the following:
```
Jupyter kernel spec Legate_SM_GPU (Legate_SM_GPU) has been installed
```
`Legate_SM_GPU` is the default kernel name.

### Running with Jupyter Notebook

You will need to start a Jupyter server, then you can use a Jupyter notebook
from any browser. Please refer to the following two sections from the README of
the [Legion Jupyter Notebook extension](https://github.com/StanfordLegion/legion/tree/master/jupyter_notebook)

* Start the Jupyter Notebook server
* Use the Jupyter Notebook in the browser

### Configuring the Jupyter Notebook

The Legate Jupyter kernel is configured according to the command line arguments
provided at install time.  Standard `legate` options for Core, Memory, and
Mult-node configuration may be provided, as well as a name for the kernel:
```
legate-jupyter --name legate_cpus_2 --cpus 2
```
Other configuration options can be seen by using the `--help` command line option.

### Magic Command

We provide a Jupyter magic command to display the IPython kernel configuration.
```
%load_ext legate.jupyter
%legate_info
```
results in output:
```
Kernel 'Legate_SM_GPU' configured for 1 node(s)

Cores:
  CPUs to use per rank : 4
  GPUs to use per rank : 0
  OpenMP groups to use per rank : 0
  Threads per OpenMP group : 4
  Utility processors per rank : 2

Memory:
  DRAM memory per rank (in MBs) : 4000
  DRAM memory per NUMA domain per rank (in MBs) : 0
  Framebuffer memory per GPU (in MBs) : 4000
  Zero-copy memory per rank (in MBs) : 32
  Registered CPU-side pinned memory per rank (in MBs) : 0
```

## Other FAQs

* *Does Legate only work on NVIDIA hardware?*
  No, Legate will run on any processor supported by Legion (e.g. x86, ARM, and
  PowerPC CPUs), and any network supported by GASNet or UCX (e.g. Infiniband,
  Cray, Omnipath, and (ROC-)Ethernet based interconnects).

* *What languages does the Legate Core API have bindings for?*
  Currently the Legate Core bindings are only available in Python. Watch
  this space for new language bindings soon or make a pull request to
  contribute your own. Legion has a C API which should make it easy to
  develop bindings in any language with a foreign function interface.

* *Do I have to build drop-in replacement libraries?*
  No! While we've chosen to provide drop-in replacement libraries for
  popular Python libraries to illustrate the benefits of Legate, you
  are both welcomed and encouraged to develop your own libraries on top
  of Legate. We promise that they will compose well with other existing
  Legate libraries.

* *What other libraries are you planning to release for the Legate ecosystem?*
  We're still working on that. If you have thoughts about what is important
  please let us know so that we can get a feel for where best to put our time.

* *Can I use Legate with other Legion libraries?*
  Yes! If you're willing to extract the Legion primitives from the `LegateStore`
  objects you should be able to pass them into other Legion libraries such as
  [FlexFlow](https://flexflow.ai/).

* *Does Legate interoperate with X?*
  Yes, probably, but we don't recommend it. Our motivation for building
  Legate is to provide the bare minimum subset of functionality that
  we believe is essential for building truly composable software that can still
  run at scale. No other systems out there met our needs. Providing
  interoperability with those other systems will destroy the very essence
  of what Legate is and significantly dilute its benefits. All that being
  said, Legion does provide some means of doing stop-the-world exchanges
  with other runtime system running concurrently in the same processes.
  If you are interested in pursuing this approach please open an issue
  on the [Legion github issue tracker](https://github.com/StanfordLegion/legion/issues)
  as it will be almost entirely orthogonal to how you use Legate.

## Contributing

See the discussion of contributing in [CONTRIBUTING.md](CONTRIBUTING.md).

## Documentation

A complete list of available features can is found in the [Legate Core
documentation](https://nv-legate.github.io/legate.core).

## Next Steps

We recommend starting by experimenting with at least one Legate application
library to test out performance and see how Legate works. If you are interested
in building your own Legate application library, we recommend that you
investigate our [Legate Hello World application library](https://github.com/nv-legate/legate.core/tree/HEAD/examples/hello) that
provides a small example of how to get started developing your own drop-in
replacement library on top of Legion using the Legate Core library.


# Hail

[![Zulip](https://img.shields.io/badge/zulip-join_chat-brightgreen.svg)](https://hail.zulipchat.com?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [![DOI](https://zenodo.org/badge/45069467.svg)](https://zenodo.org/badge/latestdoi/45069467) [![PyPI version](https://badge.fury.io/py/hail.svg)](https://badge.fury.io/py/hail)

[Hail](https://hail.is) is an open-source, general-purpose, Python-based data analysis tool with additional data types and methods for working with genomic data.

Hail is built to scale and has first-class support for multi-dimensional structured data, like the genomic data in a genome-wide association study (GWAS).

Hail is exposed as a Python library, using primitives for distributed queries and linear algebra implemented in Scala, [Spark](https://spark.apache.org/docs/latest/index.html), and increasingly C++.

See the [documentation](https://hail.is/docs/0.2/) for more info on using Hail.

### Community

Hail has been widely adopted in academia and industry, including as the analysis platform for the [genome aggregation database](https://gnomad.broadinstitute.org) and [UK Biobank rapid GWAS](https://www.nealelab.is/uk-biobank). Learn more about [Hail-powered science](https://hail.is/references.html).

### Contribute

If you'd like to discuss or contribute to the development of methods or infrastructure, please:

- see the [For Software Developers](https://hail.is/docs/0.2/getting_started_developing.html) section of the installation guide for info on compiling Hail
- chat with us about development in our [Zulip chatroom](https://hail.zulipchat.com)
- visit the [Development Forum](https://dev.hail.is) for longer-form discussions

Hail uses a continuous deployment approach to software development, which means we frequently add new features. We update users about changes to Hail via the [Discussion Forum](https://discuss.hail.is). We recommend creating an account on the Discussion Forum so that you can subscribe to these updates as well.

### Maintainer

Hail is maintained by a team in the [Neale lab](https://nealelab.is/) at the [Stanley Center for Psychiatric Research](https://www.broadinstitute.org/stanley) of the [Broad Institute of MIT and Harvard](https://www.broadinstitute.org) and the [Analytic and Translational Genetics Unit](https://www.atgu.mgh.harvard.edu/) of [Massachusetts General Hospital](https://www.massgeneral.org/).

Contact the Hail team at hail@broadinstitute.org.

### Citing Hail

If you use Hail for published work, please cite the software. You can get a
citation for the version of Hail you installed by executing:

```python
import hail as hl
print(hl.citation())
```

Which will look like:

```
Hail Team. Hail 0.2.13-81ab564db2b4. https://github.com/hail-is/hail/releases/tag/0.2.13.
```

##### Acknowledgements

The Hail team has several sources of funding at the Broad Institute:
- The Stanley Center for Psychiatric Research, which together with Neale Lab has provided an incredibly supportive and stimulating home.
- Principal Investigators Benjamin Neale and Daniel MacArthur, whose scientific leadership has been essential for solving the right problems.
- Jeremy Wertheimer, whose strategic advice and generous philanthropy have been essential for growing the impact of Hail.

We are grateful for generous support from:
- The National Institute of Diabetes and Digestive and Kidney Diseases
- The National Institute of Mental Health
- The National Human Genome Research Institute
- The Chan Zuckerberg Initiative

We would like to thank Zulip for supporting
open-source by providing free hosting, and YourKit, LLC for generously providing
free licenses for YourKit Java
Profiler for open-source development.


# tpm2-pytss
[![Tests](https://github.com/tpm2-software/tpm2-pytss/actions/workflows/tests.yaml/badge.svg)](https://github.com/tpm2-software/tpm2-pytss/actions/workflows/tests.yaml)
[![codecov](https://codecov.io/gh/tpm2-software/tpm2-pytss/branch/master/graph/badge.svg?token=Nqs8anZr2B)](https://codecov.io/gh/tpm2-software/tpm2-pytss)
[![Documentation Status](https://readthedocs.org/projects/tpm2-pytss/badge/?version=latest)](https://tpm2-pytss.readthedocs.io/en/latest/?badge=latest)
[![CodeQL](https://github.com/tpm2-software/tpm2-pytss/actions/workflows/codeql.yml/badge.svg?branch=master&event=push)](https://github.com/tpm2-software/tpm2-pytss/actions/workflows/codeql.yml)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black)
[![PyPI version](https://img.shields.io/pypi/v/tpm2-pytss.svg)](https://pypi.org/project/tpm2-pytss)

TPM2 TSS Python bindings for Enhanced System API (ESYS), Feature API (FAPI), Marshaling (MU), TCTI
Loader (TCTILdr), TCTIs, policy, and RC Decoding (rcdecode) libraries. It allows for custom TCTIs
written in Python as well. It also contains utility methods for wrapping keys to TPM 2.0 data
structures for importation into the TPM, unwrapping keys and exporting them from the TPM, TPM-less
makecredential command and name calculations, TSS2 PEM Key format support, importing Keys from PEM,
DER and SSH formats, conversion from tpm2-tools based command line strings and loading tpm2-tools
context files.

## Documentation

Documentation for the latest release is hosted at
https://tpm2-pytss.readthedocs.io/en/latest/index.html

## Installing

To install the master branch:
```bash
python3 -m pip install git+https://github.com/tpm2-software/tpm2-pytss.git
```

To install latest stable from PyPi:
```bash
python3 -m pip install tpm2-pytss
```
**NOTE**: You may need option `--user` or sitewide permissions through something like `sudo`.

This is known to work with versions 2.4.0 of tpm2-tss or higher.

## Help

- Ask a question via an [issue](https://github.com/tpm2-software/tpm2-pytss/issues/new)
- Send an email to the tpm2 list:
    - https://lists.linuxfoundation.org/mailman/listinfo/tpm2
- File a Security Bug by following the instructions in [docs/SECURITY.md](docs/SECURITY.md)

## License

tpm2-pytss is distributed under the [BSD 2 Clause License](LICENSE).


=========================================================
OCTIS : Optimizing and Comparing Topic Models is Simple!
=========================================================

.. |colab1| image:: https://colab.research.google.com/assets/colab-badge.svg
    :target: https://colab.research.google.com/github/MIND-Lab/OCTIS/blob/master/examples/OCTIS_LDA_training_only.ipynb
    :alt: Open In Colab

.. |colab2| image:: https://colab.research.google.com/assets/colab-badge.svg
    :target: https://colab.research.google.com/github/MIND-Lab/OCTIS/blob/master/examples/OCTIS_Optimizing_CTM.ipynb
    :alt: Open In Colab
.. |twitter_silvia| image:: https://img.shields.io/twitter/follow/TerragniSilvia?style=social
    :target: https://twitter.com/intent/follow?screen_name=TerragniSilvia
    :alt: Follow TerragniSilvia on Twitter
.. |twitter_betta| image:: https://img.shields.io/twitter/follow/FersiniE?style=social
    :target: https://twitter.com/intent/follow?screen_name=FersiniE
    :alt: Follow FersiniE on Twitter

.. image:: https://img.shields.io/pypi/v/octis.svg
        :target: https://pypi.python.org/pypi/octis

.. image:: https://github.com/MIND-Lab/OCTIS/workflows/Python%20package/badge.svg
        :target: https://github.com/MIND-Lab/OCTIS/actions

.. image:: https://readthedocs.org/projects/octis/badge/?version=latest
        :target: https://octis.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: https://img.shields.io/github/contributors/MIND-Lab/OCTIS
        :target: https://github.com/MIND-Lab/OCTIS/graphs/contributors/
        :alt: Contributors

.. image:: https://img.shields.io/badge/License-MIT-blue.svg
        :target: https://lbesson.mit-license.org/
        :alt: License

.. image:: https://img.shields.io/github/stars/mind-lab/OCTIS?logo=github
        :target: https://github.com/mind-lab/OCTIS/stargazers
        :alt: Github Stars

.. image:: https://pepy.tech/badge/octis/month
        :target: https://pepy.tech/project/octis
        :alt: Monthly Downloads

.. image:: https://colab.research.google.com/assets/colab-badge.svg
    :target: https://colab.research.google.com/github/MIND-Lab/OCTIS/blob/master/examples/OCTIS_Optimizing_CTM.ipynb
        :alt: Open In Colab

.. image:: https://github.com/MIND-Lab/OCTIS/blob/master/logo.png?raw=true
  :width: 100
  :alt: Logo

**OCTIS (Optimizing and Comparing Topic models Is Simple)** aims at training, analyzing and comparing
Topic Models, whose optimal hyperparameters are estimated by means of a Bayesian Optimization approach. This work has been accepted to the demo track of EACL2021. `Click to read the paper`_!

.. contents:: Table of Contents
   :depth: 2

***************
Install
***************

You can install OCTIS with the following command:
::

    pip install octis

You can find the requirements in the `requirements.txt` file.

***************
Main Features
***************

* Preprocess your own dataset or use one of the already-preprocessed benchmark datasets
* Well-known topic models (both classical and neurals)
* Evaluate your model using different state-of-the-art evaluation metrics
* Optimize the models' hyperparameters for a given metric using Bayesian Optimization
* Python library for advanced usage or simple web dashboard for starting and controlling the optimization experiments

***********************
Examples and Tutorials
***********************

To easily understand how to use OCTIS, we invite you to try our tutorials out :)

+--------------------------------------------------------------------------------+------------------+
| Name                                                                           | Link             |
+================================================================================+==================+
| How to build a topic model and evaluate the results (LDA on 20Newsgroups)      | |colab1|         |
+--------------------------------------------------------------------------------+------------------+
| How to optimize the hyperparameters of a neural topic model (CTM on M10)       | |colab2|         |
+--------------------------------------------------------------------------------+------------------+

Some tutorials on Medium:
===========================

Two guides on how to use OCTIS with practical examples:

* `A beginner's guide to OCTIS vol. 1`_ by `Emil Rijcken`_
* `A beginner's guide to OCTIS vol. 2`_ by `Emil Rijcken`_

A tutorial on topic modeling on song lyrics:

* `OCTIS - The Future of Topic Modeling`_ by `Nicolas Pogeant`_

.. _Emil Rijcken: https://emilrijcken.medium.com/
.. _A beginner's guide to OCTIS vol. 1: https://towardsdatascience.com/a-beginners-guide-to-octis-optimizing-and-comparing-topic-models-is-simple-590554ec9ba6
.. _A beginner's guide to OCTIS vol. 2: https://towardsdatascience.com/a-beginners-guide-to-octis-vol-2-optimizing-topic-models-1214e58be1e5
.. _OCTIS - The Future of Topic Modeling: https://medium.com/mlearning-ai/octis-the-future-of-topic-modeling-45ef8cd66089
.. _Nicolas Pogeant: https://medium.com/@npogeant

**************************
Datasets and Preprocessing
**************************

Load a preprocessed dataset
============================

To load one of the already preprocessed datasets as follows:

.. code-block:: python

   from octis.dataset.dataset import Dataset
   dataset = Dataset()
   dataset.fetch_dataset("20NewsGroup")

Just use one of the dataset names listed below. Note: it is case-sensitive!

Available Datasets
============================

+--------------+--------------+--------+---------+----------+----------+
|Name in OCTIS | Source       | # Docs | # Words | # Labels | Language |
+==============+==============+========+=========+==========+==========+
| 20NewsGroup  | 20Newsgroup_ |  16309 |    1612 |       20 | English  |
+--------------+--------------+--------+---------+----------+----------+
| BBC_News     | BBC-News_    |   2225 |    2949 |        5 | English  |
+--------------+--------------+--------+---------+----------+----------+
| DBLP         | DBLP_        |  54595 |    1513 |        4 | English  |
+--------------+--------------+--------+---------+----------+----------+
| M10          | M10_         |   8355 |    1696 |       10 | English  |
+--------------+--------------+--------+---------+----------+----------+
| DBPedia_IT   | DBPedia_IT_  |   4251 |    2047 |        5 | Italian  |
+--------------+--------------+--------+---------+----------+----------+
| Europarl_IT  | Europarl_IT_ |   3613 |    2000 |       NA | Italian  |
+--------------+--------------+--------+---------+----------+----------+

.. _20Newsgroup: https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html
.. _BBC-News: https://github.com/MIND-Lab/OCTIS
.. _DBLP: https://dblp.org/rec/conf/ijcai/PanWZZW16.html?view=bibtex
.. _M10: https://dblp.org/rec/conf/ijcai/PanWZZW16.html?view=bibtex
.. _DBPedia_IT: https://www.dbpedia.org/resources/ontology/
.. _Europarl_IT: https://www.statmt.org/europarl/

Load a Custom Dataset
============================
Otherwise, you can load a custom preprocessed dataset in the following way:

.. code-block:: python

   from octis.dataset.dataset import Dataset
   dataset = Dataset()
   dataset.load_custom_dataset_from_folder("../path/to/the/dataset/folder")

Make sure that the dataset is in the following format:
    * corpus file: a .tsv file (tab-separated) that contains up to three columns, i.e. the document, the partitition, and the label associated to the document (optional).
    * vocabulary: a .txt file where each line represents a word of the vocabulary

The partition can be "train" for the training partition, "test" for testing partition, or "val" for the validation partition. An example of dataset can be found here: `sample_dataset`_.

Disclaimer
~~~~~~~~~~~~~

Similarly to `TensorFlow Datasets`_ and HuggingFace's `nlp`_ library, we just downloaded and prepared public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license and to cite the right owner of the dataset.

If you're a dataset owner and wish to update any part of it, or do not want your dataset to be included in this library, please get in touch through a GitHub issue.

If you're a dataset owner and wish to include your dataset in this library, please get in touch through a GitHub issue.

Preprocess a Dataset
============================

To preprocess a dataset, import the preprocessing class and use the preprocess_dataset method.

.. code-block:: python

    import os
    import string
    from octis.preprocessing.preprocessing import Preprocessing
    os.chdir(os.path.pardir)

    # Initialize preprocessing
    preprocessor = Preprocessing(vocabulary=None, max_features=None,
                                 remove_punctuation=True, punctuation=string.punctuation,
                                 lemmatize=True, stopword_list='english',
                                 min_chars=1, min_words_docs=0)
    # preprocess
    dataset = preprocessor.preprocess_dataset(documents_path=r'..\corpus.txt', labels_path=r'..\labels.txt')

    # save the preprocessed dataset
    dataset.save('hello_dataset')

For more details on the preprocessing see the preprocessing demo example in the examples folder.

*****************************
Topic Models and Evaluation
*****************************

Train a model
==============

To build a model, load a preprocessed dataset, set the model hyperparameters and use :code:`train_model()` to train the model.

.. code-block:: python

    from octis.dataset.dataset import Dataset
    from octis.models.LDA import LDA

    # Load a dataset
    dataset = Dataset()
    dataset.load_custom_dataset_from_folder("dataset_folder")

    model = LDA(num_topics=25)  # Create model
    model_output = model.train_model(dataset) # Train the model

If the dataset is partitioned, you can:

* Train the model on the training set and test it on the test documents
* Train the model with the whole dataset, regardless of any partition.

Available Models
=================

+-------------------------------------------+-----------------------------------------------------------+
| Name                                      | Implementation                                            |
+===========================================+===========================================================+
| CTM `(Bianchi et al. 2021)`_              | https://github.com/MilaNLProc/contextualized-topic-models |
+-------------------------------------------+-----------------------------------------------------------+
| ETM `(Dieng et al. 2020)`_                | https://github.com/adjidieng/ETM                          |
+-------------------------------------------+-----------------------------------------------------------+
| HDP `(Blei et al. 2004)`_                 | https://radimrehurek.com/gensim/                          |
+-------------------------------------------+-----------------------------------------------------------+
| LDA `(Blei et al. 2003)`_                 | https://radimrehurek.com/gensim/                          |
+-------------------------------------------+-----------------------------------------------------------+
| LSI `(Landauer et al. 1998)`_             | https://radimrehurek.com/gensim/                          |
+-------------------------------------------+-----------------------------------------------------------+
| NMF `(Lee and Seung 2000)`_               | https://radimrehurek.com/gensim/                          |
+-------------------------------------------+-----------------------------------------------------------+
| NeuralLDA `(Srivastava and Sutton 2017)`_ | https://github.com/estebandito22/PyTorchAVITM             |
+-------------------------------------------+-----------------------------------------------------------+
| ProdLda `(Srivastava and Sutton 2017)`_   | https://github.com/estebandito22/PyTorchAVITM             |
+-------------------------------------------+-----------------------------------------------------------+

.. _(Bianchi et al. 2021): https://www.aclweb.org/anthology/2021.eacl-main.143/
.. _(Dieng et al. 2020): https://www.aclweb.org/anthology/2020.tacl-1.29
.. _(Blei et al. 2004): https://people.eecs.berkeley.edu/~jordan/papers/hdp.pdf
.. _(Blei et al. 2003): https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
.. _(Landauer et al. 1998): http://lsa.colorado.edu/papers/dp1.LSAintro.pdf
.. _(Lee and Seung 2000): https://papers.nips.cc/paper/1861-algorithms-for-non-negative-matrix-factorization
.. _(Srivastava and Sutton 2017): https://arxiv.org/abs/1703.01488

If you use one of these implementations, make sure to cite the right paper.

If you implemented a model and wish to update any part of it, or do not want your model to be included in this library, please get in touch through a GitHub issue.

If you implemented a model and wish to include your model in this library, please get in touch through a GitHub issue. Otherwise, if you want to include the model by yourself, see the following section.

Evaluate a model
==================

To evaluate a model, choose a metric and use the :code:`score()` method of the metric class.

.. code-block:: python

    from octis.evaluation_metrics.diversity_metrics import TopicDiversity

    metric = TopicDiversity(topk=10) # Initialize metric
    topic_diversity_score = metric.score(model_output) # Compute score of the metric

Available metrics
==================

* **Classification Metrics**:

    * F1-score_ : :code:`F1Score(dataset)`
    * Precision_ : :code:`PrecisionScore(dataset)`
    * Recall_ : :code:`RecallScore(dataset)`
    * Accuracy_ : :code:`AccuracyScore(dataset)`

.. _F1-score: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/classification_metrics.py#L117
.. _Precision: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/classification_metrics.py#L145
.. _Recall: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/classification_metrics.py#L171
.. _Accuracy: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/classification_metrics.py#L196

* **Coherence Metrics**:

    * `UMass Coherence`_ : :code:`Coherence(measure='u_mass')`
    * `C_V Coherence`_ : :code:`Coherence(measure='c_v')`
    * `UCI Coherence`_ : :code:`Coherence(measure='c_uci')`
    * `NPMI Coherence`_ : :code:`Coherence(measure='c_npmi')`
    * `Word Embedding-based Coherence Pairwise`_ : :code:`WECoherencePairwise()`
    * `Word Embedding-based Coherence Centroid`_ : :code:`WECoherenceCentroid()`

.. _`UMass Coherence`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L15
.. _`C_V Coherence`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L15
.. _`UCI Coherence`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L15
.. _`NPMI Coherence`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L15
.. _`Word Embedding-based Coherence Pairwise`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L67
.. _`Word Embedding-based Coherence Centroid`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/coherence_metrics.py#L126

* **Diversity Metrics**:

    * `Topic Diversity`_ : :code:`TopicDiversity()`
    * `InvertedRBO`_ : :code:`InvertedRBO()`
    * `Word Embedding-based InvertedRBO Matches`_ : :code:`WordEmbeddingsInvertedRBO()`
    * `Word Embedding-based InvertedRBO Centroid`_ : :code:`WordEmbeddingsInvertedRBOCentroid()`
    * `Log odds ratio`_ : :code:`LogOddsRatio()`
    * `Kullback-Liebler Divergence`_ : :code:`KLDivergence()`

.. _`Topic Diversity`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L12
.. _`InvertedRBO`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L56
.. _`Word Embedding-based InvertedRBO Matches`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L92
.. _`Word Embedding-based InvertedRBO Centroid`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L147
.. _`Log odds ratio`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L184
.. _`Kullback-Liebler Divergence`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/diversity_metrics.py#L209

* **Similarity Metrics**:

    * `Ranked-Biased Overlap`_ : :code:`RBO()`
    * `Word Embedding-based RBO Matches`_ : :code:`WordEmbeddingsRBOMatch()`
    * `Word Embedding-based RBO Centroid`_ : :code:`WordEmbeddingsRBOCentroid()`
    * `Word Embeddings-based Pairwise Similarity`_ : :code:`WordEmbeddingsPairwiseSimilarity()`
    * `Word Embeddings-based Centroid Similarity`_ : :code:`WordEmbeddingsCentroidSimilarity()`
    * `Word Embeddings-based Weighted Sum Similarity`_ : :code:`WordEmbeddingsWeightedSumSimilarity()`
    * `Pairwise Jaccard Similarity`_ : :code:`PairwiseJaccardSimilarity()`

.. _`Word Embedding-based RBO Matches`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L11
.. _`Word Embedding-based RBO Centroid`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L35
.. _`Word Embeddings-based Pairwise Similarity`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L59
.. _`Word Embeddings-based Centroid Similarity`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L103
.. _`Ranked-Biased Overlap`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L201
.. _`Word Embeddings-based Weighted Sum Similarity`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L158
.. _`Pairwise Jaccard Similarity`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/similarity_metrics.py#L223

* **Topic significance Metrics**:

    * `KL Uniform`_ : :code:`KL_uniform()`
    * `KL Vacuous`_ : :code:`KL_vacuous()`
    * `KL Background`_ : :code:`KL_background()`

.. _`KL Uniform`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/topic_significance_metrics.py#L37
.. _`KL Vacuous`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/topic_significance_metrics.py#L84
.. _`KL Background`: https://github.com/MIND-Lab/OCTIS/blob/master/octis/evaluation_metrics/topic_significance_metrics.py#L138

Implement your own Model
=========================

Models inherit from the class `AbstractModel` defined in `octis/models/model.py` .
To build your own model your class must override the `train_model(self, dataset, hyperparameters)` method which always requires at least a `Dataset` object and a `Dictionary` of hyperparameters as input and should return a dictionary with the output of the model as output.

To better understand how a model work, let's have a look at the LDA implementation.
The first step in developing a custom model is to define the dictionary of default hyperparameters values:

.. code-block:: python

    hyperparameters = {'corpus': None, 'num_topics': 100, 'id2word': None, 'alpha': 'symmetric',
        'eta': None, # ...
        'callbacks': None}

Defining the default hyperparameters values allows users to work on a subset of them without having to assign a value to each parameter.

The following step is the `train_model()` override:

.. code-block:: python

    def train_model(self, dataset, hyperparameters={}, top_words=10):

The LDA method requires a dataset, the hyperparameters dictionary and an extra (optional) argument used to select how many of the most significative words track for each topic.

With the hyperparameters defaults, the ones in input and the dataset you should be able to write your own code and return as output a dictionary with at least 3 entries:

* *topics*: the list of the most significative words foreach topic (list of lists of strings).
* *topic-word-matrix*: an NxV matrix of weights where N is the number of topics and V is the vocabulary length.
* *topic-document-matrix*: an NxD matrix of weights where N is the number of topics and D is the number of documents in the corpus.

if your model supports the training/test partitioning it should also return:

* *test-topic-document-matrix*: the document topic matrix of the test set.

*****************************
Hyperparameter Optimization
*****************************

To optimize a model you need to select a dataset, a metric and the search space of the hyperparameters to optimize.
For the types of the hyperparameters, we use :code:`scikit-optimize` types (https://scikit-optimize.github.io/stable/modules/space.html)

.. code-block:: python

    from octis.optimization.optimizer import Optimizer
    from skopt.space.space import Real

    # Define the search space. To see which hyperparameters to optimize, see the topic model's initialization signature
    search_space = {"alpha": Real(low=0.001, high=5.0), "eta": Real(low=0.001, high=5.0)}

    # Initialize an optimizer object and start the optimization.
    optimizer=Optimizer()
    optResult=optimizer.optimize(model, dataset, eval_metric, search_space, save_path="../results" # path to store the results
                                 number_of_call=30, # number of optimization iterations
                                 model_runs=5) # number of runs of the topic model
    #save the results of th optimization in a csv file
    optResult.save_to_csv("results.csv")

The result will provide best-seen value of the metric with the corresponding hyperparameter configuration, and the hyperparameters and metric value for each iteration of the optimization. To visualize this information, you have to set 'plot' attribute of Bayesian_optimization to True.

You can find more here: `optimizer README`_

*****************************
Dashboard
*****************************

OCTIS includes a user friendly graphical interface for creating, monitoring and viewing experiments.
Following the implementation standards of datasets, models and metrics the dashboard will automatically update and allow you to use your own custom implementations.

To run the dashboard you need to clone the repo.
While in the project directory run the following command:

.. code-block:: bash

    python OCTIS/dashboard/server.py

The browser will open and you will be redirected to the dashboard.
In the dashboard you can:

* Create new experiments organized in batch
* Visualize and compare all the experiments
* Visualize a custom experiment
* Manage the experiment queue

*****************************
How to cite our work
*****************************
This work has been accepted at the demo track of EACL 2021! `Click to read the paper`_!
If you decide to use this resource, please cite:

::

    @inproceedings{terragni2020octis,
        title={{OCTIS}: Comparing and Optimizing Topic Models is Simple!},
        author={Terragni, Silvia and Fersini, Elisabetta and Galuzzi, Bruno Giovanni and Tropeano, Pietro and Candelieri, Antonio},
        year={2021},
        booktitle={Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations},
        month = apr,
        year = "2021",
        publisher = "Association for Computational Linguistics",
        url = "https://www.aclweb.org/anthology/2021.eacl-demos.31",
        pages = "263--270",
    }

    @inproceedings{DBLP:conf/clic-it/TerragniF21,
      author    = {Silvia Terragni and Elisabetta Fersini},
      editor    = {Elisabetta Fersini and Marco Passarotti and Viviana Patti},
      title     = {{OCTIS 2.0: Optimizing and Comparing Topic Models in Italian Is Even
                   Simpler!}},
      booktitle = {Proceedings of the Eighth Italian Conference on Computational Linguistics,
                   CLiC-it 2021, Milan, Italy, January 26-28, 2022},
      series    = {{CEUR} Workshop Proceedings},
      volume    = {3033},
      publisher = {CEUR-WS.org},
      year      = {2021},
      url       = {http://ceur-ws.org/Vol-3033/paper55.pdf},
    }

*****************************
Team
*****************************

Project and Development Lead
=============================

- `Silvia Terragni`_ <s.terragni4@campus.unimib.it> |twitter_silvia|
- Elisabetta Fersini <elisabetta.fersini@unimib.it> |twitter_betta|
- Antonio Candelieri <antonio.candelieri@unimib.it>

Current Contributors
=============================

- Pietro Tropeano <p.tropeano1@campus.unimib.it> Framework architecture, Preprocessing, Topic Models, Evaluation metrics and Web Dashboard
- Bruno Galuzzi <bruno.galuzzi@unimib.it> Bayesian Optimization
- Silvia Terragni <s.terragni4@campus.unimib.it> Overall project

Past Contributors
=============================

* Lorenzo Famiglini <l.famiglini@campus.unimib.it> Neural models integration
* Davide Pietrasanta <d.pietrasanta@campus.unimib.it> Bayesian Optimization

*****************************
Credits
*****************************

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template. Thanks to all the developers that released their topic models' implementations. A special thanks goes to tenggaard_ who helped us find many bugs in early octis releases and to `Emil Rijcken`_ who kindly wrote two guides on how to use OCTIS :)

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`Click to read the paper`: https://www.aclweb.org/anthology/2021.eacl-demos.31/
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
.. _Silvia Terragni: https://silviatti.github.io/
.. _sample_dataset: https://github.com/MIND-Lab/OCTIS/tree/master/preprocessed_datasets/sample_dataset
.. _Optimizer README: https://github.com/MIND-Lab/topic-modeling-evaluation-framework/blob/develop-package/octis/optimization/README.md
.. _TensorFlow Datasets: https://github.com/tensorflow/datasets
.. _nlp: https://github.com/huggingface/nlp
.. _tenggaard: https://github.com/tenggaard


# ElastAlert 2

ElastAlert 2 is a standalone software tool for alerting on anomalies, spikes, or other patterns of interest from data in [Elasticsearch][10] and [OpenSearch][9].

ElastAlert 2 is backwards compatible with the original [ElastAlert][0] rules.

![CI Workflow](https://github.com/jertel/elastalert/workflows/master_build_test/badge.svg)

## Docker and Kubernetes

ElastAlert 2 is well-suited to being run as a microservice, and is available
as an image on [Docker Hub][2] and on [GitHub Container Registry][11]. For more instructions on how to
configure and run ElastAlert 2 using Docker, see [here][8].

A [Helm chart][7] is also included for easy configuration as a Kubernetes deployment.

## Documentation

Documentation, including an FAQ, for ElastAlert 2 can be found on [readthedocs.com][3]. This is the place to start if you're not familiar with ElastAlert 2 at all.

Elasticsearch 8 support is documented in the [FAQ][12].

The full list of platforms that ElastAlert 2 can fire alerts into can be found [in the documentation][4].

## Contributing

Please see our [contributing guidelines][6].

## License

ElastAlert 2 is licensed under the [Apache License, Version 2.0][5].

[0]: https://github.com/yelp/elastalert
[1]: https://github.com/jertel/elastalert2/blob/master/examples/config.yaml.example
[2]: https://hub.docker.com/r/jertel/elastalert2
[3]: https://elastalert2.readthedocs.io/
[4]: https://elastalert2.readthedocs.io/en/latest/alerts.html#alert-types
[5]: https://www.apache.org/licenses/LICENSE-2.0
[6]: https://github.com/jertel/elastalert2/blob/master/CONTRIBUTING.md
[7]: https://github.com/jertel/elastalert2/tree/master/chart/elastalert2
[8]: https://elastalert2.readthedocs.io/en/latest/running_elastalert.html
[9]: https://opensearch.org/
[10]: https://github.com/elastic/elasticsearch
[11]: https://github.com/jertel/elastalert2/pkgs/container/elastalert2%2Felastalert2
[12]: https://elastalert2.readthedocs.io/en/latest/recipes/faq.html#does-elastalert-2-support-elasticsearch-8


## README for PHANGS-ALMA Pipeline Version 2.0

### PREFACE

**Contents:** This is "version 2" of the [PHANGS](https://sites.google.com/view/phangs/home) post-processing and science-ready data product pipeline. These programs use CASA, as well as the Python package astropy and other affiliated packages (analysisutils, spectral-cube, reproject), to process data from calibrated visibilities to science-ready spectral cubes and maps. The procedures and background for key parts of the pipeline are discussed in the Astrophysical Journal Supplements Paper "PHANGS-ALMA Data Processing and Pipeline" by Leroy, Hughes, Liu, Pety, Rosolowsky, Saito, Schinnerer, Usero, Faesi, Herrera et al. [LINK](https://ui.adsabs.harvard.edu). Please consult that paper for more background and details.

**What this pipeline is for:** This pipeline is devised to process data from radio interferometer observations (from, e.g., ALMA or VLA). It is applied to calibrated visibilities, as those generated by the CASA software, and delivers science-ready spectral cubes and moment maps, along with associated uncertainty maps. In this regard, the PHANGS-ALMA pipeline offers a flexible alternative to the `scriptForImaging` script distributed by ALMA. A detailed list of the derived data products can be found in Section 7 of the paper mentioned above.

**Pipeline and Configuration Files:** These are the programs to run the PHANGS-ALMA pipeline. Configuration files for a large set of PHANGS projects, including the live version of the files for the PHANGS-ALMA CO survey, exist in a separate repository. We include a frozen set of files that can be used to reduce PHANGS-ALMA as examples here. If you need access to those other repositories or need examples, please request access as needed.

**Contact:** For issues, the preferred method is to open an issue on the github issues page. If you have specific other topics to discuss you should reach out to Adam Leroy, Erik Rosolowsky, or Daizhong Liu via email. But opening issues is better.

**Earlier Versions:** If you are looking for Version 1.0 of the pipeline, you can access it by changing branches to "version1.0". Note that this will mostly be for historical reasons. We suggest using Version 2.0 moving forward.

**Total Power:** The pipeline can feather interferometer data with complementary Total Power observations. However, as of Version 2.0 it cannot process Total Power data on its own and assumes that a Total Power cube has been produced by other means. For the PHANGS-ALMA data release, these Total Power cubes were produced with the scripts and pipeline described by Herrera et al. here: https://github.com/PhangsTeam/TP_ALMA_data_reduction/ .

### REQUIREMENTS

The pipeline runs in two separate software environments:

* [CASA](https://casa.nrao.edu/) 5.6 or 5.7 (Staging, Imaging and Post-Processing)
    * Not yet tested for CASA 6.x
* Python 3.6 or later (Derived products) with modern versions of several packages
    * [numpy](https://numpy.org)
    * [scipy](https://www.scipy.org)
    * [astropy](https://www.astropy.org)
    * [spectral-cube](https://spectral-cube.readthedocs.io/en/latest/)

We recommend a standard [anaconda](https://www.anaconda.com/) distribution for python.

### TWO WAYS TO USE THE PIPELINE

There are two ways that this pipeline might be useful. First, it provides an end-to-end path to process calibrated ALMA data (or VLA data) of the sort produced by the scriptForPI script distributed by ALMA into spectral cubes and maps. That end-to-end approach is described in "Workflow for most users." Second, the `phangsPipeline` directory contains a number of modules for use inside and outside CASA that should have general utility. These are written without requiring any broader awareness of the pipeline infrastructure and should just be generally useful. These are files named `casaSOMENAME.py` and `scSOMEOTHERNAME.py` and, to a lesser extent, `utilsYETANOTHERNAME.py`.

### WORKFLOW FOR MOST USERS

If you just want to *use* the pipeline then you will need to do three things:

( 0. Run `scriptForPI.py` to apply the observatory-provided calibration to your data. The pipeline picks up from there, it does not replace the ALMA observatory calibration and flagging pipeline. )

1. Make configuration files ("key files") that describe your project. Usually you can copy and modify an existing project to get a good start. We provide PHANGS-ALMA as an example.

2. Put together two small scripts: one to run the CASA stuff and another to run the pure python stuff. In theory these could be combined or generalized, but we usually just write a few small programs.

3. Run these scripts in order. The CASA stuff runs inside a CASA shell - the pipeline seems to work up through CASA 5.7 and has been heavily used in 5.4 and 5.6, In theory it should be workable in CASA 6.1+ but this isn't for sure yet. The pure python stuff expects a distribution with numpy, astropy, spectral-cube, and scipy and python 3.6+ or so.

**The Easiest Way** This release includes the full PHANGS-ALMA set of keys and the scripts we use to run the pipeline for PHANGS-ALMA. These are *heavily documented* - copy them to make your own script and configuration and follow the documented in those scripts to get started. To be specific:

The PHANGS-ALMA keys to reduce the data end-to-end from the archive, along with heavy documentation are in: `phangs-alma_keys/`

The script to run the CASA part of the pipeline is: `run_casa_pipeline_phangs-alma.py`

The python (v3.x) script to create derived products is: `run_derived_pipeline_phangs-alma.py`

These can run the actual PHANGS-ALMA reduction, though in practice we used slightly more complex versions of a few programs to manage the workflow. Copying and modifying these is your best bet, especially following the patterns in the key files.

### A FEW DETAILS ON PROCEDURE

The full procedure is described in our ApJ Supplements paper and the programs themselves are all in this repository, so we do not provide any extremely detailed docs here. Many individual routines are documented, though we also intend to improve the documentation in the future. Therefore we just note that broadly, the pipeline runs in four stages:

1. **Staging (in CASA)** Stage and process uv-data. This step includes continuum subtraction, line extraction, and spectral regridding.

2. **Imaging (in CASA)** Image and deconvolve the uv-data. This runs in several steps: dirty imaging, clean mask alignment, multi-scale deconvolution, re-masking, and single convolution.

3. **Post-Process (in CASA)** Process deconvolved data into science-ready data cubes. This stage includes merging with the total power and mosaicking.

4. **Derived Products (in python)** Convolution, noise estimation, masking, and calculation of science-ready data products.

The simplest way to run these is to write two small scripts and do the following:

1. Initialize CASA
2. Run a script that initializes a `keyHandler` object pointed at your key directory (see below). Then use this keyHandler to initialize handler objects for uv data, imaging, and postprocessing. Optionally restrict those objects of interest for each handler to a subset of targets, array configurations, or lines.
3. Inside the same script, run the main loop commands for each handler object.

Then exit CASA and

4. Initialize a python environment with scipy, numpy, astropy, and spectral-cube installed.
5. Run a script that initializes a `keyHandler` again pointed at your key directory, then use this keyHandler to initialize a derived product handler.
6. Run the main loop for the derived product handler.

These two scripts are the ones listed above. They are heavily annotated and should provide a good starting point.

### SINGLE DISH PROCESSING (still in development)

The pipeline also includes scripts to execute the single dish processing described by Herrera et al. (2020). These capabilities are still somewhat in development.

To use this capability:

1. The measurement set key (e.g., ms_file_key) should include entries
   labeled "tp" or similar. These "tp" entries in the measurements set
   should refer to the root directory for the delivered data set
   (i.e., up one level from calibrated/ scripts/ etc.).

2. The configuration key (e.g., config_definitions) should include a
   "singledish_config" named, e.g., "tp" or similar to be associated
   with the measurement set entries. Associated keywords are
   'bl_order' and 'chan_dv_kms'.

3. Output singledish file names need to be defined in the
   singledish_key file. These are also later used to feather or sdint
   image the interferometric data.

4. Then there is a script

### CONTENTS OF THE PIPELINE IN MORE DETAIL

**Architecture**: The pipeline is organized and run by a series of
"handler" objects. These handlers organize the list of targets, array
configurations, spectral products, and derived moments and execute
loops.

The routines to process individual data sets are in individual
modules, grouped by theme (e.g., casaImagingRoutines or
scNoiseRoutines). These routines do not know about the larger
infrastructure of arrays, targets, etc. They generally take an input
file, output file, and various keyword arguments.

A project is defined by a series of text key files in a
"key_directory". These define the measurement set inputs,
configurations, spectral line products, moments, and derived
products.

**User Control**: For the most part the user's job is to *define the
key files* and to run some scripts.