Dallinger
=======
[![Build Status](https://github.com/dallinger/Dallinger/actions/workflows/deploy.yml/badge.svg)](https://github.com/dallinger/Dallinger/actions/workflows/deploy.yml/badge.svg)
[![Demos](https://img.shields.io/badge/demos-11-edd172.svg)](http://dallinger.readthedocs.io/en/latest/#demos)
[![codecov](https://codecov.io/gh/Dallinger/Dallinger/branch/master/graph/badge.svg)](https://codecov.io/gh/Dallinger/Dallinger)
[![Code Climate](https://codeclimate.com/github/Dallinger/Dallinger/badges/gpa.svg)](https://codeclimate.com/github/Dallinger/Dallinger)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](http://en.wikipedia.org/wiki/MIT_License)

<img src="https://raw.githubusercontent.com/Dallinger/Dallinger/master/incubator.png" width="125" alt="Dallinger's incubator">

Laboratory automation for the behavioral and social sciences.

Features
--------
- Performs laboratory automation for the behavioral and social sciences
- Coordinates participant recruitment using Mechanical Turk

Install
-------
To install Dallinger, see [the instructions](http://dallinger.readthedocs.io/).

Documentation
-------------
Documentation is available on the [documentation website](http://dallinger.readthedocs.io/).

Demos
-----
Demonstration experiments are distributed separately in the `dlgr.demos` package.

Contribute
----------
- Issue Tracker: http://github.com/Dallinger/Dallinger/issues
- Source Code: http://github.com/Dallinger/Dallinger

Support
-------
If you are having issues, please [let us know](http://github.com/Dallinger/Dallinger/issues).

**Subscribe to our Slack channel**

Dallinger is *open-source*, and is *completely free* to use in
your experiments. However, to help users with technical issues and
experiment creation, the Dallinger team maintains a **Dallinger
support Slack channel**. Access to the Dallinger support Slack
requires a **$10 per month per user subscription**,
with proceeds supporting the future development of Dallinger. Users are
free to subscribe for as many or as few months as they wish.

To subscribe and gain access to the Slack support channel,
please [download](https://github.com//Dallinger/Dallinger/blob/fb5aeda9a54de7ccc2c3c6d50e2e8c43e9b0366d/docs/source/_static/download/IHO_Dallinger_invoice_TM__1_.docx) and complete this invoice. Completed invoices, along with a check for
the appropriate amount, should be sent to the address on the invoice.
Upon the invoice being processed, the listed users will be added to the
Slack channel. If you are from a developing nation and are unable to
afford a subscription, please contact us directly.

License
-------
The project is licensed under the MIT license.


# ALF

![CI](https://github.com/HorizonRobotics/alf/workflows/CI/badge.svg?branch=pytorch)

Agent Learning Framework (ALF) is a reinforcement learning framework emphasizing on the flexibility and easiness of implementing complex algorithms involving many different components. ALF is built on [PyTorch](https://pytorch.org/). The development of [previous version](https://github.com/HorizonRobotics/alf/tree/tensorflow) based on [Tensorflow 2.1](https://www.tensorflow.org/) has stopped as of Feb 2020.

## Tutorial
A draft tutorial can be accessed on [RTD](https://alf.readthedocs.io/en/latest/tutorial.html). This tutorial is still under
construction and some chapters are unfinished yet.

## Documentation

Read the ALF documentation [here](https://alf.readthedocs.io/).

## Algorithms

|Algorithm|Type|Reference|
|---------|----|---------|
|[A2C](alf/algorithms/actor_critic_algorithm.py)|On-policy RL|[OpenAI Baselines: ACKTR & A2C](https://openai.com/blog/baselines-acktr-a2c/)|
|[PPO](alf/algorithms/ppo_algorithm.py)|On-policy RL|Schulman et al. "Proximal Policy Optimization Algorithms" [arXiv:1707.06347](https://arxiv.org/abs/1707.06347)|
|[PPG](alf/algorithms/ppo_algorithm.py)|On-policy RL|Cobbe et al. "Phasic Policy Gradient" [arXiv:2009.04416](https://arxiv.org/abs/2009.04416)|
|[DDQN](alf/algorithms/dqn_algorithm.py)|Off-policy RL|Hasselt et al. "Deep Reinforcement Learning with Double Q-learning" [arXiv:1509.06461](https://arxiv.org/abs/1509.06461)|
|[DDPG](alf/algorithms/ddpg_algorithm.py)|Off-policy RL|Lillicrap et al. "Continuous control with deep reinforcement learning" [arXiv:1509.02971](https://arxiv.org/abs/1509.02971)|
|[QRSAC](alf/algorithms/qrsac_algorithm.py)|Off-policy RL|Dabney et al. "Distributional Reinforcement Learning with Quantile Regression" [arXiv:1710.10044](https://arxiv.org/abs/1710.10044)|
|[SAC](alf/algorithms/sac_algorithm.py)|Off-policy RL|Haarnoja et al. "Soft Actor-Critic Algorithms and Applications" [arXiv:1812.05905](https://arxiv.org/abs/1812.05905)|
|[OAC](alf/algorithms/oac_algorithm.py)|Off-policy RL|Ciosek et al. "Better Exploration with Optimistic Actor-Critic" [arXiv:1910.12807](https://arxiv.org/abs/1910.12807)|
|[HER](https://github.com/HorizonRobotics/alf/blob/911d9573866df41e9e3adf6cdd94ee03016bf5a8/alf/algorithms/data_transformer.py#L672)|Off-policy RL|Andrychowicz et al. "Hindsight Experience Replay" [arXiv:1707.01495](https://arxiv.org/abs/1707.01495)|
|[TAAC](alf/algorithms/taac_algorithm.py)|Off-policy RL|Yu et al. "TAAC: Temporally Abstract Actor-Critic for Continuous Control" [arXiv:2104.06521](https://arxiv.org/abs/2104.06521)|
|[SEditor](alf/algorithms/seditor_algorithm.py)|Off-policy/Safe RL|Yu et al. "Towards Safe Reinforcement Learning with a Safety Editor Policy" [NeurIPS 2022](https://proceedings.neurips.cc/paper_files/paper/2022/file/11afefdd848d1bc9ac9f1604d9f45817-Paper-Conference.pdf)|
|[DIAYN](alf/algorithms/diayn_algorithm.py)|Intrinsic motivation/Exploration|Eysenbach et al. "Diversity is All You Need: Learning Diverse Skills without a Reward Function" [arXiv:1802.06070](https://arxiv.org/abs/1802.06070)|
|[ICM](alf/algorithms/icm_algorithm.py)|Intrinsic motivation/Exploration|Pathak et al. "Curiosity-driven Exploration by Self-supervised Prediction" [arXiv:1705.05363](https://arxiv.org/abs/1705.05363)|
|[RND](alf/algorithms/rnd_algorithm.py)|Intrinsic motivation/Exploration|Burda et al. "Exploration by Random Network Distillation" [arXiv:1810.12894](https://arxiv.org/abs/1810.12894)|
|[MuZero](alf/algorithms/muzero_algorithm.py)|Model-based RL|Schrittwieser et al. "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model" [arXiv:1911.08265](https://arxiv.org/abs/1911.08265)|
|[BC](alf/algorithms/bc_algorithm.py)|Offline RL|Pomerleau "ALVINN: An Autonomous Land Vehicle in a Neural Network" [NeurIPS 1988](https://papers.nips.cc/paper/1988/hash/812b4ba287f5ee0bc9d43bbf5bbe87fb-Abstract.html)
  Bain et al. "A framework for behavioural cloning" [Machine Intelligence 1999](http://www.cse.unsw.edu.au/~claude/papers/MI15.pdf)|
|[Causal BC](alf/algorithms/causal_bc_algorithm.py)|Offline RL|Swamy et al. "Causal Imitation Learning under Temporally Correlated Noise" [ICML2022](https://proceedings.mlr.press/v162/swamy22a/swamy22a.pdf)|
|[IQL](alf/algorithms/iql_algorithm.py)|Offline RL|Kostrikov, et al. "Offline Reinforcement Learning with Implicit Q-Learning" [arXiv:2110.06169](https://arxiv.org/abs/2110.06169)|
|[MERLIN](alf/algorithms/merlin_algorithm.py)|Unsupervised learning|Wayne et al. "Unsupervised Predictive Memory in a Goal-Directed Agent"[arXiv:1803.10760](https://arxiv.org/abs/1803.10760)|
|[MoNet](alf/algorithms/monet_algorithm.py)|Unsupervised learning|Burgess et al. "MONet: Unsupervised Scene Decomposition and Representation" [arXiv:1901.11390](https://arxiv.org/abs/1901.11390)|
|[Amortized SVGD](alf/algorithms/generator.py)|General|Feng et al. "Learning to Draw Samples with Amortized Stein Variational Gradient Descent" [arXiv:1707.06626](https://arxiv.org/abs/1707.06626)|
|[HyperNetwork](alf/algorithms/hypernetwork_algorithm.py)|General|Ratzlaff and Fuxin. "HyperGAN: A Generative Model for Diverse, Performant Neural Networks" [arXiv:1901.11058](https://arxiv.org/abs/1901.11058)|
|[MCTS](alf/algorithms/mcts_algorithm.py)|General|Grill et al. "Monte-Carlo tree search as regularized policy optimization" [arXiv:2007.12509](https://arxiv.org/abs/2007.12509)|
|[MINE](alf/algorithms/mi_estimator.py)|General|Belghazi et al. "Mutual Information Neural Estimation" [arXiv:1801.04062](https://arxiv.org/abs/1801.04062)|
|[ParticleVI](alf/algorithms/particle_vi_algorithm.py)|General|Liu and Wang. "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm" [arXiv:1608.04471](https://arxiv.org/abs/1608.04471)
 Liu et al. "Understanding and accelerating particle-based variational inference" [arXiv:1807.01750](https://arxiv.org/abs/1807.01750)|
|[GPVI](alf/algorithms/generator.py)|General|Ratzlaff, Bai et al. "Generative Particle Variational Inference via Estimation of Functional Gradients" [arXiv:2103.01291](https://arxiv.org/abs/2103.01291)|
|[SVGD optimizer](alf/optimizers/optimizers.py)|General|Liu et al. "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm" [arXiv:1608.04471](https://arxiv.org/abs/1608.04471)|
|[VAE](alf/algorithms/vae.py)|General|Higgins et al. "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework" [ICLR2017](https://openreview.net/forum?id=Sy2fzU9gl)|
|[RealNVP](alf/networks/normalizing_flow_networks.py)|General|Dinh et al. "Density estimation using Real NVP" [arXiv:1605.08803](https://arxiv.org/abs/1605.08803)|
|[SpatialBroadcastDecoder](alf/networks/encoding_networks.py)|General|Watters et al. "Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs" [arXiv:1901.07017](https://arxiv.org/abs/1901.07017)|
|[VQ-VAE](alf/algorithms/vq_vae.py)|General|A van den Oord et al. "Neural Discrete Representation Learning" [NeurIPS2017](https://proceedings.neurips.cc/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf)|

## Installation

#### OS softwares
The following installation was tested on Ubuntu22.04 with CUDA 11.8.

Python3.11 is currently supported by ALF. Note that some pip packages (e.g., pybullet) need python dev files, so make sure python3.11-dev is installed:

```
sudo apt install -y python3.11 python3.11-dev
```

Boost is also required by ALF for fast parallel environments.

```
sudo apt install libboost-all-dev
```

#### Python environment
[Virtualenv](https://virtualenv.pypa.io/en/latest/) is recommended for the installation. After creating and activating a virtual env, you can run the following commands to install ALF:

```
git clone https://github.com/HorizonRobotics/alf
cd alf
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu118
```

#### For Nix Users

There is a built-in Nix-based development environment defined in [flake.nix](./flake.nix). To activate it, run

```bash
$ nix develop
```

in the root of your local repository.

## Docker
We also provide a docker image of ALF for convenience. In order to use this image, you need to have [docker](https://www.docker.com/) and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) (for ALF gpu usage) installed first.

```bash
docker run --gpus all -it horizonrobotics/cuda:11.8.0-py3.11-torch2.2-ubuntu22.04 /bin/bash
```
This will give you a shell that have all ALF and dependencies pre-installed.

The current docker image contains an ALF version on Feb 21, 2024. Regular version updates are expected in the future.

## Examples

You can train any `_conf.py` file under `alf/examples` as follows:
```bash
python -m alf.bin.train --conf=CONF_FILE --root_dir=LOG_DIR
```
* CONF_FILE is the path to your conf file which follows ALF configuration file format (basically python).
* LOG_DIR is the directory when you want to store the training results. Note that if you want to train from scratch, LOG_DIR must point to a location that doesn't exist. Othewise, it is assumed to resume the training from a previous checkpoint (if any).

During training, we use tensorboard to show the progress of training:
```bash
tensorboard --logdir=LOG_DIR
```

After training, you can evaluate the trained model and visualize environment
frames using the following command:
```bash
python -m alf.bin.play --root_dir=LOG_DIR
```

#### **Deprecated**

An older version of ALF used [gin](https://github.com/google/gin-config)
for job configuration. Its syntax is not as flexible as ALF conf (e.g., you can't easily
do math computation in a gin file). There are still some examples with `.gin`
under `alf/examples`. We are in the process of converting all `.gin` examples to `_conf.py`
examples.

You can train any `.gin` file under `alf/examples` using the following command:
```bash
cd alf/examples; python -m alf.bin.train --gin_file=GIN_FILE --root_dir=LOG_DIR
```
* GIN_FILE is the path to the gin conf (some `.gin` files under `alf/examples` might be invalid; they have not been converted to use the latest pytorch version of ALF).
* LOG_DIR has the same meaning as in the ALF conf example above.

*Warning*: When using **gin**, ALF has to be launched in the same directory with the gin file(s). If an error says that no configuration file is found, then probably you've launched ALF in a wrong place.

All the examples below are trained on a single machine Intel(R) Core(TM) i9-7960X CPU @ 2.80GHz with 32 CPUs and one RTX 2080Ti GPU.

### A2C
* [Cart pole](alf/examples/ac_cart_pole.gin). The training score took only 30 seconds to reach 200, using 8 environments.

* [Atari games](alf/examples/ac_breakout_conf.py). Need to install python package atari-py for atari game environments. The evaluation score (by taking argmax of the policy) took 1.5 hours to reach 800 on Breakout, using 64 environments.

* [Simple navigation with visual input](alf/examples/ac_simple_navigation.gin). Follow the instruction at [SocialRobot](https://github.com/HorizonRobotics/SocialRobot) to install the environment.

### PPO
* [PR2 grasping state only](alf/examples/ppo_pr2.gin). Follow the instruction at [SocialRobot](https://github.com/HorizonRobotics/SocialRobot) to install the environment.

* [Humanoid](alf/examples/ppo_bullet_humanoid.gin). Learning to walk using the pybullet Humanoid environment. Need to install python pybullet>=2.5.0 for the environment. The evaluation score reaches 3k in 50M steps, using 96 parallel environments.

### PPG

* [procgen](alf/examples/ppg_procgen_bossfight_conf.py). Game "bossfight" as an example. Need to install python package [procgen](https://github.com/openai/procgen).

* [MetaDrive](alf/examples/metadrive/ppg_metadrive_conf.py). Learning to drive on randomly generated map with interaction on the [MetaDrive](https://metadriverse.github.io/metadrive/) simulator, with BEV as input. Need to install python package [metadrive-simulator](https://github.com/metadriverse/metadrive).

### DDQN
* [DDQN on Atari](alf/examples/dqn_breakout_conf.py).  Game "Q*Bert" performance.

### DDPG
* [FetchSlide (sparse rewards)](alf/examples/ddpg_fetchslide_conf.py). Need to install the [MuJoCo](https://www.roboti.us/index.html) simulator first. This example reproduces the performance of vanilla DDPG reported in the OpenAI's Robotics environment [paper](https://arxiv.org/pdf/1802.09464.pdf). Our implementation doesn't use MPI, but obtains (evaluation) performance on par with the original implementation. (*The original MPI implementation has 19 workers, each worker containing 2 environments for rollout and sampling a minibatch of size 256 from its replay buffer for computing gradients. All the workers' gradients will be summed together for a centralized optimizer step. Our implementation simply samples a minibatch of size 5000 from a common replay buffer per optimizer step.*) The training took about 1 hour with 38 (19*2) parallel environments on a single GPU.

### SAC
* [Bipedal Walker](alf/examples/sac_bipedal_walker_conf.py).

* [FetchReach (sparse rewards)](alf/examples/sac_fetchreach.gin). Need to install the [MuJoCo](https://www.roboti.us/index.html) simulator first. The training took about 20 minutes with 20 parallel environments on a single GPU.

* [FetchSlide (sparse rewards)](alf/examples/sac_fetchslide.gin). Need to install the [MuJoCo](https://www.roboti.us/index.html) simulator first. This is the same task with the DDPG example above, but with SAC as the learning algorithm.
Also it has only 20 (instead of 38) parallel environments to improve sample efficiency. The training took about 2 hours on a single GPU.

* [Fetch Environments (sparse rewards) w/ Action Repeat](alf/examples/sac_actrepeat_fetch/sac_actrepeat_fetch.gin). We are able to achieve even better performance than reported by DDPG + Hindsight Experience Replay in some cases simply by using SAC + Action Repeat with length 3 timesteps. See [this note](docs/notes/fetch_env_sac_actrepeat_notes.rst) to view learning curves, videos, and more details.

### ICM
* [Super Mario](alf/examples/icm_super_mario_intrinsic_only.gin). Playing Super Mario only using intrinsic reward.
  Python package gym-retro>=0.7.0 is required for this experiment and also a suitable `SuperMarioBros-Nes` rom should be obtained and imported (roms are not included in gym-retro). See [this doc](https://retro.readthedocs.io/en/latest/getting_started.html#importing-roms) on how to import roms.

### RND
* [Montezuma's Revenge](alf/examples/ppo_rnd_mrevenge.gin). Training the hard exploration game Montezuma's Revenge with intrinsic rewards generated by [RND](alf/algorithms/rnd_algorithm.py). A lucky agent can get an episodic score of 6600 in 160M frames (40M steps with `frame_skip=4`). A normal agent would get an episodic score of 4000~6000 in the same number of frames. The training took about 6.5 hours with 128 parallel environments on a single GPU.

### DIAYN
* [Pendulum](alf/examples/diayn_pendulum.gin). Learning diverse skills without external rewards.

### BC
* [Pendulum](alf/examples/hybrid_rl/bc_pendulum_conf.py). Learning a control policy from [offline demonstrations](alf/examples/hybrid_rl/replay_buffer_data/pendulum_replay_buffer_from_sac_10k).

### Merlin
* [Collect Good Objects](alf/examples/merlin_dmlab_collect_good_objects.gin). Learn to collect good objects and avoid bad objects.
  `DeepmindLab` is required,  Follow the instruction at [DeepmindLab](https://github.com/deepmind/lab/blob/master/python/pip_package/README.md) to install the environment.

### MuZero
* [6x6 Go](alf/examples/muzero_go_6x6.gin). It took about a day to train a reasonable agent to play 6x6 go using one GPU.

## Citation
If you use ALF for research and find it useful, please consider citing:
```
@software{Xu2021ALF,
  title={{{ALF}: Agent Learning Framework}},
  author={Xu, Wei and Yu, Haonan and Zhang, Haichao and Hong, Yingxiang and Yang, Break and Zhao, Le and Bai, Jerry and ALF contributors},
  url={https://github.com/HorizonRobotics/alf},
  year={2021}
}
```

## Contribute to ALF

You are welcome to contribute to ALF. Please follow the guideline [here](https://alf.readthedocs.io/en/latest/contributing.html).


[![Build Status](https://dev.azure.com/TileDB-Inc/CI/_apis/build/status/TileDB-Inc.TileDB-Py?branchName=dev)](https://dev.azure.com/TileDB-Inc/CI/_build/latest?definitionId=1&branchName=dev)
![](https://raw.githubusercontent.com/TileDB-Inc/TileDB/dev/doc/anaconda.svg?sanitize=true)[![Anaconda download count badge](https://anaconda.org/conda-forge/TileDB-Py/badges/downloads.svg)](https://anaconda.org/conda-forge/TileDB-Py)

# TileDB-Py

*TileDB-Py* is a [Python](https://python.org) interface to the [TileDB Storage Engine](https://github.com/TileDB-Inc/TileDB).

# Quick Links

* [Installation](https://docs.tiledb.com/developer/installation/quick-install)
* [Build Instructions](https://docs.tiledb.com/main/how-to/installation/building-from-source/python)
* [TileDB Documentation](https://docs.tiledb.com/main/)
* [Python API reference](https://tiledb-inc-tiledb-py.readthedocs-hosted.com/en/stable)

# Quick Installation

TileDB-Py is available from either [PyPI](https://pypi.org/project/tiledb/) with ``pip``:

```
pip install tiledb
```

or from [conda-forge](https://anaconda.org/conda-forge/tiledb-py) with
[conda](https://conda.io/docs/) or [mamba](https://github.com/mamba-org/mamba#installation):

```
conda install -c conda-forge tiledb-py
```

Dataframes functionality (`tiledb.from_pandas`, `Array.df[]`) requires [Pandas](https://pandas.pydata.org/) 1.0 or higher, and [PyArrow](https://arrow.apache.org/docs/python/) 1.0 or higher.

# Contributing

We welcome contributions, please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for suggestions and
development-build instructions. For larger features, please open an issue to discuss goals and
approach in order to ensure a smooth PR integration and review process.


EvalML is an AutoML library which builds, optimizes, and evaluates machine learning pipelines using domain-specific objective functions.

**Key Functionality**

* **Automation** - Makes machine learning easier. Avoid training and tuning models by hand. Includes data quality checks, cross-validation and more.
* **Data Checks** - Catches and warns of problems with your data and problem setup before modeling.
* **End-to-end** - Constructs and optimizes pipelines that include state-of-the-art preprocessing, feature engineering, feature selection, and a variety of modeling techniques.
* **Model Understanding** - Provides tools to understand and introspect on models, to learn how they'll behave in your problem domain.
* **Domain-specific** - Includes repository of domain-specific objective functions and an interface to define your own.

## Installation

Install from [PyPI](https://pypi.org/project/evalml/):

```bash
pip install evalml
```

or from the conda-forge channel on [conda](https://anaconda.org/conda-forge/evalml):

```bash
conda install -c conda-forge evalml
```

### Add-ons
**Update checker** - Receive automatic notifications of new Woodwork releases

PyPI:

```bash
pip install "evalml[updater]"
```
Conda:
```
conda install -c conda-forge alteryx-open-src-update-checker
```

## Start

#### Load and split example data
```python
import evalml
X, y = evalml.demos.load_breast_cancer()
X_train, X_test, y_train, y_test = evalml.preprocessing.split_data(X, y, problem_type='binary')
```

#### Run AutoML
```python
from evalml.automl import AutoMLSearch
automl = AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')
automl.search()
```

#### View pipeline rankings
```python
automl.rankings
```

#### Get best pipeline and predict on new data
```python
pipeline = automl.best_pipeline
pipeline.predict(X_test)
```

## Next Steps

Read more about EvalML on our [documentation page](https://evalml.alteryx.com/):

* [Installation](https://evalml.alteryx.com/en/stable/install.html) and [getting started](https://evalml.alteryx.com/en/stable/start.html).
* [Tutorials](https://evalml.alteryx.com/en/stable/tutorials.html) on how to use EvalML.
* [User guide](https://evalml.alteryx.com/en/stable/user_guide.html) which describes EvalML's features.
* Full [API reference](https://evalml.alteryx.com/en/stable/api_reference.html)

## Support

The EvalML community is happy to provide support to users of EvalML. Project support can be found in four places depending on the type of question:
1. For usage questions, use [Stack Overflow](https://stackoverflow.com/questions/tagged/evalml) with the `evalml` tag.
2. For bugs, issues, or feature requests start a [Github issue](https://github.com/alteryx/evalml/issues).
3. For discussion regarding development on the core library, use [Slack](https://join.slack.com/t/alteryx-oss/shared_invite/zt-182tyvuxv-NzIn6eiCEf8TBziuKp0bNA).
4. For everything else, the core developers can be reached by email at open_source_support@alteryx.com

## Built at Alteryx

**EvalML** is an open source project built by [Alteryx](https://www.alteryx.com). To see the other open source projects we’re working on visit [Alteryx Open Source](https://www.alteryx.com/open-source). If building impactful data science pipelines is important to you or your business, please get in touch.


[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg)](https://github.com/prettier/prettier)

Mephisto makes crowdsourcing easier.

We provide a platform for launching, monitoring, and reviewing your crowdsourcing tasks. Tasks made on Mephisto can easily be open sourced as part of other projects. Like the [chess-playing automaton](https://en.wikipedia.org/wiki/Mephisto_(automaton)) we've adopted the name from, Mephisto hides the complexity of projects that need human interaction or input.

You can find complete details about the project on our [docs website](https://mephisto.ai).

## Quickstart

[Get started in 10 minutes](https://mephisto.ai/docs/guides/quickstart)

[How to run projects](mephisto/README.md)

## Want to help?

Check out our [guidelines for contributing](https://github.com/facebookresearch/Mephisto/blob/main/CONTRIBUTING.md) and then take a look at some of our tagged issues: [good first issue](https://github.com/facebookresearch/Mephisto/labels/good%20first%20issue), [help wanted](https://github.com/facebookresearch/Mephisto/labels/help%20wanted).

For library authors, you may also find the [how to contribute](https://mephisto.ai/docs/guides/how_to_contribute/getting_started) documentation helpful.

## License
Mephisto is MIT licensed. See the LICENSE file for details.

## Citation
If you use Mephisto in your work, please consider citing the project. It helps us prioritize features based on our users, and helps others discover implementations that may be relevant to their projects.
```bibtex
@misc{mephisto,
  doi = {10.48550/ARXIV.2301.05154},
  url = {https://arxiv.org/abs/2301.05154},
  author = {Urbanek, Jack and Ringshia, Pratik},
  keywords = {Artificial Intelligence (cs.AI), Human-Computer Interaction (cs.HC), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Mephisto: A Framework for Portable, Reproducible, and Iterative Crowdsourcing},
  publisher = {arXiv},
  year = {2024},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
```


# Open CAS Linux

[![Build Status](https://d1rxsi9lvcwnz5.cloudfront.net/master-status/ocl/build/curr-badge.svg)](https://d1rxsi9lvcwnz5.cloudfront.net/master-status/ocl/build/build.html)
[![Tests Status](https://d1rxsi9lvcwnz5.cloudfront.net/master-status/ocl/tests/curr-badge.svg)](https://d1rxsi9lvcwnz5.cloudfront.net/master-status/ocl/tests/tests.html)
[![Coverity status](https://scan.coverity.com/projects/19084/badge.svg)](https://scan.coverity.com/projects/open-cas-open-cas-linux)
[![License](https://d1rxsi9lvcwnz5.cloudfront.net/master-status/license-badge.svg)](LICENSE)

Open CAS  accelerates Linux applications by caching active (hot) data to
a local flash device inside servers. Open CAS implements caching at the
server level, utilizing local high-performance flash media as the cache drive
media inside the application server as close as possible to the CPU, thus
reducing storage latency as much as possible.
The Open Cache Acceleration Software installs into the GNU/Linux operating
system itself, as a kernel module. The nature of the integration provides a
cache solution that is transparent to users and  applications, and your
existing storage infrastructure. No storage migration effort or application
changes are required.

Open CAS is distributed on BSD-3-Clause license (see
https://opensource.org/licenses/BSD-3-Clause for full license texts).

Open CAS uses Safe string library (safeclib) that is MIT licensed.

## Installation

We recommend using the latest version, which contains all the important fixes
and performance improvements. Bugfix releases are guaranteed only for the
latest major release line (currently 22.6.x).

To download the latest Open CAS Linux release run following commands:

```
wget https://github.com/Open-CAS/open-cas-linux/releases/download/v22.6.1/open-cas-linux-22.06.1.0712.release.tar.gz
tar -xf open-cas-linux-22.06.1.0712.release.tar.gz
cd open-cas-linux-22.06.1.0712.release/
```

Alternatively, if you want recent development (unstable) version, you can clone GitHub repository:

```
git clone https://github.com/Open-CAS/open-cas-linux
cd open-cas-linux
git submodule update --init
```

### Source compile and install

To install all required python packages run the following command:

```
python3 -m pip install -r requirements.txt
```

To configure, build and install Open CAS Linux run following commands:

```
./configure
make
make install
```

The `./configure` performs check for dependencies, so if some of them are missing,
command will print their names in output. After installing missing dependencies
you need to run `./configure` once again - this time it should succeed.

> NOTE: If after installing CAS, your system boots into emergency mode due to the
> **"Failed to start opencas initialization service."** error, you need to force SELinux
> relabelling in permissive mode on your filesystem.\
> Refer to the [Open CAS documentation](https://open-cas.github.io/guide_running.html#rebooting-power-cycling-and-open-cas-linux-autostart) for details.

### RPM/DEB install

Alternatively, you can generate RPM/DEB packages from downloaded sources and
install those packages instead. To do so, simply run:

__on RPM based systems:__
```
make rpm
rm -f packages/*debug*
dnf install ./packages/open-cas-linux*.rpm
```

__on DEB based systems:__
```
make deb
apt install ./packages/open-cas-linux*.deb
```

Package generating script will inform you of any missing dependencies.
You can find detailed instructions in the [Open CAS documentation](https://open-cas.github.io/guide_installing.html#creating-rpmdeb-packages)

## Getting Started

To quickly deploy Open CAS Linux in your system please follow the instructions
available [here](https://open-cas.github.io/getting_started_open_cas_linux.html).

## Documentation

The complete documentation for Open CAS Linux is available in the
[Open CAS Linux Administration Guide](https://open-cas.github.io/guide_introduction.html).

## Running Tests

Before running tests make sure you have a platform with at least 2 disks (one for cache and one for core). Be careful as these devices will be most likely overwritten with random data during tests. Tests can be either executed locally or on a remote platform (via ssh) specified in the dut_config.

1. Go to test directory `cd test/functional`.
1. Install dependencies with command `pip3 install -r requirements.txt`.
1. Create DUT config. See example [here](test/functional/config/example_dut_config.yml).
    a) Set disks params. You need at least two disks, of which at least one is an SSD drive.
    b) For remote execution uncomment and set the `ip`, `user` and `password` fields.
    c) For local execution just leave these fields commented.
1. Run tests using command `pytest-3 --dut-config=<CONFIG>` where `<CONFIG>` is path to your config file, for example `pytest-3 --dut-config="config/dut_config.yml"`.

## Contributing

Feel like making Open CAS Linux better? Don't hesitate to submit a pull request!
You can find more information about our contribution process
[here](https://open-cas.github.io/contributing.html).
In case of any questions feel free to contact [maintainer](mailto:robert.baldyga@huawei.com).


  tellurium

    <img alt="Licence", src="https://img.shields.io/hexpm/l/tellurium"

Copyright 2014-2023
Kiri Choi, J Kyle Medley, Matthias König, Kaylene Stocking, Caroline Cannistra, Michal Galdzicki, Ciaran Welsh, Lucian Smith, Adel Heydarabadipour, and Herbert Sauro

## Introduction

Tellurium is a python environment for reproducible dynamical modeling of biological networks.
Tellurium provides the interfacial code to convert between standard formats and utilize powerful
libraries without requiring technical expertise, allowing you to focus on what’s important:
building better models. Tellurium also provides first-class support for exchangeability via
[COMBINE archives](http://co.mbine.org/documents/archive), allowing you to share your models
and simulations with other tools.

Tellurium combines state-of-the-art scientific Python libraries, such
as [NumPy](http://www.numpy.org/) and [SciPy](http://www.scipy.org/),
and includes special-purpose systems biology Python tools. Out of the box,
Tellurium includes [libroadrunner](https://github.com/sys-bio/roadrunner),
[antimony](http://antimony.sourceforge.net/), [phrasedml](http://phrasedml.sf.net/),
[libsbml](http://sbml.org/Software/libSBML), and [libsedml](https://github.com/fbergmann/libSEDML).

The Tellurium (and libroadrunner project) project is funded from the NIH/NIGMS (GM081070) and NIH/NIBIB U24EB028887.

## Documentation
* General: http://tellurium.readthedocs.org/en/latest/
* API: http://tellurium.readthedocs.io/en/latest/_apidoc/tellurium.html
* Webpage: http://tellurium.analogmachine.org/

## Usage Example

```{python}
import tellurium as te

rr = te.loada('''
    model example0
      S1 -> S2; k1*S1
      S1 = 10
      S2 = 0
      k1 = 0.1
    end
''')

result = rr.simulate(0, 40, 500)
te.plotArray(result)
```

![Tellurium front page demo](./docs/images/tellurium-front-page-image.png)

## Installation Instructions

:exclamation:**Attention to those using Python 2.7 :**
**[Python 2.7 reached EOL on January 1st, 2020](https://www.python.org/doc/sunset-python-2/). Support for Python 2.7 has been dropped, though previous releases will continue to be available.**

Tellurium can be installed as a Python package, with a front-end (Notebook, IDE), or in a cloud environment (Google Colab):

A. [Python package (pip)](#install-via-pip-no-front-end)
B. [IDE front-end](#front-end-1-spyder-for-tellurium-ide) based on [Spyder](https://www.spyder-ide.org/)
C. [Notebook front-end](#front-end-2-tellurium-notebook)
D. [Google Colab](#google-colab)

We recommend first-time users choose one of the front-ends, while developers looking to integrate Tellurium use the pip package. All python desktop IDE graphical front-ends should work. We know for certain that telluirum will work from Spyder, pyCharm and Visual Studio Code.

The IDE front-end provides a MATLAB like experience with a code editor and Python console.
The notebook front-end provides a notebook interface similar to [Jupyter](http://jupyter.org/),
and features notebook cells for inline OMEX, a human-readable representation of COMBINE archives.

For any issues with installation and setup, please check [FAQ](https://github.com/sys-bio/tellurium/wiki/FAQ) or [contact us](#contact-us).

-------

### Install via pip
[![PyPI version](https://badge.fury.io/py/tellurium.svg)](https://badge.fury.io/py/tellurium)

If you have a version of Python in an environment you're comfortable with, you can add Tellurium with pip:

```
pip install tellurium
```

Tellurium itself is Python-only, and is available for any version of Python 3.  Its main binary dependency, roadrunner, should be installed with Tellurium, and should be available for the latest three or four versions of Python (currently 3.8, 3.9, 3.10, and 3.11), for Windows, Mac (both Intel and ARM), and Linux.

For those using Anaconda, we currently do not have a separate Anaconda version of Tellurium, so you'll need to install it via pip, as above.

:exclamation:**For detailed instructions on how to setup Tellurium on Anaconda distributions,
see [this page](https://github.com/sys-bio/tellurium/wiki/FAQ#i-would-like-to-use-tellurium-on-anaconda-what-should-i-do).**

For developers, the latest stable version from the repository can be installed via
```
pip install git+https://github.com/sys-bio/tellurium.git
```

-------

### Front-end installation

Windows

#### Approach 1: Spyder for Tellurium IDE
The Tellurium Spyder installers are tested with Windows 10 and 11, and come with a choice of Python versions.

:exclamation:**Attention to those upgrading to Tellurium Spyder IDE from an older version :**
We strongly suggest you to completely remove the older version of Tellurium Spyder IDE prior to installing the latest version.

1. Download Tellurium Spyder for Windows:
    * [Python 3.7](https://sourceforge.net/projects/pytellurium/files/Tellurium-2.3/2.3.5/Tellurium-2.3.5-Python-3.7-win64-setup.exe/download)
    * [Python 3.8](https://sourceforge.net/projects/pytellurium/files/Tellurium-2.3/2.3.5/Tellurium-2.3.5-Python-3.8-win64-setup.exe/download)
    * [Python 3.10](https://sourceforge.net/projects/pytellurium/files/Tellurium-2.4/2.4.0/Tellurium-2.4.0-Python-3.10-win64-setup.exe/download)
    * [Python 3.11](https://sourceforge.net/projects/pytellurium/files/Tellurium-2.4/2.4.0/Tellurium-2.4.0-Python-3.11-win64-setup.exe/download)
2. Double-click the installer to start the installation
3. Follow the instructions

If you wish to use a different version of Python, you'll need to install Tellurium from the windows command line by using the command: pip install tellurium

NOTE: Installation requires administrative rights. It is recommended to accept the default settings.

#### Approach 2: Thonny IDE for Python
Tellurium can be used with [Thonny](https://en.wikipedia.org/wiki/Thonny), a free, open-source IDE for Python that is designed for beginners.

1. [Download the latest version of Thonny for Windows](https://thonny.org)
2. Double-click the installer `*.exe` to start the installation
3. Follow the instructions
4. Launch Thonny
5. Go to Tools > Manage packages...
6. Type `tellurium` in the search box and press the "Search on PyPI" button
7. Once the `tellurium` package is found in "Search results", click on it, and then press the "Install" button. Wait until the installation process is completed.

#### Approach 3: Tellurium Notebook
The Tellurium notebook is no longer being updated, but is still available.  It was tested with Windows 10. The notebook viewer comes with Python 3.6 (64-bit).

1. [Download Tellurium Notebook for Windows](https://sourceforge.net/projects/pytellurium/files/notebook/Tellurium%20Setup%202.1.1.exe/download)
2. Double-click the installer `*.exe` to start the installation
3. Follow the instructions

Mac OS X

For Mac users there a a number of options. By far, the easiest is to use the self-cotbained Thonny Python IDE.

#### Approach 1: Thonny IDE for Python
Tellurium can be used with [Thonny](https://en.wikipedia.org/wiki/Thonny), a free, open-source IDE for Python that is designed for beginners.

1. [Download the latest version of Thonny for Mac OS X](https://thonny.org)
2. Double-click the installer `*.pkg` to start the installation
3. Follow the instructions
4. Launch Thonny from Spotlight or directly from your Applications folder
5. Go to Tools > Manage packages...
6. Type `tellurium` in the search box and press the "Search on PyPI" button
7. Once the `tellurium` package is found in "Search results", click on it, and then press the "Install" button. Wait until the installation process is completed.

#### Approach 2: Spyder for Tellurium IDE

1. [Download the Anaconda distribution for Mac OS X.](https://www.anaconda.com/download).

2. [Follow the installer instructions.](https://docs.anaconda.com/anaconda/install/linux). The installer will install Spyder as part of the installation process. If you customize the installation, you will have a chance to select which packages to install - ensure Spyder is selected.
3. When prompted, add Anaconda to your PATH (optional, but this will make the following steps easier).
4. [Open a Terminal](https://www.wikihow.com/Open-a-Terminal-Window-in-Mac) and run the command:
```
conda install msgpack-python
```
Next, install Tellurium itself:
```
pip install tellurium
```
If this fails, try using the default absolute path to Anaconda, which will probably be something like:
```
/Users/<your-user>/opt/anaconda3/bin/pip install tellurium
```
5. Launch the Anaconda Navigator via Launchpad. In Anaconda Navigator, run Spyder and then try `import tellurium` within the Spyder editor or console.

#### Approach 3: Spyder for Tellurium IDE (only for Mac OS X Legacy)

Currently, only a Legacy version of the Mac Spyder IDE is available. We recommend using the Notebook front-end on Mac.

1. [Download Tellurium IDE for Mac OS X 10.10 or later](https://github.com/sys-bio/tellurium/releases/download/1.3.5-rc3/Tellurium-1.3.5-Spyder-2.3.8-OSX.dmg)
2. Double-click the .dmg file to open a new window
3. Double-click the Spyder icon

#### Approach 4: Tellurium Notebook
The Tellurium notebook is no longer being updated, but is still available.  It was tested with Mac OS X 10.10+. The notebook viewer comes with Python 3.6 (64-bit).

1. [Download Tellurium Notebook for Mac OS X 10.10 or later](https://sourceforge.net/projects/pytellurium/files/notebook/Tellurium-2.1.1.dmg/download)
2. You may need to [disable Gatekeeper](https://github.com/sys-bio/tellurium/wiki/FAQ#on-mac-after-downloading-tellurium-i-cant-open-it-because-it-is-from-an-unidentified-developer)
3. Double-click the `*.dmg` file to open a new window
4. Drag the Tellurium icon to your Applications
5. You can now launch Tellurium from Spotlight or directly from your applications folder

**For those who wish to use Tellurium with Spyder IDE on Mac OS X, we also suggest that you install through [Anaconda](https://www.anaconda.com/) by
following the instructions on [this page](https://github.com/sys-bio/tellurium/wiki/FAQ#i-would-like-to-use-tellurium-on-anaconda-what-should-i-do).**

Linux (RedHat)

#### Tellurium Notebook
The Tellurium notebook is no longer being updated, but is still available.  It was tested with Fedora 22+. The notebook viewer comes with Python 3.6 (64-bit).

1. [Download Tellurium Notebook (.rpm)](https://sourceforge.net/projects/pytellurium/files/notebook/Tellurium-2.1.1.rpm/download)
2. Install the package using `dnf install Tellurium-2.1.1.rpm`
3. You should be able to launch Tellurium from your activities pane. If not, log out and in again or run `tellurium` from the terminal.

Linux (Debian)

#### Tellurium Notebook
The Tellurium notebook is no longer being updated, but is still available.  It was tested with Debian 8+. The notebook viewer comes with Python 3.6 (64-bit).

1. [Download Tellurium Notebook (.deb)](https://sourceforge.net/projects/pytellurium/files/notebook/Tellurium_2.1.1_amd64.deb/download)
2. Install the package using `dpkg -i Tellurium_2.1.1_amd64.deb`
3. You should be able to launch Tellurium from your activities pane. If not, log out and in again or run `/opt/Tellurium/tellurium` from the terminal.

**For those who wish to use Tellurium with Spyder IDE on Linux, we also suggest that you install through [Anaconda](https://www.anaconda.com/) by
following the instructions on [this page](https://github.com/sys-bio/tellurium/wiki/FAQ#i-would-like-to-use-tellurium-on-anaconda-what-should-i-do).**

:question:Looking for old releases? [You can find them here.](https://sourceforge.net/projects/pytellurium/files/notebook) To install an old release over a new release, you will need to [manually wipe the data directory.](https://github.com/sys-bio/tellurium/wiki/FAQ#how-do-i-uninstall-tellurium-notebook)

-------

### Google Colab
Tellurium can be used entirely in a browser using [Google Colab](https://colab.research.google.com/).  Ideally, it will work with the following steps (updated May 2023)

1. run a cell with ```!apt-get install libncurses5```
2. run a cell with ```!pip install -q tellurium==2.2.8```
3. (Very important) Restart the runtime (From the menu: 'Runtime / Restart runtime')
4. Test by typing ```import tellurium as te```

The Python version behind Colab changes periodically, so what worked one day may stop working the next, but the following Colab notebook worked when used at ICSB 2022:  https://colab.research.google.com/drive/1wddLftHNhetbozZY29r2HRkzQLl1F_fs#scrollTo=l1bCgW46-adR and will hopefully be instructive.

-------

## Citing

If you use Tellurium in your research, we would appreciate following citations in any works you publish:

Medley et al. (2018). ["Tellurium notebooks—An environment for reproducible dynamical modeling in systems biology."](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006220) *PLoS Computational Biology*, 14(6), e1006220.

Choi et al. (2018). ["Tellurium: An extensible python-based modeling environment for systems and synthetic biology."](https://www.sciencedirect.com/science/article/pii/S0303264718301254) *Biosystems*, 171, 74-79.

## Contact Us

For general questions, requesting help, or reporting bugs, feel free to use the [GitHub issue tracker](https://github.com/sys-bio/tellurium/issues).  You can also post to the  [Tellurium-discuss](https://groups.google.com/d/forum/tellurium-discuss) mailing list.

## Legal

The source code for the Tellurium Python package is hosted at https://github.com/sys-bio/tellurium and is licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0). Tellurium uses third-party dependencies which may be licensed under different terms. Consult the documentation for the respective third-party packages for more details.

TELLURIUM AND ALL SOFTWARE BUNDLED WITH TELLURIUM (HEREAFTER "THE SOFTWARE") IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THE SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


#### Starbound Frackin' Universe Simplified Chinese Translation Project

* Project：ProjectSky
* Credit：TODO

#### 注意

* ~~请不要再往该仓库的任何分支提交拉请求，此仓库仅作为归档使用，远程gitlab仓库的更改均会自动同步至此。~~
* ~~项目移至gitlab 主页：[FrackinUniverse-sChinese-Project](https://gitlab.imsky.cc/ProjectSky-Translation-Team/FrackinUniverse-sChinese-Project)~~
* 悲报，gitlab炸了，项目暂时整回github了！

#### 停止维护

* 很遗憾，由于 Frackin' Universe 更新频繁，两个人并没有足够的精力维护这个项目的频繁更新，现项目停止维护并转交自动构建系统。

#### 自动构建说明

* 自动构建于 Frackin' Universe Release 后自动更新，自动更新最新的 Frackin' Universe 文本，并合并 FAST 分支中已有的文本。
* 简单的来说就是FU更新了一个版本后，自动构建会输出上个版本的FU汉化mod。

#### 提交拉请求说明

* 任何拉请求请提交至 FAST 分支，其他分支仅用作合并处理。

#### 协助翻译流程

* ~~在线编辑（此方法比较方便，编辑器带有语法检查，可以防止语法错误，不过浏览器崩了或其他原因会丢失数据，还是推荐克隆到本地进行编辑。）~~
* 请参阅[JOIN_US.md](https://github.com/ProjectSky/FrackinUniverse-sChinese-Project/blob/FAST/documents/JOIN_US.md)
* 汉化小白请参阅[小白专属教程](https://github.com/ProjectSky/FrackinUniverse-sChinese-Project/blob/FAST/documents/JOIN_US_FRASHMAN.md)
* 快速汉化请参阅[快速汉化教程](https://github.com/ChenJunsen/StarboundAutoTranslator)
* 批量汉化请参阅[批量汉化教程](https://github.com/ChenJunsen/starbound_auto_transalator_advanced)

#### 链接信息

* [最新版本下载](https://github.com/ProjectSky/FrackinUniverse-sChinese-Project/releases)
* [翻译进度查阅](https://projectsky.github.io/FrackinUniverse-sChinese)
* [Telegram群组](https://t.me/joinchat/DR0AV0kjlPgEmltSqwr12Q)
* [Telegram web](https://projectsky.github.io/telegram)
* 无法使用Telegram应用？使用上方WEB版本即可。


.. image:: https://badge.fury.io/py/psyneulink.svg
    :target: https://badge.fury.io/py/psyneulink
.. image:: https://github.com/PrincetonUniversity/PsyNeuLink/workflows/PsyNeuLink%20CI/badge.svg?branch=master
    :target: https://github.com/PrincetonUniversity/PsyNeuLink/actions
.. image:: https://mybinder.org/badge.svg
    :target: https://mybinder.org/v2/gh/PrincetonUniversity/PsyNeuLink/master

.. *****************************************************************************************
.. ****** NOTE:  UPDATES TO THIS PAGE SHOULD ALSO BE MADE TO docs/source.index.rst *********
.. *****************************************************************************************

Welcome to PsyNeuLink
=====================

(pronounced: /sīnyoolingk - sigh-new-link)
Documentation is available at https://princetonuniversity.github.io/PsyNeuLink/

Purpose
-------

PsyNeuLink is an open-source, software environment written in Python, and designed for the needs of
neuroscientists, psychologists, computational psychiatrists and others interested in learning about and building
models of the relationship between brain function, mental processes and behavior.

PsyNeuLink can be used as a "block modeling environment", in which to construct, simulate, document, and exchange
computational models of neural mechanisms and/or psychological processes at the subsystem and system levels.
A block modeling environment allows components to be constructed that implement various, possibly disparate
functions, and then link them together into a system to examine how they interact.  In PsyNeuLink, components are
used to implement the function of brain subsystems and/or psychological processes, the interaction of which can then
be simulated at the system level.

The purpose of PsyNeuLink is to make it as easy as possible to create new and/or import existing models, and
integrate them to simluate system-level interactions.  It provides a suite of core components for
implementing models of various forms of processing, learning, and control, and its Library includes examples that
combine these components to implement published models.  As an open source project, its suite of components is meant
to be enhanced and extended, and its library is meant to provide an expanding repository of models, written in a
concise, executable, and easy to interpret form, that can be shared, compared, and extended by the scientific community.

What PsyNeuLink **IS**
----------------------

It is:

 - *open source*, freeing users of the costs or restrictions associated with proprietary software.

 ..

 - *computationally general* -- it can be used to implement, seamlessly integrate, and simulate interactions among
   disparate components that vary in their granularity of representation and function (from individual neurons or
   neural populations to functional subsystems and abstract cognitive functions) and at any time scale of execution.

 ..

 - *integrative* -- it provides a standard and accessible environment for model comparison, sharing, and documentation;

 ..

 - *extensible* -- it has an interface (API) that allows it to be used with other powerful tools for implementing
   individual components, such as:

    * Neuron and Nengo (biophysically realistic models of neuronal function)
    * Emergent (broad class of neurally-plausible connectionist models);
    * Pytorch and TensorFlow (ODE's, deep learning);
    * ACT-R (symbolic, production system models).

.. note::
   PsyNeuLink is alpha software, that is still being actively developed.  Although it is useable, and most of the
   documented functionality is available, some features may not yet be fully implemented and/or subject to
   modification.  Please report any bugs and/or suggestions for development to psyneulinkhelp@princeton.edu.

What PsyNeuLink is **NOT**
--------------------------

The longterm goal of PsyNeuLink is to provide an environment that integrates comptutational modeling of brain function
and behavior at all levels of analysis.  While it is designed to be fully general, and can in principle be used to
implement models at any level, it is still under development, and current efficiency considerations make it more
suitable for some of forms of modeling than others. In its present form, it is well suited to the creation of
simple to moderately complex models, and for the integration of disparate models into a single environment, while in
it is presently less well suited to efforts involving massively large computations, such as:

 - extensive model fitting
 - large scale simulations
 - highly detailed biophysical models of neurons or neuronal populations

Other packages currently better suited to such applications are:
`Emergent <https://grey.colorado.edu/emergent/index.php/Main_Page>`_ for biologically-inspired neural network models
`Pytorch <https://pytorch.org>`_ and `TensorFlow <https://www.tensorflow.org>`_ (for deep learning models);
`HDDM <http://ski.clps.brown.edu/hddm_docs/>`_ (for Drift Diffusion Models);
`ACT-R <http://act-r.psy.cmu.edu>`_ (for production system models);
`Genesis <http://www.genesis-sim.org>`_,
`Neuron <https://www.neuron.yale.edu/neuron/>`_,
and `Nengo <http://www.nengo.ca>`_  (for biophysically-realistic models of neuronal function).

These packages are good for elaborate and detailed models of a particular form. In contrast, the focus in designing
PsyNeuLink has been to make it as flexible and easy to use as possible, with the ability to integrate components
constructed in other packages (including some of the ones listed above) into a single environment.  These are
characteristics that are often (at least in the initial stages of development) in tension with efficiency (think:
interpreted vs. compiled).

That said, priorities for ongoing development of PsyNeuLink are:
    i) acceleration, using just-in-time compilation methods and parallelization
       (see `Compilation`, and `Vesely et al., 2022 <http://www.cs.yale.edu/homes/abhishek/jvesely-cgo22.pdf>`_);
    ii) enhancement of the API to facilitate wrapping modules from other packages for integration into the PsyNeuLink
        environment (examples currently exist for `Pytorch <https://pytorch.org>`_ ) and translating into a standard
        `Model Description Format (MDF) <https://github.com/ModECI/MDF>`_;
    iii) integration of tools for parameter estimation, model comparison and data fitting
         (see `ParameterEstimationComposition`); and
    iv) a graphic interface for the construction of models and realtime display of their execution
        (see `PsyNeuLinkView <http://www.psyneuln.deptcpanel.princeton.edu/psyneulink-view-2/>`_).

Environment Overview
--------------------

PsyNeuLink is written in Python, and conforms to the syntax, coding standards and modular organization shared by
most Python packages.  BasicsAndPrimer provides an orientation to PsyNeuLinks Components, some examples of what
PsyNeuLink models look like, and some of its capabilities. QuickReference provides an overview of how PsyNeuLink is
organized and some of its basic principles of operation.  The Tutorial provides an interactive guide to the
construction of models using PsyNeuLink. Core contains the fundamental objects used to build PsyNeuLink models, and
Library contains extensions, including speciality components, implemented compositions, and published models.

Installation
------------

PsyNeuLink is compatible with python versions >= 3.5, and is available through `PyPI <https://pypi.python.org/pypi/PsyNeuLink>`__:

::

    pip install psyneulink

All prerequisite packages will be automatically added to your environment.

If you downloaded the source code, navigate to the cloned directory in a terminal,
switch to your preferred python3 environment, then run

::

    pip install .

Dependencies that are automatically installed (except those noted as optional) include:

   * numpy
   * matplotlib
   * toposort
   * typecheck-decorator (version 1.2)
   * pillow
   * llvmlite
   * mpi4py (optional)
   * autograd (optional)

Lists of required packages for PsyNeuLink, developing PsyNeuLink, and running the PsyNeuLink tutorial are also
stored in pip-style `requirements.txt`, `dev_requirements.txt`, and `tutorial_requirements.txt` in the source code.

PsyNeuLink is an open source project maintined on `GitHub <https://github.com>`_. The repo can be cloned
from `here <https://github.com/PrincetonUniversity/PsyNeuLink>`_.

If you have trouble installing the package, or run into other problems, please contact psyneulinkhelp@princeton.edu.

Tutorial
--------

PsyNeuLink includes a tutorial (/tutorial/PsyNeuLink Tutorial.ipynb), that provides examples of how to create basic Components
in PsyNeuLink, and combine them into Processes and a System.  The examples include construction of a simple
decision making process using a Drift Diffusion Model, a neural network model of the Stroop effect, and a
backpropagation network for learning the XOR problem.

The tutorial can be run in a browser by clicking the badge below, or `this link <https://mybinder.org/v2/gh/PrincetonUniversity/PsyNeuLink/master>`__.

.. image:: https://mybinder.org/badge.svg
    :target: https://mybinder.org/v2/gh/PrincetonUniversity/PsyNeuLink/master

To run the tutorial locally, you must run python 3.5 and install additional packages:

::

    pip install psyneulink[tutorial]

or if you downloaded the source:

::

    pip install .[tutorial]

To access the tutorial, make sure you fulfill the requirements mentioned above, download the tutorial notebook (/tutorial/PsyNeuLink Tutorial.ipynb), then run the terminal command

::

    jupyter notebook

Once the notebook opens in your browser, navigate to the location where you saved the tutorial notebook, and
click on "PsyNeuLink Tutorial.ipynb".

Help and Issue Reporting
------------------------

Help is available at psyneulinkhelp@princeton.edu.

Issues can be reported at https://github.com/PrincetonUniversity/PsyNeuLink/issues.

Contributors
------------

*(in alphabetical order)*

* **Allie Burton**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Laura Bustamante**, Princeton Neuroscience Institute, Princeton University
* **Jonathan D. Cohen**, Princeton Neuroscience Institute, Princeton University
* **Samyak Gupta**, Department of Computer Science, Princeton University
* **Abigail Hoskin**, Department of Psychology, Princeton University
* **Peter Johnson**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Justin Junge**, Department of Psychology, Princeton University
* **Qihong Lu**, Department of Psychology, Princeton University
* **Kristen Manning**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Katherine Mantel**, Princeton Neuroscience Institute, Princeton University
* **Lena Rosendahl**, Department of Mechanical and Aerospace Engineering, Princeton University
* **Dillon Smith**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Markus Spitzer**, Princeton Neuroscience Institute, Princeton University (formerly)
* **David Turner**, Princeton Neuroscience Institute, Princeton University
* **Jan Vesely**, Department of Computer Science, Rutgers University (formerly)
* **Changyan Wang**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Nate Wilson**, Princeton Neuroscience Institute, Princeton University (formerly)

With substantial and greatly appreciated assistance from:

* **Abhishek Bhattacharjee**, Department of Computer Science, Yale University
* **Mihai Capota**, Intel Labs, Intel Corporation
* **Bryn Keller**, Intel Labs, Intel Corporation
* **Susan Liu**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Garrett McGrath**, Princeton Neuroscience Institute, Princeton University
* **Sebastian Musslick**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Amitai Shenhav**, Cognitive, Linguistic, & Psychological Sciences, Brown University
* **Michael Shvartsman**, Princeton Neuroscience Institute, Princeton University (formerly)
* **Ben Singer**, Princeton Neuroscience Institute, Princeton University
* **Ted Willke**, Brain Inspired Computing Lab, Intel Corporation

Support
-------

The development of PsyNeuLink has benefited by generous support from the following agencies:

* `The National Institute of Mental Health (R21-MH117548) <https://www.nimh.nih.gov/about/organization/dtr/adult-psychopathology-and-psychosocial-interventions-research-branch/computational-psychiatry-program.shtml>`_
* `The John Templeton Foundation <https://www.templeton.org/funding-areas/science-big-questions>`_
* `The Templeton World Charitable Foundation <https://www.templetonworldcharity.org/our-priorities/diverse-intelligences>`_

License
-------

::

    Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.  You may obtain a copy of the License at:
         http://www.apache.org/licenses/LICENSE-2.0
    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
    on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and limitations under the License.


[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/hdfgroup/hsds)

# HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data

## Introduction

HSDS is a web service that implements a REST-based web service for HDF5 data stores.
Data can be stored in either a POSIX files system, or using object-based storage such as
AWS S3, Azure Blob Storage, or [MinIO](https://min.io).
HSDS can be run a single machine with or without Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure).

## Quick Start

### With Github codespaces

Launch a Codespaces environment by clicking the banner __["Open in GitHub Codespaces"](https://codespaces.new/HDFGroup/hsds)__.  Once the codespace is ready, type:
`python testall.py` in the terminal window to run the test suite.

### On your desktop/laptop

Make sure you have Python 3 and Pip installed, then:

1.  Run install: `$ ./build.sh --no-lint --no-docker` from source tree OR install from pypi: `$ pip install hsds`
2.  Create a directory the server will use to store data, example: `$ mkdir ~/hsds_data`
3.  Start server: `$ hsds --root_dir ~/hsds_data`
4.  Run the test suite. In a separate terminal run:
    -  Set user_name: `$ export USER_NAME=$USER`
    -  Set user_password: `$ export USER_PASSWORD=$USER`
    -  Set admin name: `$ export ADMIN_USERNAME=$USER`
    -  Set admin password: `$ export ADMIN_PASSWORD=$USER`
    -  Run test suite: `$ python testall.py --skip_unit`
5. (Optional) Install the h5pyd package for an h5py compatible api and tool suite: https://github.com/HDFGroup/h5pyd
6. (Optional) Post install setup (test data, home folders, cli tools, etc): [docs/post_install.md](docs/post_install.md)

To shut down the server, and the server is not running in Docker, just control-C.

If using docker, run: `$ ./stopall.sh`

Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image. Alternatively, an external identity provider such as Azure Active Directory or KeyCloak can be used. See: [docs/azure_ad_setup.md](docs/azure_ad_setup.md) for Azure AD setup instructions or [docs/keycloak_setup.md](docs/keycloak_setup.md) for KeyCloak.

## Detailed Install Instructions

### On AWS

For complete instructions to install on a single Azure VM with Docker:

- See: [docs/docker_install_aws.md](docs/docker_install_aws.md)

For complete instructions to install on AWS Kubernetes Service (EKS):

- See: [docs/kubernetes_install_aws.md](docs/kubernetes_install_aws.md)

For complete instructions to install on AWS Lambda:

- See: [docs/aws_lambda_setup.md](docs/aws_lambda_setup.md).

### On Azure

For complete instructions to install on a single Azure VM with Docker:

- See: [docs/docker_install_azure.md](docs/docker_install_azure.md)

For complete instructions to install on Azure Kubernetes Service (AKS):

- See: [docs/kubernetes_install_azure.md](docs/kubernetes_install_azure.md)

### On Prem (POSIX-based storage)

For complete instructions to install on a desktop or local server:

- See: [docs/docker_install_posix.md](docs/docker_install_posix.md)

### On DCOS (BETA)

For complete instructions to install on DCOS:

- See: [docs/docker_install_dcos.md](docs/docker_install_dcos.md)

## General Install Topics

Setting up docker:

- See [docs/setup_docker.md](docs/setup_docker.md)

Post install setup and testing:

- See [docs/post_install.md](docs/post_install.md)

Authorization, ACLs, and Role Based Access Control (RBAC):

- See [docs/authorization.md](docs/authorization.md)

Running serverless with h5pyd:

- See <https://github.com/HDFGroup/h5pyd/blob/master/README.rst>

## Writing Client Applications

As a REST service, clients be developed using almost any programming language. The
test programs under: hsds/test/integ illustrate some of the methods for performing
different operations using Python and HSDS REST API (using the requests package).

The related project: <https://github.com/HDFGroup/h5pyd> provides a (mostly) h5py-compatible
interface to the server for Python clients.

For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data
using HSDS. See: <https://github.com/HDFGroup/vol-rest>. Note: requires v1.12.0 or greater version of the HDF5 library.

## Uninstalling

HSDS only modifies the storage location that it is configured to use, so to uninstall just remove
source files, Docker images, and S3 bucket/Azure Container/directory files.

## Reporting bugs (and general feedback)

Create new issues at <http://github.com/HDFGroup/hsds/issues> for any problems you find.

For general questions/feedback, please use the HSDS forum: <https://forum.hdfgroup.org/c/hsds>.

## License

HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.

## Azure Marketplace

VM Offer for Azure Marketplace. HSDS for Azure Marketplace provides an easy way to
setup a Azure instance with HSDS. See: <https://azuremarketplace.microsoft.com/en-us/marketplace/apps/thehdfgroup1616725197741.hsdsazurevm?tab=Overview> for more information.

## Websites

- Main website: <https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/>
- Source code: <https://github.com/HDFGroup/hsds>
- Forum: <https://forum.hdfgroup.org/c/hsds>
- Documentation: <http://h5serv.readthedocs.org> (For REST API)

## Other useful resources

### HDF Group Blog Posts

- Web Caching: <https://www.hdfgroup.org/2022/10/improve-hdf5-performance-using-caching/>
- HSDS Streaming: <https://www.hdfgroup.org/2022/08/hsds-streaming/>
- Cloud Storage Options for HDF5: <https://www.hdfgroup.org/2022/08/cloud-storage-options-for-hdf5/>
- HSDS Docker Images: <https://www.hdfgroup.org/2022/07/hsds-docker-images/>
- HSDS Container Types: <https://www.hdfgroup.org/2022/07/deep-dive-hsds-container-types/>
- Using Multiprocessing in Python: <https://www.hdfgroup.org/2022/06/speed-up-cloud-access-using-multiprocessing/>
- Biosimulations - case study with HSDS and Vega: <https://www.hdfgroup.org/2022/02/biosimulations-a-platform-for-sharing-and-reusing-biological-simulations/>
- HSDS for Microsoft Azure: <https://www.hdfgroup.org/2021/08/hsds-for-azure/>
- New Features in HSDS v0.6: <https://www.hdfgroup.org/2020/10/new-features-in-hsds-version-0-6/>
- HSDS Security: <https://hdfgroup.org/wp/2015/12/serve-protect-web-security-hdf5>
- HDF for the Web: HDF Server: <https://www.hdfgroup.org/2015/04/hdf5-for-the-web-hdf-server/>

### External Blogs and Articles

- A RESTful Meeting Between MATLAB and HDF Server: <https://www.mathworks.com/matlabcentral/fileexchange/59072-a-restful-meeting-between-matlab-and-hdf-server-web-based-hdf5-access-using-matlab>
- AWS Big Data Blog: <https://aws.amazon.com/blogs/big-data/power-from-wind-open-data-on-aws/>

### Slide Decks

- HSDS v0.7 New Features, EUHUG 2022: <https://www.hdfgroup.org/wp-content/uploads/2022/05/HSDS_New_Feautres_7.0.pdf>
- HSDS Serverless, EUHUG 2021: <https://www.hdfgroup.org/wp-content/uploads/2021/07/ServerlessHSDS.pdf>
- HSDS REST, HUG 2020: <https://www.hdfgroup.org/wp-content/uploads/2020/10/HSDS_Rest_Service_HDF5_Readey.pdf>
- HSDS with Jupyter, ESIP 2018: <https://www.slideshare.net/HDFEOS/hdf-kita-lab-jupyterlab-hdf-service>
- HDF Data Services, SciPy17: <http://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf>

### Videos

- HSDS Webinar: <https://www.youtube.com/watch?v=9b5TO7drqqE>
- HSDS Overview, Allotrope Connect Day: <https://www.youtube.com/watch?v=nRHXEkhlfZ0>
- The Use of HSDS on SlideRule, HUG 2020: <https://www.youtube.com/watch?v=i-KIoGqdEMg>
- HDF Data Services, SciPy 2017: <https://www.youtube.com/watch?v=EmnCz1Hg-VM>
- RESTful HDF, SciPy 2015: <https://www.youtube.com/watch?v=JSFZ3i3WcjQ>

### Papers

- restfulSE: A semantically rich interface for cloud-scale genomics with Bioconductor: <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6392152>
- RESTful HDF5 White Paper: <https://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf>


| CI           | [![GitHub Workflow Status][github-ci-badge]][github-ci-link] [![GitHub Workflow Status][github-upstream-ci-badge]][github-upstream-ci-link] [![Code Coverage Status][codecov-badge]][codecov-link] [![asv-badge]][asv-link] |
| :----------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| **Docs**     |                                                                    [![Documentation Status][rtd-badge]][rtd-link]                                                                    |
| **Package**  |                            [![Conda][conda-badge]][conda-link] [![PyPI][pypi-badge]][pypi-link] [![Python Version][python-version-badge]][python-version-badge-link]                 |
| **License**  |                                                                        [![License][license-badge]][repo-link]                                                                        |
| **Citing**  |                                                                              [![DOI][doi-badge]][doi-link]                                                                            |

GeoCAT-comp is the computational component of the
[GeoCAT](https://geocat.ucar.edu/) project. GeoCAT-comp provides computational
functions for operating on geosciences data. Many of these functions originated
in NCL and were translated into Python in GeoCAT-comp; however, developers are
welcome to suggest novel computational functions for geosciences data.

# Documentation

[GeoCAT Homepage](https://geocat.ucar.edu/)

[GeoCAT Contributor's Guide](https://github.com/NCAR/geocat-comp/blob/main/CONTRIBUTING.md)

[GeoCAT-comp documentation on Read the Docs](https://geocat-comp.readthedocs.io)

# Installation and build instructions

Please see our documentation for
[installation and build instructions](https://github.com/NCAR/geocat-comp/blob/main/INSTALLATION.md).

# Citing GeoCAT-comp

If you use this software, please cite it as described at the [GeoCAT-comp - Citation](
https://geocat-comp.readthedocs.io/en/latest/citation.html) page.

[github-ci-badge]: https://img.shields.io/github/actions/workflow/status/NCAR/geocat-comp/ci.yml?branch=main&label=CI&style=for-the-badge
[github-ci-link]: https://github.com/NCAR/geocat-comp/actions/workflows/ci.yml
[github-upstream-ci-badge]: https://img.shields.io/github/actions/workflow/status/NCAR/geocat-comp/upstream-dev-ci.yml?branch=main&label=Upstream%20CI&style=for-the-badge
[github-upstream-ci-link]: https://github.com/NCAR/geocat-comp/actions/workflows/upstream-dev-ci.yml
[codecov-badge]: https://img.shields.io/codecov/c/github/NCAR/geocat-comp.svg?logo=codecov&style=for-the-badge&color=brightgreen
[codecov-link]: https://codecov.io/gh/NCAR/geocat-comp/coverage.yml
[asv-badge]: https://img.shields.io/badge/benchmarked%20by-asv-green.svg?style=for-the-badge
[asv-link]: https://ncar.github.io/geocat-comp-asv/
[rtd-badge]: https://img.shields.io/readthedocs/geocat-comp/latest.svg?style=for-the-badge
[rtd-link]: https://geocat-comp.readthedocs.io/en/latest/?badge=latest
[pypi-badge]: https://img.shields.io/pypi/v/geocat-comp?logo=pypi&style=for-the-badge
[pypi-link]: https://pypi.org/project/geocat-comp
[python-version-badge]: https://img.shields.io/pypi/pyversions/geocat-comp?style=for-the-badge
[python-version-badge-link]: https://img.shields.io/pypi/pyversions/geocat-comp
[conda-badge]: https://img.shields.io/conda/vn/conda-forge/geocat-comp?logo=anaconda&style=for-the-badge
[conda-link]: https://anaconda.org/conda-forge/geocat-comp
[license-badge]: https://img.shields.io/github/license/NCAR/geocat-comp?style=for-the-badge
[comment]: <> ([doi-badge]: https://img.shields.io/badge/DOI-10.5065%2Fa8pp--4358-brightgreen?style=for-the-badge)
[comment]: <> ([doi-link]: https://doi.org/10.5065/a8pp-4358)
[doi-badge]: https://zenodo.org/badge/DOI/10.5281/zenodo.6607205.svg
[doi-link]: https://doi.org/10.5281/zenodo.6607205
[repo-link]: https://github.com/NCAR/geocat-comp


# SynthTool (for client libraries)

Synthtool is a library of common templates and code that makes generating
source code easier.

See the inner [README](./synthtool/README.md) for more details.


# Kolibri Studio

[![Python tests](https://github.com/learningequality/studio/actions/workflows/pythontest.yml/badge.svg)](https://github.com/learningequality/studio/actions/workflows/pythontest.yml) [![Javascript Tests](https://github.com/learningequality/studio/actions/workflows/frontendtest.yml/badge.svg)](https://github.com/learningequality/studio/actions/workflows/frontendtest.yml)

[Kolibri Studio](https://studio.learningequality.org) is a web application designed to deliver educational materials to [Kolibri](http://learningequality.org/kolibri/). It supports:

- Organizing and publishing content channels in the format suitable for import from Kolibri
- Curating content and remixing of existing channels into custom channels aligned to various educational standards, country curricula, and special needs
- Creating learning pathways and assessments
- Uploading new content through the web interface or programatically using [ricecooker-powered](https://github.com/learningequality/ricecooker) content import scripts

Kolibri Studio uses the [Django framework](https://www.djangoproject.com/) for the backend and [Vue.js](https://vuejs.org/) for the frontend.

If you are looking for help setting up custom content channels, uploading and organizing resources using Kolibri Studio, please refer to the [User Guide](https://kolibri-studio.readthedocs.io/en/latest/).

<!-- Also update CONTRIBUTING.md (duplicate) -->
## How can I contribute?

1. 📙 **Skim through the [Developer documentation](./docs/_index.md)** to understand where to refer later on.
2. 💻 **Follow the [Local development instructions](./docs/local_dev_docker.md) to set up your development server.**
3. 🔍 **Search for issues tagged as [help wanted](https://github.com/learningequality/studio/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22+no%3Aassignee) or [good first issue](https://github.com/learningequality/studio/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+no%3Aassignee).**
4. 🗣 **Ask us for an assignment in the comments of an issue you've chosen.** Please request assignment of a reasonable amount of issues at a time. Once you finish your current issue or two, you are welcome to ask for more.

**❓ Where to ask questions**

- For anything development related, refer to the [Developer documentation](./docs/_index.md) at first. Some answers may already be there.
- For questions related to a specific issue or assignment requests, use the corresponding issue's comments section.
- Visit [GitHub Discussions](https://github.com/learningequality/studio/discussions) to ask about anything related to contributing or to troubleshoot development server issues.

**👥 How to connect**

- We encourage you to visit [GitHub Discussions](https://github.com/learningequality/studio/discussions) to connect with the Learning Equality team as well as with other contributors.
- If you'd like to contribute on a regular basis, we are happy to invite you to our open-source community Slack channel. Get in touch with us at info@learningequality.org to receive an invitation.

---

🕖 Please allow us a few days to reply to your comments. If you don't hear from us within a week, reach out via [GitHub Discussions](https://github.com/learningequality/studio/discussions).

As soon as you open a pull request, it may take us a week or two to review it as we're a small team. We appreciate your contribution and will provide feedback.

---

*Thank you for your interest in contributing! Learning Equality was founded by volunteers dedicated to helping make educational materials more accessible to those in need, and every contribution makes a difference.*


# TileCloud Chain

The goal of TileCloud Chain is to provide tools around tile generation on a chain like:

Source: WMS, Mapnik.

Optionally using an SQS queue, AWS host, SNS topic.

Destination in WMTS layout, on S3, on Berkeley DB (`bsddb`), on MBTiles, or on local filesystem.

Features:

- Generate tiles.
- Drop empty tiles.
- Drop tiles outside a geometry or a bbox.
- Use MetaTiles.
- Generate the legend images.
- Generate GetCapabilities.
- Generate OpenLayers example page.
- Obtain the hash of an empty tile.
- In the future, measure tile generation speed.
- Calculate cost and generation time.
- In the future, manage the AWS hosts that generate tiles.
- Delete empty tiles.
- Copy files between caches.
- Be able to use an SQS queue to dispatch the generation.
- Post processing the generated tiles.
- ...

Legacy features:

- bsddb support
- sqlite (mbtiles) support
- mapnik support (should be updated for Python3)

## Screenshot

Screenshot of the admin page with queue stored on PostgreSQL:

![TileCloud Chain](./admin-screenshot.png)

## Get it

Create the config file `tilegeneration/config.yaml` see as [example](https://github.com/camptocamp/tilecloud-chain/blob/master/example/tilegeneration/config.yaml).

### Support

Only the latest release is supported and version < 1.11 contains security issues.

## From sources

Build it:

```bash
git submodule update --recursive
python3 -m venv .build/venv
.build/venv/bin/pip install -r requirements.txt
.build/venv/bin/pip install -e .
.build/venv/bin/pip install -r dev-requirements.txt
```

## Run prospector

```bash
.build/venv/bin/prospector
```

## Run the tests

Setup your environment:

```bash
touch tilecloud_chain/OpenLayers.js
docker build --tag camptocamp/tilecloud-chain .
docker-compose -p tilecloud up
```

To run the tests:

```bash
docker-compose -p tilecloud exec test python setup.py nosetests --logging-filter=tilecloud,tilecloud_chain --attr '!'nopy3
```

## Documentation

As documentation you can read the [USAGE.rst](https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/USAGE.rst)
and the [configuration reference](https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/CONFIG.md).

## VSCode

You can add that in your workspace configuration to use the JSON schema:

```json
{
  "yaml.schemas": {
    "../tilecloud-chain/tilecloud_chain/schema.json": [
      "tilecloud-chain/tilecloud_chain/tests/tilegeneration/*.yaml"
    ]
  }
}
```

## Contributing

Install the pre-commit hooks:

```bash
pip install pre-commit
pre-commit install --allow-missing-config
```


# Intake-esm

- [Intake-esm](#intake-esm)
  - [Badges](#badges)
  - [Motivation](#motivation)
  - [Overview](#overview)
  - [Installation](#installation)

## Badges

| CI           | [![GitHub Workflow Status][github-ci-badge]][github-ci-link] [![Code Coverage Status][codecov-badge]][codecov-link] [![pre-commit.ci status][pre-commit.ci-badge]][pre-commit.ci-link] |
| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| **Docs**     |                                                                     [![Documentation Status][rtd-badge]][rtd-link]                                                                     |
| **Package**  |                                                          [![Conda][conda-badge]][conda-link] [![PyPI][pypi-badge]][pypi-link]                                                          |
| **License**  |                                                                         [![License][license-badge]][repo-link]                                                                         |
| **Citation** |                                                                         [![Zenodo][zenodo-badge]][zenodo-link]                                                                         |

## Motivation

Computer simulations of the Earth’s climate and weather generate huge amounts of data.
These data are often persisted on HPC systems or in the cloud across multiple data
assets of a variety of formats ([netCDF](https://www.unidata.ucar.edu/software/netcdf/), [zarr](https://zarr.readthedocs.io/en/stable/), etc...). Finding, investigating,
loading these data assets into compute-ready data containers costs time and effort.
The data user needs to know what data sets are available, the attributes describing
each data set, before loading a specific data set and analyzing it.

Finding, investigating, loading these assets into data array containers
such as xarray can be a daunting task due to the large number of files
a user may be interested in. Intake-esm aims to address these issues by
providing necessary functionality for searching, discovering, data access/loading.

## Overview

`intake-esm` is a data cataloging utility built on top of [intake](https://github.com/intake/intake), [pandas](https://pandas.pydata.org/), and [xarray](https://xarray.pydata.org/en/stable/), and it's pretty awesome!

- Opening an ESM catalog definition file: An Earth System Model (ESM) catalog file is a JSON file that conforms
  to the [ESM Collection Specification](./docs/source/reference/esm-catalog-spec.md). When provided a link/path to an esm catalog file, `intake-esm` establishes
  a link to a database (CSV file) that contains data assets locations and associated metadata
  (i.e., which experiment, model, the come from). The catalog JSON file can be stored on a local filesystem
  or can be hosted on a remote server.

  ```python

  In [1]: import intake

  In [2]: import intake_esm

  In [3]: cat_url = intake_esm.tutorial.get_url("google_cmip6")

  In [4]: cat = intake.open_esm_datastore(cat_url)

  In [5]: cat
  Out[5]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s>
  ```

- Search and Discovery: `intake-esm` provides functionality to execute queries against the catalog:

  ```python
  In [5]: cat_subset = cat.search(
     ...:     experiment_id=["historical", "ssp585"],
     ...:     table_id="Oyr",
     ...:     variable_id="o2",
     ...:     grid_label="gn",
     ...: )

  In [6]: cat_subset
  Out[6]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s)>
  ```

- Access: when the user is satisfied with the results of their query, they can load data assets (netCDF and/or Zarr stores) into xarray datasets:

  ```python

    In [7]: dset_dict = cat_subset.to_dataset_dict()

    --> The keys in the returned dictionary of datasets are constructed as follows:
            'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
    |███████████████████████████████████████████████████████████████| 100.00% [2/2 00:18<00:00]
  ```

See [documentation](https://intake-esm.readthedocs.io/en/latest/) for more information.

## Installation

Intake-esm can be installed from PyPI with pip:

```bash
python -m pip install intake-esm
```

It is also available from `conda-forge` for conda installations:

```bash
conda install -c conda-forge intake-esm
```

[github-ci-badge]: https://github.com/intake/intake-esm/actions/workflows/ci.yaml/badge.svg
[github-ci-link]: https://github.com/intake/intake-esm/actions/workflows/ci.yaml
[codecov-badge]: https://img.shields.io/codecov/c/github/intake/intake-esm.svg?logo=codecov
[codecov-link]: https://codecov.io/gh/intake/intake-esm
[rtd-badge]: https://readthedocs.org/projects/intake-esm/badge/?version=latest
[rtd-link]: https://intake-esm.readthedocs.io/en/latest/?badge=latest
[pypi-badge]: https://img.shields.io/pypi/v/intake-esm?logo=pypi
[pypi-link]: https://pypi.org/project/intake-esm
[conda-badge]: https://img.shields.io/conda/vn/conda-forge/intake-esm?logo=anaconda
[conda-link]: https://anaconda.org/conda-forge/intake-esm
[zenodo-badge]: https://img.shields.io/badge/DOI-10.5281%20%2F%20zenodo.3491062-blue.svg
[zenodo-link]: https://doi.org/10.5281/zenodo.3491062
[license-badge]: https://img.shields.io/github/license/intake/intake-esm
[repo-link]: https://github.com/intake/intake-esm
[pre-commit.ci-badge]: https://results.pre-commit.ci/badge/github/intake/intake-esm/main.svg
[pre-commit.ci-link]: https://results.pre-commit.ci/latest/github/intake/intake-esm/main


# Knowledge Graph Exchange

[![Python](https://img.shields.io/badge/python-3.9+-blue.svg)]()
![Run tests](https://github.com/biolink/kgx/workflows/Run%20tests/badge.svg)[![Documentation Status](https://readthedocs.org/projects/kgx/badge/?version=latest)](https://kgx.readthedocs.io/en/latest/?badge=latest)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=biolink_kgx&metric=alert_status)](https://sonarcloud.io/dashboard?id=biolink_kgx)
[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=biolink_kgx&metric=sqale_rating)](https://sonarcloud.io/dashboard?id=biolink_kgx)
[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=biolink_kgx&metric=coverage)](https://sonarcloud.io/dashboard?id=biolink_kgx)
[![PyPI](https://img.shields.io/pypi/v/kgx)](https://img.shields.io/pypi/v/kgx)
[![Docker](https://img.shields.io/static/v1?label=Docker&message=biolink/kgx:latest&color=orange&logo=docker)](https://hub.docker.com/r/biolink/kgx)

KGX (Knowledge Graph Exchange) is a Python library and set of command line utilities for exchanging
Knowledge Graphs (KGs) that conform to or are aligned to the [Biolink Model](https://biolink.github.io/biolink-model/).

The core datamodel is a [Property Graph](https://neo4j.com/developer/graph-database/) (PG), represented
internally in Python using a [networkx MultiDiGraph model](https://networkx.github.io/documentation/stable/reference/classes/generated/networkx.MultiDiGraph.edges.html).

KGX allows conversion to and from:

 * RDF serializations (read/write) and SPARQL endpoints (read)
 * Neo4j endpoints (read) or Neo4j dumps (write)
 * CSV/TSV and JSON (see [associated data formats](./data-preparation.md) and [example script to load CSV/TSV to Neo4j](./examples/scripts/load_csv_to_neo4j.py))
 * Reasoner Standard API format
 * OBOGraph JSON format

KGX will also provide validation, to ensure the KGs are conformant to the Biolink Model: making sure nodes are
categorized using Biolink classes, edges are labeled using valid Biolink relationship types, and valid properties are used.

Internal representation is a property graph, specifically a networkx MultiDiGraph.

The structure of this graph is expected to conform to the Biolink Model standard, as specified in the [KGX format specification](specification/kgx-format.md).

In addition to the main code-base, KGX also provides a series of [command line operations](https://kgx.readthedocs.io/en/latest/examples.html#using-kgx-cli).

### Example usage
Validate:
```bash
poetry run kgx validate -i tsv tests/resources/merge/test2_nodes.tsv tests/resources/merge/test2_edges.tsv
```

Merge:
```bash
poetry run kgx merge —merge-config tests/resources/test-merge.yaml
```

Graph Summary:
```bash
poetry run kgx graph-summary -i tests/resources/graph_nodes.tsv  -o summary.txt
```

Transform:
```bash
poetry run kgx transform —transform-config tests/resources/test-transform-tsv-rdf.yaml
```

### Error Detection and Reporting

Non-redundant JSON-formatted structured error logging is now provided in KGX Transformer, Validator, GraphSummary and MetaKnowledgeGraph operations.  See the various unit tests for the general design pattern (using the Validator as an example here):

```python
from kgx.validator import Validator
from kgx.transformer import Transformer

Validator.set_biolink_model("2.11.0")

# Validator assumes the currently set Biolink Release
validator = Validator()

transformer = Transformer(stream=True)

transformer.transform(
    input_args = {
        "filename": [
            "graph_nodes.tsv",
            "graph_edges.tsv",
        ],
        "format": "tsv",
    },
    output_args={
        "format": "null"
    },
    inspector=validator,
)

# Both the Validator and the Transformer can independently capture errors

# The Validator, from the overall semantics of the graph...
# Here, we just report severe Errors from the Validator (no Warnings)
validator.write_report(open("validation_errors.json", "w"), "Error")

# The Transformer, from the syntax of the input files...
# Here, we catch *all* Errors and Warnings (by not providing a filter)
transformer.write_report(open("input_errors.json", "w"))
```

The JSON error outputs will look something like this:

```json
{
    "ERROR": {
        "MISSING_EDGE_PROPERTY": {
            "Required edge property 'id' is missing": [
                "A:123->X:1",
                "B:456->Y:2"
            ],
            "Required edge property 'object' is missing": [
                "A:123->X:1"
            ],
            "Required edge property 'predicate' is missing": [
                "A:123->X:1"
            ],
            "Required edge property 'subject' is missing": [
                "A:123->X:1",
                "B:456->Y:2"
            ]
        }
    },
    "WARNING": {
        "DUPLICATE_NODE": {
          "Node 'id' duplicated in input data": [
            "MONDO:0010011",
            "REACT:R-HSA-5635838"
          ]
        }
    }
}

```

This system reduces the significant redundancies of earlier line-oriented KGX  logging text output files, in that graph entities with the same class of error are simply aggregated in lists of names/identifiers at the leaf level of the JSON structure.

The top level JSON tags originate from the `MessageLevel` class and the second level tags from the `ErrorType` class in the [error_detection](kgx/error_detection.py) module, while the third level messages are hard coded as `log_error` method messages in the code.

It is likely that additional error conditions within KGX can be efficiently captured and reported in the future using this general framework.

## Installation

#### Installing from PyPI

KGX is available on PyPI and can be installed using
[pip](https://pip.pypa.io/en/stable/installing/) as follows,

```bash
pip install kgx
```

To install a particular version of KGX, be sure to specify the version number,

```bash
pip install kgx==0.5.0
```

#### Installing from GitHub

Clone the GitHub repository and then install,

```bash
git clone https://github.com/biolink/kgx
cd kgx
poetry install
```

### Setting up a testing environment for Neo4j

This release of KGX supports graph source and sink transactions with the 4.3 release of Neo4j.

KGX has a suite of tests that rely on Docker containers to run Neo4j specific tests.

To set up the required containers, first install [Docker](https://docs.docker.com/get-docker/)
on your local machine.

Once Docker is up and running, run the following commands:

```bash
docker run -d --rm --name kgx-neo4j-integration-test -p 7474:7474 -p 7687:7687 --env NEO4J_AUTH=neo4j/test neo4j:4.3
```

```bash
docker run -d --rm --name kgx-neo4j-unit-test -p 8484:7474 -p 8888:7687 --env NEO4J_AUTH=neo4j/test neo4j:4.3
```

**Note:** Setting up the Neo4j container is optional. If there is no container set up
then the tests that rely on them are skipped.


[![Build Status](https://travis-ci.org/wevote/WeVoteServer.svg?branch=master)](https://travis-ci.org/wevote/WeVoteServer) [![Coverage Status](https://coveralls.io/repos/wevote/WeVoteServer/badge.svg?branch=master&service=github)](https://coveralls.io/github/wevote/WeVoteServer?branch=master) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

# README for WeVoteServer

This WeVoteServer repository contains a Python/Django-powered API endpoints server. We take in ballot data from
Google Civic API, Ballotpedia, Vote Smart, MapLight, TheUnitedStates.io and the Voting Information Project. We then serve
it up to voters, and let voters Support/Oppose and Like ballot items. We are also building tools to capture
and share voter guide data.

You can see our current alpha version for recent national and some reigonal elections here:  https://WeVote.US/

To get started as a We Vote developer, sign the Contributor License Agreement.

## Installing Python/Django API Server

To install and develop the WeVote API server, follow the instructions below based on your preferred environment.

1. Use these [Simplified Instructions for Mac](docs/README_MAC_SIMPLIFIED_INSTALL.md) leveraging the free (and powerful) PyCharm IDE and debugger (**recommended!**)

2. [Install directly](docs/README_API_INSTALL.md) on your Linux machine (or use WSL on Windows)

3. Use [Docker](docs/README_API_INSTALL_DOCKER.md)

## Installing We Vote Mobile Web Application (React/Flux)

The website front end application is powered by the [We Vote WebApp](https://github.com/wevote/WebApp)

We distribute the We Vote WebApp in an Apache Cordova wrapper, with some native features, to provide [iOS and Android](https://github.com/wevote/WeVoteCordova) apps.

See the iOS [We Vote 2018 Ballot, @WeVote](https://itunes.apple.com/us/app/we-vote-2018-ballot-wevote/id1347335726?mt=8) in the iTunes app store for iPhones and iPads.

See the Android [We Vote 2018 Ballot, @WeVote](https://play.google.com/store/apps/details?id=org.wevote.cordova&hl=en_US) in the Google Play store for most Android phone and tablet devices.

We also have a [ReactNative for iOS and Android](https://github.com/wevote/WeVoteReactNative) that is currently on hold.

## After Installation: Working with WeVoteServer Day-to-Day

[Read about working with WeVoteServer on a daily basis](docs/README_WORKING_WITH_WE_VOTE_SERVER.md)

Need to upgrade Python?

3a. [Install Python/Django on Mac](docs/README_API_INSTALL_PYTHON_MAC.md)

3b. [Install Python/Django on Linux](docs/README_API_INSTALL_PYTHON_LINUX.md)

If you need to test donations and have not updated your openssl and pyopenssl during install and setup, you will need
[to update your local](docs/README_DONATION_SETUP.md).

See [Instructions for Scheduled Tasks](docs/README_SCHEDULED_TASKS.md)

[How to run the WeVoteServer tests](docs/README_DJANGO_TESTS.md)

[How to run the WeVoteServer Locust Load Tests](loadtest/README.md)

## Join Us

We meet weekly on Google hangouts, and on a team Slack. Please contact Dale.McGrew@WeVote.US for more information.


# Aleph Core Channel Node (CCN)

Next generation network of decentralized big data applications. Development follows the [Aleph Whitepaper](https://github.com/moshemalawach/aleph-whitepaper).

## Documentation

Albeit still incomplete as it is a work in progress, documentation
can be found at http://pyaleph.readthedocs.io/ or
built from this repository with `$ python setup.py docs`

## Deployment

We recommend following the
[Installing a Core Channel Node](https://pyaleph.readthedocs.io/en/latest/guides/install.html)
section of the documentation to install a node.

## Development

Do you want to contribute to the development of the CCN?
Here is the procedure to install the development environment.
We recommend using Ubuntu 20.04.

### 1. Install dependencies

```bash
sudo apt install python3 python3-pip python3-venv build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libgmp-dev libsecp256k1-dev
```

### 2. Install Python requirements

Clone the repository and run the following commands from the root directory:

```
python3 -m virtualenv venv
source venv/bin/activate
pip install -e .[testing,docs]
```

You're ready to go!

### Developer setup using Nix

We started to add Nix as an easy way to setup a development environment.
This is still a work in progress and not all dependencies are covered yet.

To use it, you need to [have Nix installed on your system](https://nixos.org/download.html). Then you can run:

```bash
nix-shell
```
This will provide you with a shell with PostgreSQL, Redis, and IPFS running.

## Software used

The Aleph CCN is written in Python and requires Python v3.8+. It will not work with older versions of Python.

It also relies on [IPFS](https://ipfs.io/).

## License

The Aleph CCN is open-source software, released under [The MIT License (MIT)](LICENSE.txt).


# Hail utilities for gnomAD

[![PyPI](https://img.shields.io/pypi/v/gnomad)](https://pypi.org/project/gnomad/)

This repo contains a number of [Hail](https://hail.is/) utility functions and scripts for the [gnomAD project](http://gnomad.broadinstitute.org) and the [Translational Genomics Group](https://the-tgg.org). As we continue to expand the size of our datasets, we are constantly seeking to find ways to reduce the complexity of our workflows and to make these functions more generic. As a result, the interface for many of these functions will change over time as we generalize their implementation for more flexible use within our scripts. We are also continously adapting our code to regular changes in the Hail interface. These repos thus represent only a snapshot of the gnomAD code base and are shared without guarantees or warranties.

We therefore encourage users to browse through the [API reference](https://broadinstitute.github.io/gnomad_methods/api_reference/) to identify modules and functions that will be useful in their own pipelines, and to edit and reconfigure relevant code to suit their particular analysis and QC needs.