## The SAS Testing and Interoperability Repository

This repository contains code and data for testing the compliance
of Spectrum Access System (SAS) software. The SAS is defined by
the FCC in proceeding 12-354 as the system authorizing priority
access and general access to the 3550-3700MHz Citizens Broadband
Radio Service.

This repository contains procedures, documentation, and tests
for such software, and for the devices authorized by it.

To contribute, please first read the CONTRIBUTING file in the
repository for instructions.

### Data

Some of the required data is provided in folder `data/`. Scripts used to
retrieve or generate these data are in src/data/.

USGS NED Terrain and NLCD Land cover data are not provided as part of the
data/ folder, but kept instead in a separate Git repository found at:
 https://github.com/Wireless-Innovation-Forum/Common-Data

The process relies in cloning that repository, unzipping the geo data and
pointing the SAS reference models to point to those data.

See the corresponding [README.md](https://github.com/Wireless-Innovation-Forum/Common-Data/README.md)
for more details on the integration of those geo data for use with SAS.

### Code prerequisites

Note: see last section for an example of full installation.

The scripts in the SAS repository depend on local environment setup to run.
Here are packages and software needed to correctly operate the scripts.

* Python 3.11 (https://www.python.org/downloads/release/python-3117/)

This is the Python runtime which interprets the scripts in the
src/ directory. It may already be running on many platforms, and
if it is not already installed, is widely available. Type the command
python --version to check the Python version installed on your
platform.

**NOTE**: The current code was designed to work with both Python 2.7 and 3.7.
It is recommended to only use Python 3.11 from now on. Currently untested with Python 3.12 and above.

* pip (https://pip.pypa.io/en/stable/installing/)

This is the Python package management system. If you have Python installed on
your system, it may already be available as well. The latest distributions of
Python come with pip pre-bundled so you may not need to retrieve
it separately. To check it's availability on your system run the command
pip --version.

* libgeos (http://trac.osgeo.org/geos/)

This binary library is needed to support Python shapely libraries.
It will likely need to be installed separately on your system. Platform-specific
instructions can be found at the above URL.

* lxml (http://lxml.de/installation.html)

This binary library provides the mechanism to support Python XML libraries.
It will likely need to be installed separately, and you may need to configure
platform-specific C++ building capabilities to complete the installation. It
can typically be installed with pip.

* libgdal (https://pypi.python.org/pypi/GDAL/ and http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries)

This binary library provides geospatial data library methods used to manipulate
geographic datasets. It will likely need to be installed separately for your
platform, and you may need to configure platform-specific C++ build capabilities
to complete the installation. Source-based installation is an option. Once the
libgdal package is installed, you will need to download and install the Python
bindings for the library.

* numpy (http://www.scipy.org/scipylib/download.html)

This library provides a variety of numerics methods and Python bindings to use
them. It will likely need to be installed separately for your platform, and
may be installed as part of the SciPy packages.

* shapely (https://pypi.python.org/pypi/Shapely)

This Python library can be installed with pip. It provides
libraries for processing geometrical objects and shapes in Python.

* pyJWT (https://pypi.python.org/pypi/PyJWT)
* cryptography (https://pypi.python.org/pypi/cryptography)
* pykml (https://pythonhosted.org/pykml/installation.html)

This Python library can be installed with pip. It provides
libraries for handling KML geospatial files in Python.

* pyshp (https://github.com/GeospatialPython/pyshp)

This Python library can be installed with pip. It provides
libraries for handling SHP geospatial files in Python.

* ftputil (http://ftputil.sschwarzer.net/trac/wiki/Documentation)

This Python library can be installed with pip. It provides a
high-level interface for retrieving FTP files with Python.

* jsonschema (https://github.com/Julian/jsonschema)

This Python library can be installed with pip. It provides
JSON schema validation support.

* pyopenssl (https://github.com/pyca/pyopenssl)

This Python library can be installed with pip. It provides
support for the OpenSSL library for Python.

* mock (https://pypi.python.org/pypi/mock)

This Python library can be installed with pip. It provides
support for creating testing mocks.

* functools32 (https://pypi.python.org/pypi/functools32)

This Python library can be installed with pip. It provides
support for advanced functools such as OrderedDict and lru_cache memoization.

* psutil (https://github.com/giampaolo/psutil)

This Python library can be installed with pip. It provides
information on running processes and system (RAM,..).

* portpicker (https://pypi.org/project/portpicker)

This Python library can be installed with pip. It allows to
find unused network ports on a host.

For all these Python packages, a good way to test their installation is to
run the Python interpreter and then issue an import xyz command,
where xyz is the name of the package. Error-free import means
that the package is installed successfully.

* Security certificates, as described in
src/harness/testcases/testdata/certs/README.md

### Example of installation: Python3 installation using venv in Linux

Create a venv
```shell
    python3.11 -m venv ./winnf3
```
Activate a venv
```shell
    source ./winnf3/bin/activate
```
Install packages
```shell
    python -m pip install -r ./requirements.txt
```

### Example of installation: Python3 installation using miniconda in Linux

This example uses the MiniConda environment manager.

Install miniconda from this page: https://docs.anaconda.com/free/miniconda/

#### Option 1: use conda-environment.yml file
```shell
    conda env create -f conda-environment.yml
    conda activate winnf3
```
#### Option 2: use conda commands
Create a conda Python 3.11 environment named `winnf3`:

```shell
    conda create --name winnf3 python=3.11
```

Activate the environment on a command shell:

```shell
    conda activate winnf3
```

Install the required packages.

For the reference models and various libs:

```shell
    conda install numpy
    conda install shapely
    conda install gdal
    conda install lxml
    conda install jsonschema
    conda install matplotlib
    conda install cartopy
    pip3 install pygc
    pip3 install pykml
    pip3 install xlsxwriter
```

Additionally for the test harness:

```shell
    conda install cryptography
    pip3 install jwt
    pip3 install portpicker
    conda install pyopenssl
    conda install pycurl
    pip3 install psutil
```

Additionally for the data scripts:

```shell
    pip3 install ftputil
```


# OSBS

[![unittests status badge]][unittests status link]
[![coveralls status badge]][coveralls status link]
[![lgtm python badge]][lgtm python link]
[![lgtm alerts badge]][lgtm alerts link]
[![linters status badge]][linters status link]

Python module and command line client for OpenShift Build Service.

It is able to query OpenShift v3 for various stuff related to building images.
It can initiate builds, list builds, get info about builds, get build logs and
so on. All of this can be done by the OSBS site administrator from command line
and from python. Regular users submitting builds can interact with this system
using Koji, as described in the guide linked below.

## Getting Started

We have a [development setup guide][] on how to set up a whole build system for
local development.

## Deploying OpenShift Build System

We have an [osbs instance setup guide][] detailing how you can set up your own
instance.

## Contributing

If you would like to help out, that's great! Please read the [contributing
guide][] .

[coveralls status badge]: https://coveralls.io/repos/containerbuildsystem/osbs-client/badge.svg?branch=master
[coveralls status link]: https://coveralls.io/r/containerbuildsystem/osbs-client?branch=master
[lgtm python badge]: https://img.shields.io/lgtm/grade/python/g/containerbuildsystem/osbs-client.svg?logo=lgtm&logoWidth=18
[lgtm python link]: https://lgtm.com/projects/g/containerbuildsystem/osbs-client/context:python
[lgtm alerts badge]: https://img.shields.io/lgtm/alerts/g/containerbuildsystem/osbs-client.svg?logo=lgtm&logoWidth=18
[lgtm alerts link]: https://lgtm.com/projects/g/containerbuildsystem/osbs-client/alerts
[linters status badge]: https://github.com/containerbuildsystem/osbs-client/workflows/Linters/badge.svg?branch=master&event=push
[linters status link]: https://github.com/containerbuildsystem/osbs-client/actions?query=event%3Apush+branch%3Amaster+workflow%3A%22Linters%22
[unittests status badge]: https://github.com/containerbuildsystem/osbs-client/workflows/Unittests/badge.svg?branch=master&event=push
[unittests status link]: https://github.com/containerbuildsystem/osbs-client/actions?query=event%3Apush+branch%3Amaster+workflow%3A%22Unittests%22
[development setup guide]: ./docs/development-setup.md
[osbs instance setup guide]: ./docs/osbs_instance_setup.md
[contributing guide]: ./CONTRIBUTING.md


[![Build Status](https://travis-ci.com/brunorijsman/rift-python.svg?branch=master)](https://travis-ci.com/brunorijsman/rift-python)   [![Code Coverage](https://codecov.io/gh/brunorijsman/rift-python/branch/master/graph/badge.svg)](https://codecov.io/gh/brunorijsman/rift-python)

# Routing In Fat Trees (RIFT)

This repository contains a Python implementation of the Routing In Fat Trees (RIFT) protocol specified in Internet Draft (ID)
[draft-draft-rift-15](https://www.ietf.org/archive/id/draft-ietf-rift-rift-15.pdf)

The code is currently still a work in progress (see Feature List below for the status).

# Documentation

* [Feature List](doc/features.md)

* [Installation Instructions](doc/installation.md)

* [Startup Instructions](doc/startup.md)

* [Command Line Options](doc/command-line-options.md)

* [Command Line Interface (CLI)](doc/command-line-interface.md)

* Feature Guides:

  * [Flooding Reduction Feature Guide](doc/flooding-reduction-feature-guide.md)

  * [Disaggregation Feature Guide](doc/disaggregation-feature-guide.md)

    * [Positive Disaggregation Feature Guide](doc/positive-disaggregation-feature-guide.md)

    * [Negative Disaggregation Feature Guide](doc/negative-disaggregation-feature-guide.md)

  * [Security Feature Guide](doc/security-feature-guide.md)

* [Logging](doc/logging.md)

* [Log Visualization](doc/log-visualization.md)

* [Configuration File](doc/configuration-file.md) (Also known as the Topology File)

* [Configuration File Generator](doc/configuration-file-generator.md) (Also known as the Meta-Topology File)

* [Implementation Deviations from Specification](doc/deviations.md)

* [Automated Testing](doc/automated-testing.md)

* [Continuous Integration](doc/continuous-integration.md)

* [Docker](doc/docker.md)

# Presentations and Reports

## IETF 102 (Montreal)

A version of this code (tag ietf-102-rift-hackathon) was used to conduct interoperability testing in the hackathon at the Internet Engineering Task Force (IETF) meeting number 102 in Montreal (July 2018).

* [IETF 102 RIFT Hackathon: Presentation Slides (Microsoft PowerPoint)](ietf-102/ietf-102-rift-hackathon-slides.pptx)

* [IETF 102 RIFT Hackathon: Report](ietf-102/ietf-102-rift-hackathon-detailed-report.md)

## Interim Zero Touch Provisioning (ZTP) Interoperability Testing

* [Zero Touch Provisioning (ZTP) Preliminary Interoperability Testing Report](doc/ztp-interop-testing-report.md)

## RIFT Security Reviews

* [RIFT Security Review version 1 (out of date)](http://bit.ly/rift-security-review)

* [RIFT Security Review version 2](http://bit.ly/rift-security-review-v2)

## IETF 103 (Bangkok)

* [IETF 103 Presentation: Open Source Update (PDF)](ietf-103/ietf-103---rift-wg---open-source-update.pdf)

* [IETF 103 Background Information: Flooding Oscillation (PDF)](ietf-103/ietf-103---flooding-oscillations.pdf)

* [Video of IETF 103 Presentation "RIFT Open Source Implementation: Status Update, Lessons Learned, and Interop Testing"](https://youtu.be/YbBlhd-ePWs?t=1897)

## IETF 104 (Prague)

* [IETF 104 Working Group Presentation: Open Source Update (PDF)](ietf-104/ietf-104---rift-wg---open-source-update.pdf)

* [IETF 104 Working Group Presentation: Open Source Update (YouTube)](https://youtu.be/0OkDLqH4wck?t=3273)

* [IETF 104 Hackathon Presentation: Chaos Monkey Testing (PDF)](ietf-104/ietf-104---rift-hackathon---chaos-monkey-testing.pdf)

* [IETF 104 Hackathon Presentation: Chaos Monkey Testing (YouTube)](https://youtu.be/0OkDLqH4wck?t=3788)

* [IETF 104 Hackathon Instructions Document (PDF)](http://bit.ly/rift-hackathon-ietf-104)

* [IETF 104 Hackathon Instructions Video (YouTube)](https://www.youtube.com/watch?v=GqebgPmA4Xc)

## IETF 105 (Montreal)

* [IETF 105 Working Group Presentation: Open Source Update (PDF)](ietf-105/ietf-105---rift-wg---open-source-update.pdf)

## IETF 108 (Virtual)

* [IETF 108 Working Group Presentation: Open Source Update (PDF)](ietf-108/ietf-108---rift-wg---open-source-update.pdf)

* [IETF 108 Working Group Presentation: Open Source Update (Video)](https://www.youtube.com/watch?v=qhiiTDPuku0)

* [RIFT automatic disaggregation blog](https://hikingandcoding.com/2020/07/22/rift-disaggregation/)

## Blogs

* [Blog post on automatic disaggregation in RIFT](https://hikingandcoding.wordpress.com/2020/07/22/rift-disaggregation/)


# Production release

## Pre-production release

* Update the description of the tracking pull request
* Verify the changes made during the previous post production release

## Production release

* Manually trigger the pipeline tfrs-release.yaml

## Post production release

* Merge the tracking pull request to master
* Create the release tag from master amd make it as the lasted release (this is done automatically by pipeline create-release.yaml)
* Create the new release branch from master
* Update tfrs-release.yaml
  * name
  * PR_NUMBER
  * RELEASE_NAME
* Update .pipeline/lib/config.js
  * const version
  * releaseBranch
* Update frontend/package.json
  * version
* update dev-release.yaml
  * name
  * branches
  * PR_NUMBER
  * RELEASE_NAME
* Commit all the above changes and create the tracking pull request to merge the new release branch to master. Need to update the PR_NUMBER after the tracking pull request is created.

# TFRS Pipelines

## Primary Pipelines

* dev-release.yaml (TFRS Dev release-2.10.0): the pipeline is automatically triggered when there is a commit to the release branch
* tfrs-release.yaml (TFRS release-2.10.0): the pipeline builds the release and deploys on Test and Prod, it needs to be manually triggered
* create-release.yaml (Create Release after merging to master): tag and create the release after merging release branch to master. The description of the tracking pull request becomes release notes

* dev-jan-release.yaml (TFRS Dev Jan Release): the pipeline build Jan 2024 release and deploy on dev for every commit
* dev-release.yaml (TFRS Dev release-2.9.0): the pipeline is automatically triggered when there is a commit to the release branch
* tfrs-release.yaml (TFRS release-2.9.0): the pipelin builds the release and deploy on Test and Prod, it needs to be manually triggered

## Other Pipelines

* branch-deploy-template.yaml (Branch Deploy Template): a pipeline template to deploy a branch
* build-template.yaml (Build Template): a pipeline template to build branch or pull request
* cleanup-cron-workflow-runs.yaml (Scheduled cleanup old workflow runs): a cron job to cleanup the old workflows
* cleanup-workflow-runs.yaml  (Cleanup old workflow runs): manually cleanup teh workflow runs
* pr-dev-cicd.yaml (TFRS Dev Jan PR CICD): the pipeline builds Jan 2024 pull requests and deploy on dev if the pull request title ends with build-on-dev
* pr-dev-database-template.yaml (PR Dev Database Template): the template to create database for pull request build
* pr-deploy-template (PR Dev Deploy Template): the template deploys pull request build to dev
* pr-teardown.yaml (TFRS Dev Jan PR Teardown): tear down the Jan 2024 pull request builds from dev


An essential toolset that eases WordPress site and server administration

  Key Features •
  Usage •
  RoadMap •
  Changelog •
  Credits •
  License

 WordOps.net •
Documentation •
Community Forum •
Dashboard demo

---

## Key Features

-   **Easy to install** : One step automated installer with migration from EasyEngine v3 support
-   **Fast deployment** : Fast and automated WordPress, Nginx, PHP, MySQL & Redis installation
-   **Custom Nginx build** : Nginx 1.26.1 - TLS v1.3 HTTP/3 QUIC & Brotli support
-   **Up-to-date** : PHP 7.4, 8.0, 8.1, 8.2 & 8.3 - MariaDB 11.4 LTS & Redis 7.0
-   **Secured** : Hardened WordPress security with strict Nginx location directives
-   **Powerful** : Optimized Nginx configurations with multiple cache backends support
-   **SSL** : Domain, Subdomain & Wildcard Let's Encrypt SSL certificates with DNS API support
-   **Modern** : Strong ciphers_suite, modern TLS protocols and HSTS support (Grade A+ on [ssllabs](https://www.ssllabs.com/ssltest/analyze.html?d=demo.wordops.eu&latest))
-   **Monitoring** : Live Nginx vhost traffic with ngx_vts_module and server monitoring with Netdata
-   **User Friendly** : WordOps dashboard with server status/monitoring and tools ([demo](https://demo.wordops.eu))
-   **Release cycle** : WordOps stable releases are published in June and December.

---

## Requirements

### Operating System

#### Recommended

-   Ubuntu 24.04 LTS (Noble)
-   Ubuntu 22.04 LTS (Jammy)
-   Ubuntu 20.04 LTS (Focal)

#### Also compatible

-   Debian 10 (Buster)
-   Debian 11 (Bullseye)
-   Debian 12 (Bookworm)

#### For testing purpose only

-   Raspbian 10 (Buster)
-   Raspbian 11 (Bullseye)

## Getting Started

```bash
wget -qO wo wops.cc && sudo bash wo      # Install WordOps
sudo wo site create example.com --wp     # Install required packages & setup WordPress on example.com
```

Detailed Getting Started guide with additional installation methods can be found in [the documentation](https://docs.wordops.net/getting-started/installation-guide/).

## Usage

### Standard WordPress sites

```bash
wo site create example.com --wp                # install wordpress with [Current supported PHP release](https://endoflife.date/php) without any page caching
wo site create example.com --wp  --php83       # install wordpress with PHP 8.3  without any page caching
wo site create example.com --wpfc              # install wordpress + nginx fastcgi_cache
wo site create example.com --wpredis           # install wordpress + nginx redis_cache
wo site create example.com --wprocket          # install wordpress with WP-Rocket plugin
wo site create example.com --wpce              # install wordpress with Cache-enabler plugin
wo site create example.com --wpsc              # install wordpress with wp-super-cache plugin
```

### WordPress multisite with subdirectory

```bash
wo site create example.com --wpsubdir            # install wpmu-subdirectory without any page caching
wo site create example.com --wpsubdir --wpsc     # install wpmu-subdirectory with wp-super-cache plugin
wo site create example.com --wpsubdir --wpfc     # install wpmu-subdirectory + nginx fastcgi_cache
wo site create example.com --wpsubdir --wpredis  # install wpmu-subdirectory + nginx redis_cache
wo site create example.com --wpsubdir --wprocket # install wpmu-subdirectory + WP-Rocket plugin
wo site create example.com --wpsubdir --wpce     # install wpmu-subdirectory + Cache-Enabler plugin
```

### WordPress multisite with subdomain

```bash
wo site create example.com --wpsubdomain            # install wpmu-subdomain without any page caching
wo site create example.com --wpsubdomain --wpsc     # install wpmu-subdomain with wp-super-cache plugin
wo site create example.com --wpsubdomain --wpfc     # install wpmu-subdomain + nginx fastcgi_cache
wo site create example.com --wpsubdomain --wpredis  # install wpmu-subdomain + nginx redis_cache
wo site create example.com --wpsubdomain --wprocket # install wpmu-subdomain + WP-Rocket plugin
wo site create example.com --wpsubdomain --wpce     # install wpmu-subdomain + Cache-Enabler plugin
```

### Non-WordPress sites

```bash
wo site create example.com --html     # create example.com for static/html sites
wo site create example.com --php      # create example.com with [Current supported PHP release](https://endoflife.date/php)
wo site create example.com --php80      # create example.com with php 8.0 support
wo site create example.com --php81      # create example.com with php 8.1 support
wo site create example.com --php82      # create example.com with php 8.2 support
wo site create example.com --mysql    # create example.com with php 8.2 & mysql support
wo site create example.com --mysql --php83   # create example.com with php 8.3 & mysql support
wo site create example.com --proxy=127.0.0.1:3000 #  create example.com with nginx as reverse-proxy
```

### Switch between PHP versions

```bash
wo site update example.com --php74 # switch to PHP 7.4
wo site update example.com --php80 # switch to PHP 8.0
wo site update example.com --php81 # switch to PHP 8.1
wo site update example.com --php82 # switch to PHP 8.2
wo site update example.com --php83 # switch to PHP 8.3
```

### Sites secured with Let's Encrypt

```bash
wo site create example.com --wp -le #  wordpress & letsencrypt
wo site create sub.example.com --wp -le # wordpress & letsencrypt subdomain
wo site create example.com --wp --letsencrypt --hsts # wordpress & letsencrypt with HSTS
wo site create example.com --wp -le=wildcard --dns=dns_cf # wordpress & wildcard SSL certificate with Cloudflare DNS API
```

## Update WordOps

```bash
wo update
```

## Support

If you feel there is a bug directly related to WordOps, or if you want to suggest new features for WordOps, feel free to open an issue.
For any other questions about WordOps or if you need support, please use the [Community Forum](https://community.wordops.net/).

# Contributing

If you'd like to contribute, please fork the repository and make changes as you'd like. Pull requests are warmly welcome.
There is no need to be a developer or a system administrator to contribute to WordOps project. You can still contribute by helping us to improve [WordOps documentation](https://github.com/WordOps/docs.wordops.net).
Otherwise, you can still contribute to the project by making a donation on [Ko-Fi](https://ko-fi.com/wordops).

## Credits

-   Source : [EasyEngine](https://github.com/easyengine/easyengine)

Apps & Tools shipped with WordOps :

-   [Acme.sh](https://github.com/Neilpang/acme.sh)
-   [WP-CLI](https://github.com/wp-cli/wp-cli)
-   [Netdata](https://github.com/netdata/netdata)
-   [phpMyAdmin](https://www.phpmyadmin.net/)
-   [Composer](https://github.com/composer/composer)
-   [Adminer](https://www.adminer.org/)
-   [phpRedisAdmin](https://github.com/erikdubbelboer/phpRedisAdmin)
-   [opcacheGUI](https://github.com/amnuts/opcache-gui)
-   [eXtplorer](https://github.com/soerennb/extplorer)
-   [Webgrind](https://github.com/jokkedk/webgrind)
-   [MySQLTuner](https://github.com/major/MySQLTuner-perl)
-   [Fail2Ban](https://github.com/fail2ban/fail2ban)
-   [ClamAV](https://github.com/Cisco-Talos/clamav-devel)
-   [cheat.sh](https://github.com/chubin/cheat.sh)
-   [ProFTPd](https://github.com/proftpd/proftpd)
-   [Nginx-ultimate-bad-bot-blocker](https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/)
-   [Nanorc](https://github.com/scopatz/nanorc)

Third-party debian packages shipped with WordOps :

-   [Nginx-wo by WordOps](https://build.opensuse.org/package/show/home:virtubox:WordOps/nginx)
-   [PHP by Ondřej Surý](https://launchpad.net/~ondrej/+archive/ubuntu/php)
-   [Redis](https://redis.io/docs/getting-started/installation/install-redis-on-linux/)

WordPress Cache Plugins supported by WordOps :

-   [Nginx-helper](https://github.com/rtCamp/nginx-helper)
-   [Cache-Enabler](https://github.com/keycdn/cache-enabler)
-   [Redis-object-cache](https://github.com/tillkruss/redis-cache)
-   [WP-Super-Cache](https://github.com/Automattic/wp-super-cache)
-   [WP-Rocket](https://github.com/wp-media/wp-rocket)

## License

-   [MIT](http://opensource.org/licenses/MIT) © [WordOps](https://wordops.net)


.. contents:: **tcconfig**
   :backlinks: top
   :depth: 2

Summary
=========
`tcconfig <https://github.com/thombashi/tcconfig>`__ is a tc command wrapper. Make it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).

|PyPI pkg ver| |Supported Python ver| |CI status| |CodeQL|

.. |PyPI pkg ver| image:: https://badge.fury.io/py/tcconfig.svg
    :target: https://badge.fury.io/py/tcconfig
    :alt: PyPI package version

.. |Supported Python ver| image:: https://img.shields.io/pypi/pyversions/tcconfig.svg
    :target: https://pypi.org/project/tcconfig
    :alt: Supported Python versions

.. |CI status| image:: https://github.com/thombashi/tcconfig/actions/workflows/ci.yml/badge.svg
    :target: https://github.com/thombashi/tcconfig/actions/workflows/ci.yml
    :alt: Test CI status

.. |CodeQL| image:: https://github.com/thombashi/tcconfig/actions/workflows/build_and_release.yml/badge.svg
   :target: https://github.com/thombashi/tcconfig/actions/workflows/build_and_release.yml
   :alt: Build CI status

Traffic control
------------------------

Setup traffic shaping rules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Easy to apply traffic shaping rules to specific networks:

- Outgoing/Incoming packets
- Source/Destination IP-address/network (IPv4/IPv6)
- Source/Destination ports

Available parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following parameters can be set to network interfaces:

- Network bandwidth rate ``[G/M/K bps]``
- Network latency ``[microseconds/milliseconds/seconds/minutes]``
- Packet loss rate ``[%]``
- Packet corruption rate ``[%]``
- Packet duplicate rate ``[%]``
- Packet reordering rate  ``[%]``

Targets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Network interfaces: e.g. ``eth0``
- Docker container (``veth`` corresponding with a container)

.. image:: docs/gif/tcset_example.gif

Usage
=======
Set traffic control (``tcset`` command)
-----------------------------------------
``tcset`` is a command to add a traffic control rule to a network interface (device).

e.g. Set a limit on bandwidth up to 100Kbps
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: console

    # tcset eth0 --rate 100Kbps

e.g. Set network latency
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use time units (such as us/sec/min/etc.) to designate delay time.

Set 100 milliseconds of network latency
'''''''''''''''''''''''''''''''''''''''''''''''''''
.. code-block:: console

    # tcset eth0 --delay 100ms

Set 10 seconds of network latency
'''''''''''''''''''''''''''''''''''''''''''''''''''
.. code-block:: console

    # tcset eth0 --delay 10sec

Set 0.5 minutes (30 seconds) network latency
'''''''''''''''''''''''''''''''''''''''''''''''''''
.. code-block:: console

    # tcset eth0 --delay 0.5min

You can also use the following time units:

.. table::

    +------------+----------------------------------------------------------+
    |    Unit    |                Available specifiers (str)                |
    +============+==========================================================+
    |hours       |``h``/``hour``/``hours``                                  |
    +------------+----------------------------------------------------------+
    |minutes     |``m``/``min``/``mins``/``minute``/``minutes``             |
    +------------+----------------------------------------------------------+
    |seconds     |``s``/``sec``/``secs``/``second``/``seconds``             |
    +------------+----------------------------------------------------------+
    |milliseconds|``ms``/``msec``/``msecs``/``millisecond``/``milliseconds``|
    +------------+----------------------------------------------------------+
    |microseconds|``us``/``usec``/``usecs``/``microsecond``/``microseconds``|
    +------------+----------------------------------------------------------+

e.g. Set 0.1% packet loss
^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: console

    # tcset eth0 --loss 0.1%

e.g. All of the above settings at once
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: console

    # tcset eth0 --rate 100Kbps --delay 100ms --loss 0.1%

e.g. Specify the IP address of the traffic control
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: console

    # tcset eth0 --delay 100ms --network 192.168.0.10

e.g. Specify the IP network and port of traffic control
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: console

    # tcset eth0 --delay 100ms --network 192.168.0.0/24 --port 80

Set traffic control to a docker container
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Execute ``tcconfig`` with ``--docker`` option on a Docker host:

.. code-block:: console

    # tcset <container name or ID> --docker ...

You could use ``--src-container``/``--dst-container`` options to specify the source/destination container.

Set traffic control within a docker container
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You need to run a container with ``--cap-add NET_ADMIN`` option
if you would like to set a tc rule within a container:

.. code-block:: console

    docker run -d --cap-add NET_ADMIN -t <docker image>

A container image that builtin tcconfig can be available at https://hub.docker.com/r/thombashi/tcconfig/

Delete traffic control (``tcdel`` command)
------------------------------------------
``tcdel`` is a command to delete traffic shaping rules from a network interface (device).

e.g. Delete traffic control of ``eth0``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can delete all of the shaping rules for the ``eth0`` with ``-a``/``--all`` option:

.. code-block:: console

    # tcdel eth0 --all

Display traffic control configurations (``tcshow`` command)
-----------------------------------------------------------
``tcshow`` is a command to display the current traffic control settings for network interface(s).

Example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: console

    # tcset eth0 --delay 10ms --delay-distro 2  --loss 0.01% --rate 0.25Mbps --network 192.168.0.10 --port 8080
    # tcset eth0 --delay 1ms --loss 0.02% --rate 500Kbps --direction incoming
    # tcshow eth0
    {
        "eth0": {
            "outgoing": {
                "dst-network=192.168.0.10/32, dst-port=8080, protocol=ip": {
                    "filter_id": "800::800",
                    "delay": "10.0ms",
                    "delay-distro": "2.0ms",
                    "loss": "0.01%",
                    "rate": "250Kbps"
                }
            },
            "incoming": {
                "protocol=ip": {
                    "filter_id": "800::800",
                    "delay": "1.0ms",
                    "loss": "0.02%",
                    "rate": "500Kbps"
                }
            }
        }
    }

For more information
----------------------
More examples are available at
https://tcconfig.rtfd.io/en/latest/pages/usage/index.html

Installation
============
Installation: pip
------------------------------
``tcconfig`` can be installed from `PyPI <https://pypi.python.org/pypi>`__ via
`pip <https://pip.pypa.io/en/stable/installing/>`__ (Python package manager) command.

.. code:: console

    sudo pip install tcconfig

Installation: dpkg (Debian/Ubuntu)
--------------------------------------------
.. code:: console

    curl -sSL https://raw.githubusercontent.com/thombashi/tcconfig/master/scripts/installer.sh | sudo bash

Dependencies
============
- Python 3.7+
- `Python package dependencies (automatically installed) <https://github.com/thombashi/tcconfig/network/dependencies>`__

Linux packages
--------------
- mandatory: required for ``tc`` command:
    - `Ubuntu`/`Debian`: ``iproute2``
    - `Fedora`/`RHEL`: ``iproute-tc``
- optional: required when you use the ``--iptables`` option:
    - ``iptables``

Linux kernel module
----------------------------
- ``sch_netem``

Optional Python packages
----------------------------
- `Pygments <http://pygments.org/>`__

Documentation
===============
https://tcconfig.rtfd.io/

Troubleshooting
=================
https://tcconfig.rtfd.io/en/latest/pages/troubleshooting.html

Docker image
==============
https://hub.docker.com/r/thombashi/tcconfig/

Sponsors
====================================
|chasbecker| |shiguredo| |b4tman| |Arturi0| |github|

.. |chasbecker| image:: https://avatars.githubusercontent.com/u/44389260?s=48&u=6da7176e51ae2654bcfd22564772ef8a3bb22318&v=4
   :target: https://github.com/chasbecker
   :alt: ex-sponsor: Charles Becker (chasbecker)
.. |shiguredo| image:: https://avatars.githubusercontent.com/u/2549434?s=48&v=4
   :target: https://github.com/shiguredo
   :alt: ex-sponsor: 時雨堂 (shiguredo)
.. |b4tman| image:: https://avatars.githubusercontent.com/u/3658062?s=48&v=4
   :target: https://github.com/b4tman
   :alt: onetime: Dmitry Belyaev (b4tman)
.. |Arturi0| image:: https://avatars.githubusercontent.com/u/46711571?s=48&u=57687c0e02d5d6e8eeaf9177f7b7af4c9f275eb5&v=4
   :target: https://github.com/Arturi0
   :alt: onetime: Arturi0
.. |github| image:: https://avatars.githubusercontent.com/u/9919?s=48&v=4
   :target: https://github.com/github
   :alt: onetime: GitHub (github)

`Become a sponsor <https://github.com/sponsors/thombashi>`__


# Onlineweb 4

[![codecov](https://codecov.io/gh/dotkom/onlineweb4/branch/main/graph/badge.svg)](https://codecov.io/gh/dotkom/onlineweb4)
[![Open in Remote - Containers](https://img.shields.io/static/v1?label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/dotkom/onlineweb4)

## Contributing

## Development environment

The by far easiest way to start developing is to use Visual Studio Code (VS Code) with
[remote-containers](https://code.visualstudio.com/docs/remote/containers).

After starting the container you can run the following to run the test-suite:

```sh
poetry install
poetry shell

npm ci
npm run build

# run tests
pytest

# install pre-commit hooks to automatically check formatting
pre-commit install

# to run
./manage.py migrate

./manage.py runserver

# OnlineWeb4 should be available on localhost:8000
```

## Authentication

Authentication goes through our Auth0-tenant, to get access to development configuration contact dotkom@online.ntnu.no.

## Frameworks

### Frontend

Frontend code is located in the `assets` folder. It's a mixture of old jQuery code and newer React code.
The code is bundled with [esbuild](https://esbuild.github.io).

### Backend

Python 3 with [Django](https://docs.djangoproject.com/) is used as a backend and is mostly located in the `apps` folder.
Django templates are in `templates` folder.
The API uses [Django REST framework](http://www.django-rest-framework.org/)

To find the list of available commands when using `python manage.py` (alternatively `./manage.py`), see the
[docs](https://docs.djangoproject.com/en/5.0/ref/django-admin/).

## Installation - Git and repository setup

### Setting up your Git user and cloning the repo

We recommend using GitHub's own [Command Line Interface (CLI)](https://cli.github.com):

```shell
gh auth login
gh repo clone dotkom/onlineweb4
cd onlineweb4
```

Normal flow for making a PR:

```sh
git switch -c new-feature
# exciting code
py.test
# can use `git add -ip`
git add <files>
git commit
gh pr create
```

### Local installation

To run the project you need Python 3.12 and node >=20 for bundling the frontend-dependencies.

We recommend using either [pyenv](https://github.com/pyenv/pyenv) or [rye](https://rye-up.com) to manage your Python-versions.
You will need [Poetry](https://python-poetry.org) to install the dependencies, we recommend using [pipx](https://github.com/pypa/pipx) to instal Poetry.
Using the devcontainer above automatically install the correct python-version and has pipx and poetry pre-installed.

For Node you can use [NVM](https://github.com/nvm-sh/nvm).

### Testing and linting

We use [Ruff](https://docs.astral.sh/ruff/) for linting and formatting.
To run the linting run `ruff check .`.
You can add git hooks that automatically check the linting when committing with `pre-committ install`.

```shell
pre-commit install
```

And run the lints manually by running

```shell
pre-commit run --all-files
```

To run the tests you can call

```shell
# most tests using Django templates require the `webpack-stats.json` to exists
npm run build

py.test
```

Which should work as long as you have properly set up the Python environment.

### Creating migrations

After doing changes to a model, you will need to make an migration for the database-scheme.
You can automatically create those commands by running the following:

```shell
python manage.py makemigrations
```

Note that migrations should be properly formatted and linted as other files, but should be fixed with our
pre-commit hooks.

## Continuous Integration and Continuous Delivery

Pushes made to the `main` branch will trigger a redeployment of the application on [dev.online.ntnu.no](https://dev.online.ntnu.no).

Pull requests trigger containerized builds that perform code style checks and tests.
You can view the details of these tests by clicking the "detail" link in the pull request checks status area.

> **Important:** We have integration tests with Stripe that require valid test-API-keys, those tests are **not**
> run by default locally, or when creating a PR from a fork of this repository. To run them, first get ahold of
> the appropriate testing keys, then add them to an `.env` file in the project root,
> the name of the environment variables are in
> [`.devcontainer/devcontainer.env`](.devcontainer/devcontainer.env).

### Manual release

### Zappa

The project is deployed to AWS Lambda with the use of [Zappa](https://github.com/zappa/zappa). To
deploy (should be done automatically), build a Docker image and push it to AWS ECR.
Then you can run `zappa update <stage> -d <docker-ecr-image>`. You'll have also have to build NPM and deploy static first if this has been changed since last deploy.

#### Example for prod

```shell
# create git tag / github release with release notes first
# if this is prod add `-prod` suffix
VERSION=4.X.X-prod
# OR VERSION=4.X.X if dev
STAGE=prod
REGION=eu-north-1
# log in to AWS in some way first https://docs.aws.amazon.com/cli/latest/userguide/getting-started-prereqs.html#getting-started-prereqs-keys
# jq is just to extract the "Account" json-key automatically
AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq .Account)
DOCKER_REGISTRY=$AWS_ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com
TAG=$DOCKER_REGISTRY/onlineweb4-zappa:$VERSION
aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $DOCKER_REGISTRY

# If zappa is not available you must install it, alternatively use devcontainer:
poetry install -E prod
# then either run `poetry shell` first, or prepend `poetry run` before the command
zappa save-python-settings-file $STAGE

docker build . --build-arg VERSION=$VERSION -t $IMAGE --no-cache
docker push $TAG
zappa update $STAGE -d $TAG

# If you also have changed static files you must run the following:
docker build . --target=static-files -t ow4-static
ID=$(docker create ow4-static)
docker cp $ID:/srv/app/static static
BUCKET_NAME=$( yq ".${STAGE}.environment_variables.OW4_S3_BUCKET_NAME" zappa_settings.yml )
aws s3 sync static "s3://${BUCKET_NAME}/static" --delete --acl=public-read
```

## API

Onlineweb4 comes with an API located at `/api/v1`.
Autogenerated Swagger-docs can be found at `/api/schema/swagger-ui`.


# Echopype

[![ssec](https://img.shields.io/badge/SSEC-Project-purple?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA0AAAAOCAQAAABedl5ZAAAACXBIWXMAAAHKAAABygHMtnUxAAAAGXRFWHRTb2Z0d2FyZQB3d3cuaW5rc2NhcGUub3Jnm+48GgAAAMNJREFUGBltwcEqwwEcAOAfc1F2sNsOTqSlNUopSv5jW1YzHHYY/6YtLa1Jy4mbl3Bz8QIeyKM4fMaUxr4vZnEpjWnmLMSYCysxTcddhF25+EvJia5hhCudULAePyRalvUteXIfBgYxJufRuaKuprKsbDjVUrUj40FNQ11PTzEmrCmrevPhRcVQai8m1PRVvOPZgX2JttWYsGhD3atbHWcyUqX4oqDtJkJiJHUYv+R1JbaNHJmP/+Q1HLu2GbNoSm3Ft0+Y1YMdPSTSwQAAAABJRU5ErkJggg==&style=plastic)](https://escience.washington.edu/echopype/)

Echopype is a package built to enable interoperability and scalability in ocean sonar data processing. These data are widely used for obtaining information about the distribution and abundance of marine animals, such as fish and krill. Our ability to collect large volumes of sonar data from a variety of ocean platforms has grown significantly in the last decade. However, most of the new data remain under-utilized. echopype aims to address the root cause of this problem - the lack of interoperable data format and scalable analysis workflows that adapt well with increasing data volume - by providing open-source tools as entry points for scientists to make discovery using these new data.

Watch the [echopype talk](https://www.youtube.com/watch?v=qboH7MyHrpU)
at SciPy 2019 for background, discussions and a quick demo!

## Documentation

Learn more about echopype in the official documentation at https://echopype.readthedocs.io. Check out executable examples in the companion repository https://github.com/OSOceanAcoustics/echopype-examples.

## Contributing

You can find information about how to contribute to echopype at our [Contributing Page](https://echopype.readthedocs.io/en/latest/contributing.html).

##

In collaboration with the [Integrated Ocean Observing System (IOOS)](https://ioos.noaa.gov/), the Echopype team aims to recruit talented [Google Summer of Code (GSoC)](https://summerofcode.withgoogle.com/)
participants to help us upgrade the robustness and scalability of the Echopype package.

If you are a GSoC 2024 contributor, please head over to [GSoC contributor's guide](gsoc_contrib_guide.md) to get more information specific to the program.

## Echopype doesn't run on your data?

Please report any bugs by [creating issues on GitHub](https://medium.com/nyc-planning-digital/writing-a-proper-github-issue-97427d62a20f).

[Pull requests](https://jarednielsen.com/learn-git-fork-pull-request/) are always welcome!

## Contributors

[![Contributors](https://contrib.rocks/image?repo=OSOceanAcoustics/echopype)](https://github.com/OSOceanAcoustics/echopype/graphs/contributors)

Wu-Jung Lee ([@leewujung](https://github.com/leewujung))
founded the echopype project in 2018 and continue to be the primary contributor
together with Praneeth Ratna([@praneethratna](https://github.com/praneethratna)).
Emilio Mayorga ([@emiliom](https://github.com/emiliom)),
Landung "Don" Setiawan ([@lsetiawan](https://github.com/lsetiawan)),
Brandon Reyes ([@b-reyes](https://github.com/b-reyes)),
Kavin Nguyen ([@ngkavin](https://github.com/ngkavin))
and Imran Majeed ([@imranmaj](https://github.com/imranmaj))
have contributed significantly to the code.
Valentina Staneva ([@valentina-s](https://github.com/valentina-s)) is also part of the development team.

A complete list of direct contributors is on our [GitHub Contributors Page](https://github.com/OSOceanAcoustics/echopype/graphs/contributors).

## Acknowledgement

We thank Dave Billenness of ASL Environmental Sciences for
providing the AZFP Matlab Toolbox as reference for developing
support for the AZFP echosounder,
and Rick Towler ([@rhtowler](https://github.com/rhtowler))
of the NOAA Alaska Fisheries Science Center
for providing low-level file parsing routines for
Simrad EK60 and EK80 echosounders.

We also thank funding support from
the National Science Foundation and NOAA Ocean Exploration,
and software engineering support from
the University of Washington Scientific Software Engineering Center (SSEC),
as part of the Schmidt Futures Virtual Institute for Scientific Software (VISS) in 2023.

## License

Echopype is licensed under the open source [Apache 2.0 license](https://opensource.org/licenses/Apache-2.0).

---------------

Copyright (c) 2018-2024, Echopype Developers.


====
unyt
====

.. image:: https://img.shields.io/pypi/v/unyt.svg
        :target: https://pypi.python.org/pypi/unyt

.. image:: https://img.shields.io/conda/vn/conda-forge/unyt.svg
        :target: https://anaconda.org/conda-forge/unyt
        :alt: conda-forge

.. image:: https://github.com/yt-project/unyt/actions/workflows/ci.yml/badge.svg?branch=main
        :target: https://github.com/yt-project/unyt/actions/workflows/ci.yml

.. image:: https://github.com/yt-project/unyt/actions/workflows/bleeding-edge.yaml/badge.svg?branch=main
        :target: https://github.com/yt-project/unyt/actions/workflows/bleeding-edge.yaml

.. image:: https://readthedocs.org/projects/unyt/badge/?version=latest
        :target: https://unyt.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: http://joss.theoj.org/papers/dbc27acb614dd33eb02b029ef20e7fe7/status.svg
        :target: http://joss.theoj.org/papers/dbc27acb614dd33eb02b029ef20e7fe7
        :alt: Code Paper

|

 .. image:: https://raw.githubusercontent.com/yt-project/unyt/master/docs/_static/yt_logo_small.png
         :target: https://yt-project.org
         :alt: The yt Project

A package for handling numpy arrays with units.

Often writing code that deals with data that has units can be confusing. A
function might return an array but at least with plain NumPy arrays, there is no
way to easily tell what the units of the data are without somehow knowing *a
priori*.

The ``unyt`` package (pronounced like "unit") provides a subclass of NumPy's
``ndarray`` class that knows about units. For example, one could do:

    >>> import unyt as u
    >>> distance_traveled = [3.4, 5.8, 7.2] * u.mile
    >>> print(distance_traveled.to('km'))
    [ 5.4717696  9.3341952 11.5872768] km

And a whole lot more! See `the documentation <http://unyt.readthedocs.io>`_ for
installation instructions, more examples, and full API reference.

This package only depends on ``numpy`` and ``sympy``.  Notably, it does *not*
depend on ``yt`` and it is written in pure Python.

Code of Conduct
---------------

The ``unyt`` package is part of `The yt Project
<https://yt-project.org>`_. Participating in ``unyt`` development therefore
happens under the auspices of the `yt community code of conduct
<http://yt-project.org/doc/developing/developing.html#yt-community-code-of-conduct>`_. If
for any reason you feel that the code of conduct has been violated, please send
an e-mail to confidential@yt-project.org with details describing the
incident. All emails sent to this address will be treated with the strictest
confidence by an individual who does not normally participate in yt development.

License
-------

The unyt package is licensed under the BSD 3-clause license.

Citation
--------

If you make use of unyt in work that leads to a publication we would appreciate
a mention in the text of the paper or in the acknowledgements along with a
citation to our `paper
<https://joss.theoj.org/papers/dbc27acb614dd33eb02b029ef20e7fe7>`_ in the
Journal of Open Source Software. You can use the following BibTeX::

 @article{Goldbaum2018,
   doi = {10.21105/joss.00809},
   url = {https://doi.org/10.21105/joss.00809},
   year  = {2018},
   month = {aug},
   publisher = {The Open Journal},
   volume = {3},
   number = {28},
   pages = {809},
   author = {Nathan J. Goldbaum and John A. ZuHone and Matthew J. Turk and Kacper Kowalik and Anna L. Rosen},
   title = {unyt: Handle,  manipulate,  and convert data with units in Python},
   journal = {Journal of Open Source Software}
 }

Or the following citation format:

  Goldbaum et al., (2018). unyt: Handle, manipulate, and convert data with units in Python . Journal of Open Source Software, 3(28), 809, https://doi.org/10.21105/joss.00809


pytds
=====

.. image:: https://secure.travis-ci.org/denisenkom/pytds.png?branch=master
   :target: https://travis-ci.org/denisenkom/pytds

.. image:: https://ci.appveyor.com/api/projects/status/a5h4y29063crqtet?svg=true
   :target: https://ci.appveyor.com/project/denisenkom/pytds

.. image:: http://img.shields.io/pypi/v/python-tds.svg
   :target: https://pypi.python.org/pypi/python-tds/

.. image:: https://codecov.io/gh/denisenkom/pytds/branch/master/graph/badge.svg
  :target: https://codecov.io/gh/denisenkom/pytds

`Python DBAPI`_ driver for MSSQL using pure Python TDS (Tabular Data Stream) protocol implementation.
Doesn't depend on ADO or FreeTDS.  Can be used on any platform, including Linux, MacOS, Windows.

It can be used with https://pypi.python.org/pypi/django-sqlserver as a Django database backend.

Features
--------

* Fully supports new MSSQL 2008 date types: datetime2, date, time, datetimeoffset
* MARS
* Bulk insert
* Table-valued parameters
* TLS connection encryption
* Kerberos support on non-Windows platforms (requires kerberos package)

Installation
------------

To install run this command:

.. code-block:: bash

    $ pip install python-tds

If you want to use TLS you should also install pyOpenSSL package:

.. code-block:: bash

   $ pip install pyOpenSSL

For a better performance install bitarray package too:

.. code-block:: bash

    $ pip install bitarray

To use Kerberos on non-Windows platforms (experimental) install kerberos package:

.. code-block:: bash

    $ pip install kerberos

Documentation
-------------
Documentation is available at https://python-tds.readthedocs.io/en/latest/.

Example
-------

To connect to database do

.. code-block:: python

    import pytds
    with pytds.connect('server', 'database', 'user', 'password') as conn:
        with conn.cursor() as cur:
            cur.execute("select 1")
            cur.fetchall()

To enable TLS you should also provide cafile parameter which should be a file name containing trusted CAs in PEM format.

For detailed documentation of connection parameters see: `pytds.connect`_

.. _Python DBAPI: http://legacy.python.org/dev/peps/pep-0249/
.. _pytds.connect: https://python-tds.readthedocs.io/en/latest/pytds.html#pytds.connect


# pyGeo
[![Build Status](https://dev.azure.com/mdolab/Public/_apis/build/status/mdolab.pygeo?branchName=main)](https://dev.azure.com/mdolab/Public/_build/latest?definitionId=17&branchName=main)
[![Documentation Status](https://readthedocs.com/projects/mdolab-pygeo/badge/?version=latest)](https://mdolab-pygeo.readthedocs-hosted.com/en/latest/?badge=latest)
[![codecov](https://codecov.io/gh/mdolab/pygeo/branch/main/graph/badge.svg?token=N2L58WGCDI)](https://codecov.io/gh/mdolab/pygeo)
[![DOI](https://joss.theoj.org/papers/10.21105/joss.05319/status.svg)](https://doi.org/10.21105/joss.05319)

pyGeo is an object oriented geometry manipulation framework for multidisciplinary design optimization (MDO).
It can be used for MDO within the [MACH framework](https://github.com/mdolab/MACH-Aero) and within [OpenMDAO](https://github.com/OpenMDAO/OpenMDAO) through [MPhys](https://github.com/OpenMDAO/mphys).
Its parameterization options include a free form deformation (FFD) implementation, an interface to NASA's [OpenVSP](https://openvsp.org/) parametric geometry tool, and an interface to the CAD package [ESP](https://acdl.mit.edu/ESP/).
It also includes geometric constraints and utility functions for geometry manipulation.

![](doc/images/DPW4_FFD-27745.gif)

## Documentation
Please see the [documentation](https://mdolab-pygeo.readthedocs-hosted.com/en/latest/) for installation details and API documentation.

To locally build the documentation, enter the `doc` folder and enter `make html` in terminal.
You can then view the built documentation in the `_build` folder.

## Citation
Please cite pyGeo in any publication for which you find it useful.
Citation information can be found [here](https://mdolab-pygeo.readthedocs-hosted.com/en/latest/citation.html).

## License
pyGeo is licensed under the Apache License, Version 2.0 (the "License"). See `LICENSE` for the full license.

## Copyright
Copyright (c) 2012 University of Toronto
Copyright (c) 2014 University of Michigan
Additional copyright (c) 2014 Gaetan K. W. Kenway, Charles A. Mader, and Joaquim R. R. A. Martins
All rights reserved.


.. image:: https://img.shields.io/pypi/v/pmxbot.svg
   :target: https://pypi.org/project/pmxbot

.. image:: https://img.shields.io/pypi/pyversions/pmxbot.svg

.. image:: https://github.com/pmxbot/pmxbot/actions/workflows/main.yml/badge.svg
   :target: https://github.com/pmxbot/pmxbot/actions?query=workflow%3A%22tests%22
   :alt: tests

.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json
    :target: https://github.com/astral-sh/ruff
    :alt: Ruff

.. image:: https://readthedocs.org/projects/pmxbot/badge/?version=latest
   :target: https://pmxbot.readthedocs.io/en/latest/?badge=latest

.. image:: https://img.shields.io/badge/skeleton-2024-informational
   :target: https://blog.jaraco.com/skeleton

.. image:: https://tidelift.com/badges/package/pypi/pmxbot
   :target: https://tidelift.com/subscription/pkg/pypi-pmxbot?utm_source=pypi-pmxbot&utm_medium=readme

pmxbot is bot for IRC and Slack written in
`Python <https://python.org>`_. Originally built for internal use
at `YouGov <https://yougov.com/>`_,
it's been sanitized and set free upon the world. You can find out more details
on `the project website <https://github.com/pmxbot/pmxbot>`_.

Commands
========

pmxbot listens to commands prefixed by a '!'
If it's a command, it knows it will reply, take an action, etc.
It can search the web, store quotes you, track karma, make decisions,
and do just about anything else you could want. It stores logs and quotes
and karma in either a sqlite or MongoDB
database, and there's a web interface for reviewing the logs and karma.

Contains
========

pmxbot will respond to things you say if it detects words and phrases it's
been told to recognize. For example, mention sql on rails.

Requirements
============

`pmxbot` requires Python 3. It also requires a few python packages as defined
in setup.py. Some optional dependencies are installed with
`extras
<https://packaging.python.org/installing/#installing-setuptools-extras>`_:

- mongodb: Enable MongoDB persistence (instead of sqlite).
- irc: IRC bot client.
- slack: Slack bot client.
- viewer: Enable the web viewer application.

Testing
=======

`pmxbot` includes a test suite that does some functional tests written against
the Python IRC server and quite a few unit tests as well. Install
`tox <https://pypi.org/project/tox>`_ and run ``tox`` to invoke the tests.

Configuration
=============

Configuration is based on very easy YAML files. Check out config.yaml in the
source tree for an example.

Usage
=====

Once you've setup a config file, you just need to call ``pmxbot config.yaml``
and it will join and connect. We recommend running pmxbot under
your favorite process supervisor to make it
automatically restart if it crashes (or terminates due to a planned
restart).

Custom Features
===============

Setuptools Entry Points Plugin
------------------------------

``pmxbot`` provides an extension mechanism for adding commands, and uses this
mechanism even for its own built-in commands.

To create a setuptools
entry point plugin, package your modules using
the setuptools tradition and install it alongside pmxbot. Your package
should define an entry point in the group ``pmxbot_handlers`` by including
something similar to the following in the package's setup.py::

    entry_points = {
        'pmxbot_handlers': [
            'plugin name = pmxbot.mymodule',
        ],
    },

During startup,
pmxbot will load ``pmxbot.mymodule``. ``plugin name`` can be anything, but should
be a name suitable to identify the plugin (and it will be displayed during
pmxbot startup).

Note that the ``pmxbot`` package is a namespace package, and you're welcome
to use that namespace for your plugin (e.g.
`pmxbot.nsfw <https://github.com/pmxbot/pmxbot.nsfw>`_).

If your plugin requires any initialization, specify an initialization function
(or class method) in the entry point. For example::

    'plugin name = pmxbot.mymodule:initialize_func'

On startup, pmxbot will call ``initialize_func`` with no parameters.

Within the script you'll want to import the decorator(s) you need to use with::

    from pmxbot.core import command, contains, regexp, execdelay, execat`.

You'll
then decorate each function with the appropriate line so pmxbot registers it.

A command (!g) gets the @command decorator::

  @command(aliases=('tt', 'tear', 'cry'))
  def tinytear(rest):
    "I cry a tiny tear for you."
    if rest:
      return "/me sheds a single tear for %s" % rest
    else:
      return "/me sits and cries as a single tear slowly trickles down its cheek"

A response (when someone says something) uses the @contains decorator::

  @contains("sqlonrails")
  def yay_sor():
    karma.Karma.store.change('sql on rails', 1)
    return "Only 76,417 lines..."

Each handler may solicit any of the following parameters:

 - channel (the channel in which the message occurred)
 - nick (the nickname that triggered the command or behavior)
 - rest (any text after the command)

A more complicated response (when you want to extract data from a message) uses
the @regexp decorator::

    @regexp("jira", r"(?<![a-zA-Z0-9/])(OPS|LIB|SALES|UX|GENERAL|SUPPORT)-\d\d+")
    def jira(client, event, channel, nick, match):
        return "https://jira.example.com/browse/%s" % match.group()

For an example of how to implement a setuptools-based plugin, see one of the
many examples in the pmxbot project itself or one of the popular third-party
projects:

 - `motivation <https://github.com/pmxbot/motivation>`_.
 - `wolframalpha <https://github.com/jaraco/wolframalpha>`_.
 - `jaraco.translate <https://github.com/jaraco/jaraco.translate>`_.
 - `excuses <https://github.com/pmxbot/excuses>`_.

Web Interface
=============

pmxbot includes a web server for allowing users to view the logs, read the
help, and check karma. You specify the host, port, base path, logo, title,
etc with the same YAML config file. Just run like ``pmxbotweb config.yaml``
and it will start up. Like pmxbot, use of a supervisor is recommended to
restart the process following termination.

pmxbot as a Slack bot (native)
==============================

To use pmxbot as a Slack bot, install with ``pmxbot[slack]``,
and set ``slack token`` in your config to the token from your
`Bot User <https://api.slack.com/bot-users>`_. Easy, peasy.

pmxbot as a Slack bot (IRC)
===========================

As Slack provides an IRC interface, it's easy to configure pmxbot for use
in Slack. Here's how:

0. Install with ``pmxbot[irc]``.
1. `Enable the IRC Gateway <https://slack.zendesk.com/hc/en-us/articles/201727913-Connecting-to-Slack-over-IRC-and-XMPP>`.
2. Create an e-mail for the bot.
3. Create the account for the bot in Slack and activate its account.
4. Log into Slack using that new account and `get the IRC gateway
   password <https://my.slack.com/account/gateways>` for that
   account.
5. Configure the pmxbot as you would for an IRC server, but use these
   settings for the connection:

    message rate limit: 2.5
    password: <gateway password>
    server_host: <team name>.irc.slack.com
    server_port: 6667

   The rate limit is necessary because Slack will kick the bot if it issues more than 25 messages in 10 seconds, so throttling it to 2.5 messages per
   second avoids hitting the limit.
6. Consider leaving 'log_channels' and 'other_channels' empty, especially
   if relying on Slack logging. Slack will automatically re-join pmxbot to
   any channels to which it has been ``/invited``.

For Enterprise
==============

Available as part of the Tidelift Subscription.

This project and the maintainers of thousands of other packages are working with Tidelift to deliver one enterprise subscription that covers all of the open source you use.

`Learn more <https://tidelift.com/subscription/pkg/pypi-pmxbot?utm_source=pypi-pmxbot&utm_medium=referral&utm_campaign=github>`_.


| **IronPython 3 has been released and is now available for download at https://github.com/IronLanguages/ironpython3!** |
| --- |

IronPython
===
IronPython is an open-source implementation of the Python programming language which is tightly integrated with the .NET Framework. IronPython can use the .NET Framework and Python libraries, and other .NET languages can use Python code just as easily.

IronPython can be obtained at [http://ironpython.net/](http://ironpython.net/).

| **What?** | **Where?** |
| --------: | :------------: |
| **Windows/Linux/macOS Builds** | [![Azure build status](https://dotnet.visualstudio.com/IronLanguages/_apis/build/status/ironpython2)](https://dotnet.visualstudio.com/IronLanguages/_build/latest?definitionId=42) [![Github build status](https://github.com/IronLanguages/ironpython2/workflows/CI/badge.svg)](https://github.com/IronLanguages/ironpython2/actions?workflow=CI) |
| **Downloads** | [![NuGet](https://img.shields.io/nuget/v/IronPython.svg)](https://www.nuget.org/packages/IronPython/) [![Release](https://img.shields.io/github/release/IronLanguages/ironpython2.svg)](https://github.com/IronLanguages/ironpython2/releases/latest)|
| **Help** | [![Gitter chat](https://badges.gitter.im/IronLanguages/ironpython.svg)](https://gitter.im/IronLanguages/ironpython) [![StackExchange](https://img.shields.io/stackexchange/stackoverflow/t/ironpython.svg)](http://stackoverflow.com/questions/tagged/ironpython) |

Comparison of IronPython vs. C# for 'Hello World'

C#:

```cs
using System;
class Hello
{
    static void Main()
    {
        Console.WriteLine("Hello World");
    }
}
```

IronPython:
```py
print "Hello World"
```
IronPython is a Dynamic Language that runs on the .NET DLR ([Dynamic Language Runtime](http://en.wikipedia.org/wiki/Dynamic_Language_Runtime)) in contrast with VB.NET and C# which are [static languages](http://en.wikipedia.org/wiki/Type_system).

IronPython can also import DLL files compiled in other languages and use functions defined therein. For example:

```py
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import *
```

# Code of Conduct
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community.
For more information see the [.NET Foundation Code of Conduct](https://dotnetfoundation.org/code-of-conduct).

# Documentation

Documentation can be found here: http://ironpython.net/documentation/dotnet/

## Additional information

Please see http://wiki.github.com/IronLanguages/main for information on:
- Setting up a development environment with easy access to utility scripts
- [Building](Documentation/building.md)
- Running test

## Chat/Communication

Join our Gitter-Chat under: https://gitter.im/IronLanguages/ironpython


# Music Structure Analysis Framework #

A Python framework to analyze music structure.

[![PyPI version](https://badge.fury.io/py/msaf.svg)](https://badge.fury.io/py/msaf)
[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/urinieto/msaf/master/LICENSE.md)
[![Build Status](https://github.com/urinieto/msaf/actions/workflows/test.yaml/badge.svg)](https://github.com/urinieto/msaf/actions/workflows/test.yaml)
[![Coverage Status](https://coveralls.io/repos/github/urinieto/msaf/badge.svg?branch=master)](https://coveralls.io/github/urinieto/msaf?branch=master)
[![Documentation Status](https://readthedocs.org/projects/msaf/badge/?version=latest)](https://msaf.readthedocs.io/en/latest/?badge=latest)

## Documentation ##

See https://msaf.readthedocs.io for a complete reference manual and introductory tutorials.

## Installation ##

From the root folder, type:

    pip install .

(Note: you may need to create and activate a Python virtual environment with `python -m venv .venv` and `source .venv/bin/activate` first, depending on your system configuration).

## Demonstration Notebook ##

You can follow a thorough example on this titanic [Jupyter Notebook](https://github.com/urinieto/msaf/blob/main/examples/Run%20MSAF.ipynb).

## Citing MSAF ##

Nieto, O., Bello, J. P., Systematic Exploration Of Computational Music Structure Research. Proc. of the 17th International Society for Music Information Retrieval Conference (ISMIR). New York City, NY, USA, 2016 ([PDF](https://ccrma.stanford.edu/~urinieto/MARL/publications/ISMIR2016-NietoBello.pdf)).

## Credits ##

Created by [Oriol Nieto](http://marl.smusic.nyu.edu/nieto/) (<oriol@nyu.edu>).


<!-- This README is also published on PyPI. GitHub relative links, like
[link](docs/README.md) do not work on PyPI, so don't use them here. -->

PyRTL
=====

[![PyPI version](https://badge.fury.io/py/pyrtl.svg)](http://badge.fury.io/py/pyrtl)
[![Build Status](https://github.com/UCSBarchlab/PyRTL/actions/workflows/python-test.yml/badge.svg)](https://github.com/UCSBarchlab/PyRTL/actions/workflows/python-test.yml)
[![Code Coverage](https://codecov.io/github/UCSBarchlab/PyRTL/coverage.svg?branch=development)](https://codecov.io/github/UCSBarchlab/PyRTL?branch=development)
[![Documentation Status](https://readthedocs.org/projects/pyrtl/badge/?version=latest)](http://pyrtl.readthedocs.org/en/latest/?badge=latest)
[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/UCSBarchlab/PyRTL/development?filepath=%2Fipynb-examples%2F)

PyRTL provides a collection of classes for Pythonic [register-transfer
level](https://en.wikipedia.org/wiki/Register-transfer_level) design,
simulation, tracing, and testing suitable for teaching and research.
Simplicity, usability, clarity, and extensibility rather than performance or
optimization is the overarching goal. Features include:

* Elaboration-through-execution, meaning all of Python can be used including
  introspection
* Design, instantiate, and simulate all in one file and without leaving Python
* Export to, or import from, common HDLs (BLIF-in, Verilog-out currently
  supported)
* Examine execution with waveforms on the terminal or export to `.vcd` as
  projects scale
* Elaboration, synthesis, and basic optimizations all included
* Small and well-defined internal core structure means writing new transforms
  is easier
* Batteries included means many useful components are already available and
  more are coming every week

What README would be complete without a screenshot? Below you can see the
waveform rendered right on the terminal for a small state machine written in
PyRTL.

![Command-line waveform for PyRTL state machine](https://github.com/UCSBarchlab/PyRTL/blob/development/docs/screenshots/pyrtl-statemachine.png?raw=true "PyRTL State Machine Screenshot")

### Tutorials and Documentation

* For users, more info and demo code is available on the [PyRTL project web
  page](http://ucsbarchlab.github.io/PyRTL/).
* Try the examples in the `examples/` directory. You can also [try the examples
  on
  MyBinder](https://mybinder.org/v2/gh/UCSBarchlab/PyRTL/development?filepath=%2Fipynb-examples%2F).
* Full reference documentation is available at https://pyrtl.readthedocs.io/

### Package Contents

If you are just getting started with PyRTL it is suggested that you start with
the `examples/` first to get a sense of the "thinking with PyRTLs" required to
design hardware in this way. If you are looking for a deeper understanding,
dive into the code for the object `Block`. It is the core data structure at the
heart of PyRTL and defines its semantics at a high level -- everything is
converted to or from the small, simple set of primitives defined there.

The package contains the following files and directories:
* **`pyrtl/`**  The src directory for the module
* **`pyrtl/rtllib/`** Finished PyRTL libraries which are hopefully both useful
  and documented
* **`examples/`** A set of hardware design examples that show the main idea
  behind pyrtl
* **`tests/`** A set of unit tests for PyRTL which you can run with `pytest`
* **`docs/`** Location of the sphinx documentation

Testing requires the Python packages `tox` and `pytest`. Once installed a
complete test of the system should be possible with the simple command `tox`
and nothing more.

### Contributing to PyRTL

*Picking a first project*

* One of the earliest things you should submit is a unit test that hits some
  [uncovered lines of code in
  PyRTL](https://codecov.io/github/UCSBarchlab/PyRTL?branch=development). For
  example, pick a `PyrtlError` that is not covered and add a unit test in
  `tests/` that will hit it.
* After you have that down check in the [PyRTL
  Issues](https://github.com/UCSBarchlab/PyRTL/issues) list for a feature that
  is marked as "beginner friendly".
* Once you have that down, ask for access to the PyRTL-research repo where we
  keep a list of more advanced features and designs that could use more help!

*Coding style*

* All major functionality should have unit tests covering and documenting their
  use
* All public functions and methods should have useful docstrings
* All code needs to conform to
  [PEP8](https://www.python.org/dev/peps/pep-0008/) conventions
* No new root-level dependencies on external libs, import locally if required
  for special functions

*Workflow*

* A useful reference for working with Git is this [Git
  tutorial](https://www.atlassian.com/git/tutorials/)
* A useful Git Fork workflow for working on this repo is [found
  here](http://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/)
* The `development` branch is the primary stable working branch (everyone is
  invited to submit pull requests)
* Bugs and minor enhancements tracked directly through the [issue
  tracker](https://github.com/UCSBarchlab/PyRTL/issues)
* When posting a bug please post a small chunk of code that captures the bug,
  e.g. [Issue #56](https://github.com/UCSBarchlab/PyRTL/issues/56)
* When pushing a fix to a bug or enhancement please reference issue number in
  commit message, e.g. [Fix to Issue
  #56](https://github.com/UCSBarchlab/PyRTL/commit/1d5730db168a9e4490c580cb930075715468047a)

*Documentation*

* All important functionality should have an executable example in `examples/`
* All classes should have a block comment with high level description of the
  class
* All functions should follow the following (Sphinx parsable) docstring format:
  ```python
  """One Line Summary (< 80 chars) of the function, followed by period.

  :param param_name : Description of this parameter.
  :param param_name : Longer parameter descriptions take up a newline with four
      leading spaces like this.
  :return: Description of function's return value.

  A long description of what this function does. Talk about what the user
  should expect from this function and also what the users needs to do to use
  the function (this part is optional).

  """

  # Developer Notes (Optional):
  #
  # These would be anything that the user does not need to know in order to use
  # the functions.
  # These notes can include internal workings of the function, the logic behind
  # it, or how to extend it.
  ```
* Sphinx parses [Python type
  annotations](https://docs.python.org/3/library/typing.html), so put type
  information into annotations instead of docstrings.
* The Sphinx-generated documentation is published to
  https://pyrtl.readthedocs.io/
* PyRTL's Sphinx build process is documented in
  [`docs/README.md`](https://github.com/UCSBarchlab/PyRTL/blob/development/docs/README.md).
* PyRTL's release process is documented in
  [`release/README.md`](https://github.com/UCSBarchlab/PyRTL/blob/development/release/README.md).

### Using PyRTL

We love to hear from users about their projects, and if there are issues we
will try our best to push fixes quickly. You can read more about how we have
been using it in our research at UCSB both in simulation and on FPGAs in [our
PyRTL paper at FPL](http://www.cs.ucsb.edu/~sherwood/pubs/FPL-17-pyrtl.pdf).

### Related Projects

It is always important to point out that PyRTL builds on the ideas of several
other related projects as we all share the common goal of trying to make
hardware design a better experience! You can read more about those
relationships on our [PyRTL project web
page](http://ucsbarchlab.github.io/PyRTL/).


QCEngine
========
[![Build Status](https://github.com/MolSSI/QCEngine/workflows/CI/badge.svg?branch=master)](https://github.com/MolSSI/QCEngine/actions?query=workflow%3ACI)
[![codecov](https://img.shields.io/codecov/c/github/MolSSI/QCEngine.svg?logo=Codecov&logoColor=white)](https://codecov.io/gh/MolSSI/QCEngine)
[![Documentation Status](https://img.shields.io/github/actions/workflow/status/MolSSI/QCEngine/CI.yml?label=docs&logo=readthedocs&logoColor=white)](https://molssi.github.io/QCEngine/)
[![Conda (channel only)](https://img.shields.io/conda/vn/conda-forge/qcengine?color=blue&logo=anaconda&logoColor=white)](https://anaconda.org/conda-forge/qcengine)
[![Chat on Slack](https://img.shields.io/badge/chat-on_slack-808493.svg?longCache=true&style=flat&logo=slack)](https://join.slack.com/t/qcarchive/shared_invite/enQtNDIzNTQ2OTExODk0LTE3MWI0YzBjNzVhNzczNDM0ZTA5MmQ1ODcxYTc0YTA1ZDQ2MTk1NDhlMjhjMmQ0YWYwOGMzYzJkZTM2NDlmOGM)
![python](https://img.shields.io/badge/python-3.7+-blue.svg)

<!--[![Azure Build Status](https://dev.azure.com/MolSSI/QCArchive/_apis/build/status/MolSSI.QCEngine?branchName=master)](https://dev.azure.com/MolSSI/QCArchive/_build/latest?definitionId=5&branchName=master)-->
Quantum chemistry program executor and IO standardizer ([QCSchema](https://github.com/MolSSI/QCSchema)) for quantum chemistry.

# Example

A simple example of QCEngine's capabilities is as follows:

```python
>>> import qcengine as qcng
>>> import qcelemental as qcel

>>> mol = qcel.models.Molecule.from_data("""
O  0.0  0.000  -0.129
H  0.0 -1.494  1.027
H  0.0  1.494  1.027
""")

>>> inp = qcel.models.AtomicInput(
    molecule=mol,
    driver="energy",
    model={"method": "SCF", "basis": "sto-3g"},
    keywords={"scf_type": "df"}
    )
```

These input specifications can be executed with the ``compute`` function along with a program specifier:

```python
>>> ret = qcng.compute(inp, "psi4")
```

The results contain a complete record of the computation:

```python
>>> ret.return_result
-74.45994963230625

>>> ret.properties.scf_dipole_moment
[0.0, 0.0, 0.6635967188869244]

>>> ret.provenance.cpu
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
```

See the [documentation](https://molssi.github.io/QCEngine/) for more information.

# License

BSD-3C. See the [License File](LICENSE) for more information.


Marvin
======

Marvin is the ultimate tool to visualise and analyse MaNGA data. It is
developed and maintained by the MaNGA team.

|Build Package| |Run Test Suite| |CodeCov| |Coverage Status| |PyPI| |DOI| |astropy|
|readthedocs|

Installation
------------

To install Marvin:

::

    pip install sdss-marvin

If you would like to contribute to Marvin's development, you can clone
this git repo, and run pip install in editable mode which will install all the
dev dependencies:

::

    git clone https://github.com/sdss/marvin
    cd marvin
    pip install -e ".[dev,web,db]"

This will install all dependencies needed for development and testing, including for
the Marvin web/api server and the Marvin use of the MaNGA database.

Testing
-------

To run the test suite on Marvin, use ``pytest tests/``.  By default this will run all tests against
Marvin's core file access mode, as well as its database and remote access modes. The test suite uses
the following two galaxies: `8485-1901` and `7443-12701` for testing. To successfully run all tests
you will need:

*  The file data for the two test galaxies
*  A local MaNGA database with the galaxy loaded into it
*  A local Marvin web server running

You can download the necessary test data with the command.
::

   python bin/get_test_data.py

You can run a local Marvin web server with the following command.  This will start a local web server running
in debug mode on port 5000.
::

   cd bin/
   run_marvin -d -p 5000

You can disable database and web server tests by using Pytest markers.  The availble markers and options are:

*  **slow**: Run only slow tests.
*  **uses_db**: Run only tests that use the database.
*  **uses_web**: Run only tests that use the web/api server.
*  **--local-only**: Run only tests that use local data.

To run the test suite minimally against the core marvin package with local file access, use the following command:
::

   pytest --local-only -m "not uses_db and not uses_web and not slow" tests/

What is Marvin?
---------------

Marvin is a complete ecosystem designed for overcoming the challenge of
searching, accessing, and visualizing the MaNGA data. It consists of a
three-pronged approach of a web app, a python package, and an API. The
web app, Marvin-web, provides an easily accessible interface for
searching the MaNGA data and visual exploration of individual MaNGA
galaxies or of the entire sample. The python package, in particular
Marvin-tools, allows users to easily and efficiently interact with the
MaNGA data via local files, files retrieved from the `Science Archive
Server <https://sas.sdss.org>`__, or data directly grabbed from the
database. The tools come mainly in the form of convenience functions and
classes for interacting with the data. An additional tool is a powerful
query functionality that uses the API to query the MaNGA databases and
return the search results to your python session. Marvin-API is the
critical link that allows Marvin-tools and Marvin-web to interact with
the databases, which enables users to harness the statistical power of
the MaNGA data set.

Documentation
-------------

You can find the latest Marvin documentation
`here <http://sdss-marvin.readthedocs.io/en/latest/>`__.

Citation and Acknowledgements
-----------------------------

If you use Marvin for work/research presented in a publication (whether
directly, or as a dependency to another package), we ask that you cite
the `Marvin Software <https://zenodo.org/record/292632>`__ (BibTeX). We
provide the following as a standard acknowledgment you can use if there
is not a specific place to cite the DOI:

::

    *This research made use of Marvin, a core Python package and web framework for MaNGA data, developed by Brian Cherinka, José Sánchez-Gallego, Brett Andrews, and Joel Brownstein. (MaNGA Collaboration, 2018).*

Marvin's Bibtex entry to use:

::

    @ARTICLE{2019AJ....158...74C,
           author = {{Cherinka}, Brian and {Andrews}, Brett H. and
             {S{\'a}nchez-Gallego}, Jos{\'e} and {Brownstein}, Joel and
             {Argudo-Fern{\'a}ndez}, Mar{\'\i}a and {Blanton}, Michael and
             {Bundy}, Kevin and {Jones}, Amy and {Masters}, Karen and
             {Law}, David R. and {Rowlands}, Kate and {Weijmans}, Anne-Marie and
             {Westfall}, Kyle and {Yan}, Renbin},
            title = "{Marvin: A Tool Kit for Streamlined Access and Visualization of the SDSS-IV MaNGA Data Set}",
          journal = {\aj},
         keywords = {astronomical databases: miscellaneous, methods: data analysis, surveys, Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Astrophysics of Galaxies},
             year = 2019,
            month = aug,
           volume = {158},
           number = {2},
              eid = {74},
            pages = {74},
              doi = {10.3847/1538-3881/ab2634},
    archivePrefix = {arXiv},
           eprint = {1812.03833},
     primaryClass = {astro-ph.IM},
           adsurl = {https://ui.adsabs.harvard.edu/abs/2019AJ....158...74C},
          adsnote = {Provided by the SAO/NASA Astrophysics Data System}
    }

License
-------

Marvin is licensed under a 3-clause BSD style license - see the
``LICENSE.md`` file.

.. |Build Package| image:: https://github.com/sdss/marvin/actions/workflows/build.yml/badge.svg
   :target: https://github.com/sdss/marvin/actions/workflows/build.yml
.. |Run Test Suite| image:: https://github.com/sdss/marvin/actions/workflows/test.yml/badge.svg
   :target: https://github.com/sdss/marvin/actions/workflows/test.yml
.. |CodeCov| image:: https://codecov.io/gh/sdss/marvin/branch/main/graph/badge.svg
   :target: https://codecov.io/gh/sdss/marvin
.. |Coverage Status| image:: https://coveralls.io/repos/github/sdss/marvin/badge.svg?branch=master
   :target: https://coveralls.io/github/sdss/marvin?branch=master
.. |PyPI| image:: https://img.shields.io/pypi/v/sdss-marvin.svg
   :target: https://pypi.python.org/pypi/sdss-marvin
.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.596700.svg
   :target: https://doi.org/10.5281/zenodo.596700
.. |astropy| image:: http://img.shields.io/badge/powered%20by-AstroPy-orange.svg?style=flat
   :target: http://www.astropy.org/
.. |readthedocs| image:: https://readthedocs.org/projects/docs/badge/
   :target: http://sdss-marvin.readthedocs.io/en/latest/


[![Zenodo](https://zenodo.org/badge/29540972.svg)](https://zenodo.org/badge/latestdoi/29540972)
[![Tests](https://github.com/TIGRLab/datman/actions/workflows/run-tests.yml/badge.svg)](https://github.com/TIGRLab/datman/actions/workflows/run-tests.yml)
![Coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/DESm1th/cffe8a0a522de7697edacbf74f70a7e1/raw/datman-coverage.json)

[![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](#contributors-)

Datman is a tool for managing MRI studies and performing basic quality control
on MRI data. For more information, including installation instructions,
[view our docs here.](http://imaging-genetics.camh.ca/datman/)

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

Dawn E. Smith
💻 📖 🎨 🤔 🚇 🚧 💬 👀 ⚠ 📢

Michael Joseph
💻 🚧 👀

Jerry Jeyachandra
💻 🚧 👀

slimnsour
💻 🚧 👀

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!


[![Package version](https://badge.fury.io/py/eo-learn.svg)](https://pypi.org/project/eo-learn)
[![Conda version](https://img.shields.io/conda/vn/conda-forge/eo-learn.svg)](https://anaconda.org/conda-forge/eo-learn)
[![Supported Python versions](https://img.shields.io/pypi/pyversions/eo-learn.svg?style=flat-square)](https://pypi.org/project/eo-learn)
[![Build Status](https://github.com/sentinel-hub/eo-learn/actions/workflows/ci_action.yml/badge.svg?branch=master)](https://github.com/sentinel-hub/eo-learn/actions)
[![Docs status](https://readthedocs.org/projects/eo-learn/badge/?version=latest)](https://eo-learn.readthedocs.io)
[![License](https://img.shields.io/pypi/l/eo-learn.svg)](https://github.com/sentinel-hub/eo-learn/blob/master/LICENSE)
[![Overall downloads](http://pepy.tech/badge/eo-learn)](https://pepy.tech/project/eo-learn)
[![Last month downloads](https://pepy.tech/badge/eo-learn/month)](https://pepy.tech/project/eo-learn)
[![Docker pulls](https://img.shields.io/docker/pulls/sentinelhub/eolearn.svg)](https://hub.docker.com/r/sentinelhub/eolearn)
[![Code coverage](https://codecov.io/gh/sentinel-hub/eo-learn/branch/master/graph/badge.svg)](https://codecov.io/gh/sentinel-hub/eo-learn)
[![DOI](https://zenodo.org/badge/135559956.svg)](https://zenodo.org/badge/latestdoi/135559956)
<img align="right" src="docs/source/figures/eo-learn-logo.png" alt="" width="300"/>

# eo-learn

**eo-learn makes extraction of valuable information from satellite imagery easy.**

The availability of open Earth observation (EO) data through the Copernicus and Landsat programs represents an
unprecedented resource for many EO applications, ranging from ocean and land use and land cover monitoring,
disaster control, emergency services and humanitarian relief. Given the large amount of high spatial resolution
data at high revisit frequency, techniques able to automatically extract complex patterns in such _spatio-temporal_
data are needed.

**`eo-learn`** is a collection of open source Python packages that have been developed to seamlessly access and process
_spatio-temporal_ image sequences acquired by any satellite fleet in a timely and automatic manner. **`eo-learn`** is
easy to use, it's design modular, and encourages collaboration -- sharing and reusing of specific tasks in a typical
EO-value-extraction workflows, such as cloud masking, image co-registration, feature extraction, classification, etc. Everyone is free
to use any of the available tasks and is encouraged to improve the, develop new ones and share them with the rest of the community.

**`eo-learn`** makes extraction of valuable information from satellite imagery as easy as defining a sequence of operations to be performed on satellite imagery. Image below illustrates a processing chain that maps water in satellite imagery by thresholding the Normalised Difference Water Index in user specified region of interest.

![](docs/source/figures/eo-learn-illustration.png)

**`eo-learn`** _library acts as a bridge between Earth observation/Remote sensing field and Python ecosystem for data science and machine learning._ The library is written in Python and uses NumPy arrays to store and handle remote sensing data. Its aim is to make entry easier for non-experts to the field of remote sensing on one hand and bring the state-of-the-art tools for computer vision, machine learning, and deep learning existing in Python ecosystem to remote sensing experts.

## Package Overview

**`eo-learn`** package is structured into several modules according to different functionalities. Some modules contain extensions under the `extra` subfolder. Those modules typically require additional package dependencies which don't get installed by default, since they are usually very specific to the task.

The modules are:

- **`core`** - The main module which implements basic building blocks (`EOPatch`, `EOTask` and `EOWorkflow`) and commonly used functionalities.
- **`coregistration`** - Tasks which deal with image co-registration.
- **`features`** - A collection of utilities for extracting data properties and feature manipulation.
- **`geometry`** - Geometry-related tasks used for transformation and conversion between vector and raster data.
- **`io`** - Input/output tasks that deal with obtaining data from Sentinel Hub services or saving and loading data locally.
- **`mask`** - Tasks used for masking of data and calculation of cloud/snow/other masks.
- **`ml-tools`** - Various tools that can be used before or after the machine learning process.
- **`visualization`** - Visualization tools for the core elements of eo-learn.

## Installation

### Requirements

The package requires Python version **>=3.8**.

#### Linux

Before installing `eo-learn` on **Linux** it is recommended to install the following system libraries:

```bash
sudo apt-get install gcc libgdal-dev graphviz proj-bin libproj-dev libspatialindex-dev
```

#### Mac OS

Before installing `eo-learn` on `Mac OS` it is recommended to install the following system libraries with [Homebrew](https://brew.sh/):

```bash
brew install graphviz gcc gdal cmake spatialindex proj
```

#### Windows

Before installing `eo-learn` on **Windows** it is recommended to install the following packages from [Unofficial Windows wheels repository](https://www.lfd.uci.edu/~gohlke/pythonlibs/):

```bash
gdal
rasterio
shapely
fiona
```

### PyPI distribution

`eo-learn` is available on PyPI and can be installed with:

```bash
pip install eo-learn
```

For some modules there are extra dependencies available, related to specific tasks. These were kept separate in order to keep the `eo-learn` installation light. You can install these with, e.g.:

```bash
pip install "eo-learn[EXTRA]"
pip install "eo-learn[VISUALIZATION]"
```

The full list (including their descriptions) is available here:

- `RAY` for installing ray and its dependencies
- `ZARR` for installing the zarr functionality for chunked timestamp saving/loading
- `EXTRA` for installing interpolation- and clustering-specific dependencies, or for installing `s2cloudless` in cloud masking
- `VISUALIZATION` for using plotting libraries and utilities
- `FULL` for installing all dependencies described so far
- `DOCS` for developers, dependencies for building documentation
- `DEV` for developers, dependencies for testing and code contribution

### Conda Forge distribution

The package requires a Python environment **>=3.8**.

Thanks to the maintainers of the conda forge feedstock (@benhuff, @dcunn, @mwilson8, @oblute, @rluria14), `eo-learn` can
be installed using `conda-forge` as follows:

```bash
conda config --add channels conda-forge
conda install eo-learn
```

### Run with Docker

A docker image with the latest released version of `eo-learn` is available at [Docker Hub](https://hub.docker.com/r/sentinelhub/eolearn). It provides a full installation of `eo-learn` together with a Jupyter notebook environment. You can pull and run it with:

```bash
docker pull sentinelhub/eolearn:latest
docker run -p 8888:8888 sentinelhub/eolearn:latest
```

An extended version of the `latest` image additionally contains all example notebooks and data to get you started with `eo-learn`. Run it with:

```bash
docker pull sentinelhub/eolearn:latest-examples
docker run -p 8888:8888 sentinelhub/eolearn:latest-examples
```

Both docker images can also be built manually from GitHub repository:

```bash
docker build -f docker/eolearn.dockerfile . --tag=sentinelhub/eolearn:latest
docker build -f docker/eolearn-examples.dockerfile . --tag=sentinelhub/eolearn:latest-examples
```

## Documentation

For more information on the package content, visit [readthedocs](https://eo-learn.readthedocs.io/).

## More Examples

Examples and introductions to the package can be found [here](https://github.com/sentinel-hub/eo-learn/tree/master/examples). A larger collection of examples is available at the [`eo-learn-examples`](https://github.com/sentinel-hub/eo-learn-examples) repository. While the examples there are not always up-to-date they can be a great source of ideas.

In the past, `eo-learn` served as a collection of many useful tasks, originating from various contributors or projects. In order to keep `eo-learn` light and easy to maintain, we have decided to move these specific tasks to [`eo-learn-examples/extra-tasks`](https://github.com/sentinel-hub/eo-learn-examples/tree/main/extra-tasks), .

## Contributions

The list of all `eo-learn` contributors are listed in the [credits file](./CREDITS.md). If you would like to contribute to `eo-learn`, please check our [contribution guidelines](./CONTRIBUTING.md).

## Blog posts and papers

- [Introducing eo-learn](https://medium.com/sentinel-hub/introducing-eo-learn-ab37f2869f5c) (by Devis Peressutti)
- [Land Cover Classification with eo-learn: Part 1 - Mastering Satellite Image Data in an Open-Source Python Environment](https://medium.com/sentinel-hub/land-cover-classification-with-eo-learn-part-1-2471e8098195) (by Matic Lubej)
- [Land Cover Classification with eo-learn: Part 2 - Going from Data to Predictions in the Comfort of Your Laptop](https://medium.com/sentinel-hub/land-cover-classification-with-eo-learn-part-2-bd9aa86f8500) (by Matic Lubej)
- [Land Cover Classification with eo-learn: Part 3 - Pushing Beyond the Point of “Good Enough”](https://medium.com/sentinel-hub/land-cover-classification-with-eo-learn-part-3-c62ed9ecd405) (by Matic Lubej)
- [Innovations in satellite measurements for development](https://blogs.worldbank.org/opendata/innovations-satellite-measurements-development)
- [Use eo-learn with AWS SageMaker](https://medium.com/@drewbo19/use-eo-learn-with-aws-sagemaker-9420856aafb5) (by Drew Bollinger)
- [Spatio-Temporal Deep Learning: An Application to Land Cover Classification](https://www.researchgate.net/publication/333262625_Spatio-Temporal_Deep_Learning_An_Application_to_Land_Cover_Classification) (by Anze Zupanc)
- [Tree Cover Prediction with Deep Learning](https://medium.com/dataseries/tree-cover-prediction-with-deep-learning-afeb0b663966) (by Daniel Moraite)
- [NoRSC19 Workshop on eo-learn](https://github.com/sentinel-hub/norsc19-eo-learn-workshop)
- [Tracking a rapidly changing planet](https://medium.com/@developmentseed/tracking-a-rapidly-changing-planet-bc02efe3545d) (by Development Seed)
- [Land Cover Monitoring System](https://medium.com/sentinel-hub/land-cover-monitoring-system-84406e3019ae) (by Jovan Visnjic and Matej Aleksandrov)
- [eo-learn Webinar](https://www.youtube.com/watch?v=Rv-yK7Vbk4o) (by Anze Zupanc)
- [Cloud Masks at Your Service](https://medium.com/sentinel-hub/cloud-masks-at-your-service-6e5b2cb2ce8a)
- [ML examples for Common Agriculture Policy](https://medium.com/sentinel-hub/area-monitoring-concept-effc2c262583)
  - [High-Level Concept](https://medium.com/sentinel-hub/area-monitoring-concept-effc2c262583)
  - [Data Handling](https://medium.com/sentinel-hub/area-monitoring-data-handling-c255b215364f)
  - [Outlier detection](https://medium.com/sentinel-hub/area-monitoring-observation-outlier-detection-34f86b7cc63)
  - [Identifying built-up areas](https://medium.com/sentinel-hub/area-monitoring-how-to-train-a-binary-classifier-for-built-up-areas-7f2d7114ed1c)
  - [Similarity Score](https://medium.com/sentinel-hub/area-monitoring-similarity-score-72e5cbfb33b6)
  - [Bare Soil Marker](https://medium.com/sentinel-hub/area-monitoring-bare-soil-marker-608bc95712ae)
  - [Mowing Marker](https://medium.com/sentinel-hub/area-monitoring-mowing-marker-e99cff0c2d08)
  - [Pixel-level Mowing Marker](https://medium.com/sentinel-hub/area-monitoring-pixel-level-mowing-marker-968402a8579b)
  - [Crop Type Marker](https://medium.com/sentinel-hub/area-monitoring-crop-type-marker-1e70f672bf44)
  - [Homogeneity Marker](https://medium.com/sentinel-hub/area-monitoring-homogeneity-marker-742047b834dc)
  - [Parcel Boundary Detection](https://medium.com/sentinel-hub/parcel-boundary-detection-for-cap-2a316a77d2f6)
  - Land Cover Classification (still to come)
  - Minimum Agriculture Activity (still to come)
  - [Combining the Markers into Decisions](https://medium.com/sentinel-hub/area-monitoring-combining-markers-into-decisions-d74f70fe7721)
  - [The Challenge of Small Parcels](https://medium.com/sentinel-hub/area-monitoring-the-challenge-of-small-parcels-96121e169e5b)
  - [Traffic Light System](https://medium.com/sentinel-hub/area-monitoring-traffic-light-system-4a1348481c40)
  - [Expert Judgement Application](https://medium.com/sentinel-hub/expert-judgement-application-67a07f2feac4)
- [Scale-up your eo-learn workflow using Batch Processing API](https://medium.com/sentinel-hub/scale-up-your-eo-learn-workflow-using-batch-processing-api-d183b70ea237) (by Maxim Lamare)

## Questions and Issues

Feel free to ask questions about the package and its use cases at [Sentinel Hub forum](https://forum.sentinel-hub.com/) or raise an issue on [GitHub](https://github.com/sentinel-hub/eo-learn/issues).

You are welcome to send your feedback to the package authors, EO Research team, through any of [Sentinel Hub communication channel](https://sentinel-hub.com/develop/communication-channels).

## License

See [LICENSE](https://github.com/sentinel-hub/eo-learn/blob/master/LICENSE).

## Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements No. 776115, No. 101004112, No. 101059548 and No. 101086461.