# Django React Boilerplate

[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](code_of_conduct.md)
[![License: MIT](https://img.shields.io/github/license/vintasoftware/django-react-boilerplate.svg)](LICENSE.txt)

## About

A [Django](https://www.djangoproject.com/) project boilerplate/template with a multitude of state-of-the-art libraries and tools. If pairing Django with React is a possibility for your project or spinoff, this is the best solution available. Save time with tools like:

-   [React](https://react.dev/), for building interactive UIs
-   [TypeScript](https://www.typescriptlang.org/), for static type checking
-   [Poetry](https://python-poetry.org/), for managing the environment and its dependencies
-   [django-js-reverse](https://github.com/vintasoftware/django-js-reverse), for generating URLs on JS
-   [React Bootstrap](https://react-bootstrap.github.io/), for responsive styling
-   [Webpack](https://webpack.js.org/), for bundling static assets
-   [Celery](https://docs.celeryq.dev/en/stable/), for background worker tasks
-   [WhiteNoise](https://whitenoise.readthedocs.io/en/stable/) with [brotlipy](https://github.com/python-hyper/brotlicffi), for efficient static files serving
-   [ruff](https://github.com/astral-sh/ruff) and [ESLint](https://eslint.org/) with [pre-commit](https://pre-commit.com/) for automated quality assurance (does not replace proper testing!)

For continuous integration, a [Github Action](https://github.com/features/actions) configuration `.github/workflows/main.yml` is included.

Also, includes a Render.com `render.yaml` and a working Django `production.py` settings, enabling easy deployments with ['Deploy to Render' button](https://render.com/docs/deploy-to-render). The `render.yaml` includes the following:

-   PostgreSQL, for DB
-   Redis, for Celery

## Features Catalogue

### Frontend

-   `react` for building interactive UIs
-   `react-dom` for rendering the UI
-   `react-router` for page navigation
-   `webpack` for bundling static assets
-   `webpack-bundle-tracker` for providing the bundled assets to Django
-   Styling
    -   `bootstrap` for providing responsive stylesheets
    -   `react-bootstrap` for providing components built on top of Bootstrap CSS without using plugins
    -   `sass` for providing compatibility with SCSS files
-   State management and backend integration
    -   `axios` for performing asynchronous calls
    -   `cookie` for easy integration with Django using the `csrftoken` cookie
    -   `openapi-ts` for generating TypeScript client API code from the backend OpenAPI schema
    -   `history` for providing browser history to Connected React Router
-   Utilities
    -   `lodash` for general utility functions
    -   `classnames` for easy working with complex CSS class names on components
    -   `react-refresh` for improving QoL while developing through automatic browser refreshing

### Backend

-   `django` for building backend logic using Python
-   `djangorestframework` for building a REST API on top of Django
-   `drf-spectacular` for generating an OpenAPI schema for the Django REST API
-   `django-webpack-loader` for rendering the bundled frontend assets
-   `django-js-reverse` for easy handling of Django URLs on JS
-   `django-upgrade` for automatically upgrading Django code to the target version on pre-commit
-   `django-guid` for adding a unique correlation ID to log messages from Django requests
-   `psycopg` for using PostgreSQL database
-   `sentry-sdk` for error monitoring
-   `python-decouple` for reading environment variables on settings files
-   `celery` for background worker tasks
-   `django-csp` for setting the draft security HTTP header Content-Security-Policy
-   `django-permissions-policy` for setting the draft security HTTP header Permissions-Policy
-   `django-defender` for blocking brute force attacks against login
-   `whitenoise` and `brotlipy` for serving static assets

## Share your project!

Several people have leveraged our boilerplate to start spinoffs or to boost their efforts in the challenging pursuit of securing funding. Starting with a solid foundation allows you to create more resilient products and focus on what really matters: discovering and delivering value to your customers. If you are one of those people, we're eager to help you even more! We can spread the word about your project across our social media platforms, giving you access to a broader audience.

Send us an email at contact@vintasoftware.com telling us a bit more about how our boilerplate helped you boost your project.

## Project bootstrap [![main](https://github.com/vintasoftware/django-react-boilerplate/actions/workflows/main.yml/badge.svg)](https://github.com/vintasoftware/django-react-boilerplate/actions/workflows/main.yml) [![Known Vulnerabilities](https://snyk.io/test/github/vintasoftware/django-react-boilerplate/badge.svg)](https://snyk.io/test/github/vintasoftware/django-react-boilerplate)

-   [ ] Make sure you have Python 3.12 installed
-   [ ] Install Django with `pip install django`, to have the `django-admin` command available
-   [ ] Open the command line and go to the directory you want to start your project in
-   [ ] Start your project using (replace `project_name` with your project name and remove the curly braces):
    ```
    django-admin startproject {{project_name}} --extension py,json,yml,yaml,toml --name Dockerfile,README.md,.env.example,.gitignore,Makefile --template=https://github.com/vintasoftware/django-react-boilerplate/archive/refs/heads/main.zip
    ```
    Alternatively, you may start the project in the current directory by placing a `.` right after the project name, using the following command:
    ```
    django-admin startproject {{project_name}} . --extension py,json,yml,yaml,toml --name Dockerfile,README.md,.env.example,.gitignore,Makefile --template=https://github.com/vintasoftware/django-react-boilerplate/archive/refs/heads/main.zip
    ```
In the next steps, always remember to replace {{project_name}} with your project's name (in case it isn't yet):
-   [ ] Above: don't forget the `--extension` and `--name` params!
-   [ ] Go into project's root directory: `cd {{project_name}}`
-   [ ] Change the first line of README to the name of the project
-   [ ] Add an email address to the `ADMINS` settings variable in `{{project_name}}/backend/{{project_name}}/settings/base.py`
-   [ ] Change the `SERVER_EMAIL` to the email address used to send e-mails in `{{project_name}}/backend/{{project_name}}/settings/production.py`

After completing ALL of the above, remove this `Project bootstrap` section from the project README. Then follow `Running` below.

## Running

### Tools

-   Setup [editorconfig](http://editorconfig.org/), [ruff](https://github.com/astral-sh/ruff) and [ESLint](http://eslint.org/) in the text editor you will use to develop.

### Setup

-   Do the following:
    -   Create a git-untracked `local.py` settings file:
        `cp backend/{{project_name}}/settings/local.py.example backend/{{project_name}}/settings/local.py`
    -   Create a git-untracked `.env.example` file:
        `cp backend/.env.example backend/.env`

### If you are using Docker:

-   Open the `backend/.env` file on a text editor and uncomment the line `DATABASE_URL=postgres://{{project_name}}:password@db:5432/{{project_name}}`
-   Open a new command line window and go to the project's directory
-   Run the initial setup:
    `make docker_setup`
-   Create the migrations for `users` app:
    `make docker_makemigrations`
-   Run the migrations:
    `make docker_migrate`
-   Run the project:
    `make docker_up`
-   Access `http://localhost:8000` on your browser and the project should be running there
    -   When you run `make docker_up`, some containers are spinned up (frontend, backend, database, etc) and each one will be running on a different port
    -   The container with the React app uses port 3000. However, if you try accessing it on your browser, the app won't appear there and you'll probably see a blank page with the "Cannot GET /" error
    -   This happens because the container responsible for displaying the whole application is the Django app one (running on port 8000). The frontend container is responsible for providing a bundle with its assets for [django-webpack-loader](https://github.com/django-webpack/django-webpack-loader) to consume and render them on a Django template
-   To access the logs for each service, run:
    `make docker_logs <service name>` (either `backend`, `frontend`, etc)
-   To stop the project, run:
    `make docker_down`

#### Adding new dependencies

-   Open a new command line window and go to the project's directory
-   Update the dependencies management files by performing any number of the following steps:
    -   To add a new **frontend** dependency, run `npm install <package name> --save`
        > The above command will update your `package.json`, but won't make the change effective inside the container yet
    -   To add a new **backend** dependency, run `docker compose run --rm backend bash` to open an interactive shell and then run `poetry add {dependency}` to add the dependency. If the dependency should be only available for development user append `-G dev` to the command.
    -   After updating the desired file(s), run `make docker_update_dependencies` to update the containers with the new dependencies
        > The above command will stop and re-build the containers in order to make the new dependencies effective

### If you are not using Docker:

#### Setup the backend app

-   Open the `backend/.env` file on a text editor and do one of the following:
    -   If you wish to use SQLite locally, uncomment the line `DATABASE_URL=sqlite:///backend/db.sqlite3`
    -   If you wish to use PostgreSQL locally, uncomment and edit the line `DATABASE_URL=postgres://{{project_name}}:password@db:5432/{{project_name}}` in order to make it correctly point to your database URL
        -   The url format is the following: `postgres://USER:PASSWORD@HOST:PORT/NAME`
    -   If you wish to use another database engine locally, add a new `DATABASE_URL` setting for the database you wish to use
        -   Please refer to [dj-database-url](https://github.com/jazzband/dj-database-url#url-schema) on how to configure `DATABASE_URL` for commonly used engines
-   Open a new command line window and go to the project's directory
-   Run `poetry install`

#### Run the backend app

-   Go to the `backend` directory
-   Create the migrations for `users` app:
    `poetry run python manage.py makemigrations`
-   Run the migrations:
    `poetry run python manage.py migrate`
-   Generate the OpenAPI schema:
    `poetry run python manage.py spectacular --color --file schema.yml`
-   Run the project:
    `poetry run python manage.py runserver`

#### Setup and run the frontend app

-   Open a new command line window and go to the project's directory
-   `npm install`
-   `npm run openapi-ts`
    -   This is used to generate the TypeScript client API code from the backend OpenAPI schema
-   `npm run dev`
    -   This is used to serve the frontend assets to be consumed by [django-webpack-loader](https://github.com/django-webpack/django-webpack-loader) and not to run the React application as usual, so don't worry if you try to check what's running on port 3000 and see an error on your browser
-   Open a browser and go to `http://localhost:8000` to see the project running

#### Setup Celery

-   `poetry run celery --app={{project_name}} worker --loglevel=info`

#### Setup Redis

-   Ensure that Redis is already installed on your system. Once confirmed, run `redis-server --port 6379` to start the Redis server.
-   If you wish to use Redis for Celery, you need to set the `CELERY_BROKER_URL` environment variable in the `backend/.env` file to `redis://localhost:6379/0`.
    -   The `/0` at the end of the URL specifies the database number on the Redis server. Redis uses a zero-based numbering system for databases, so `0` is the first database. If you don't specify a database number, Redis will use the first database by default.
    -   Note: Prefer RabbitMQ over Redis for Broker, mainly because RabbitMQ doesn't need visibility timeout. See [Recommended Celery Django settings for reliability](https://gist.github.com/fjsj/da41321ac96cf28a96235cb20e7236f6).

#### Mailhog

-   For development, we use Mailhog to test our e-mail workflows, since it allows us to inspect the messages to validate they're correctly built
    -   Docker users already have it setup and running once they start the project
    -   For non-Docker users, please have a look [here](https://github.com/mailhog/MailHog#installation) for instructions on how to setup Mailhog on specific environments
        > The project expects Mailhog SMTP server to be running on port 1025, you may alter that by changing `EMAIL_PORT` on settings

### Testing

`make test`

Will run django tests using `--keepdb` and `--parallel`. You may pass a path to the desired test module in the make command. E.g.:

`make test someapp.tests.test_views`

### Adding new pypi libs

To add a new **backend** dependency, run `poetry add {dependency}`. If the dependency should be only available for development user append `-G dev` to the command.

### API Schema and Client generation

We use the [`DRF-Spectacular`](https://drf-spectacular.readthedocs.io/en/latest/readme.html) tool to generate an OpenAPI schema from our Django Rest Framework API. The OpenAPI schema serves as the backbone for generating client code, creating comprehensive API documentation, and more.

The API documentation pages are accessible at `http://localhost:8000/api/schema/swagger-ui/` or `http://localhost:8000/api/schema/redoc/`.

> [!IMPORTANT]
> Anytime a view is created, updated, or removed, the schema must be updated to reflect the changes. Failing to do so can lead to outdated client code or documentation.
>
> To update the schema, run:
> - If you are using Docker: `make docker_backend_update_schema`
> - If you are not using Docker: `poetry run python manage.py spectacular --color --file schema.yml`

We use the [`openapi-ts`](https://heyapi.vercel.app/openapi-ts/get-started.html) tool to generate TypeScript client code from the OpenAPI schema. The generated client code is used to interact with the API in a type-safe manner.

> [!IMPORTANT]
> Anytime the API schema is updated, the client code must be regenerated to reflect the changes. Failing to do so can lead to type errors in the client code.
>
> To update the client code, run:
> - If you are using Docker: `make docker_frontend_update_api`
> - If you are not using Docker: `npm run openapi-ts`

> [!NOTE]
> If `pre-commit` is properly enabled, it will automatically update both schema and client before each commit whenever necessary.

## Github Actions

To enable Continuous Integration through Github Actions, we provide a `proj_main.yml` file. To connect it to Github you need to rename it to `main.yml` and move it to the `.github/workflows/` directory.

You can do it with the following commands:

```bash
mkdir -p .github/workflows
mv proj_main.yml .github/workflows/main.yml
```

## Production Deployment

### Setup

This project comes with an `render.yaml` file, which can be used to create an app on Render.com from a GitHub repository.

Before deploying, please make sure you've generated an up-to-date `poetry.lock` file containing the Python dependencies. This is necessary even if you've used Docker for local runs. Do so by following [these instructions](#setup-the-backend-app).

After setting up the project, you can init a repository and push it on GitHub. If your repository is public, you can use the following button:

[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy)

If you are in a private repository, access the following link replacing `$YOUR_REPOSITORY_URL$` with your repository link.

-   `https://render.com/deploy?repo=$YOUR_REPOSITORY_URL$`

Keep reading to learn how to configure the prompted environment variables.

#### `ALLOWED_HOSTS`

Chances are your project name isn't unique in Render, and you'll get a randomized suffix as your full app URL like: `https://{{project_name}}-a1b2.onrender.com`.

But this will only happen after the first deploy, so you are not able to properly fill `ALLOWED_HOSTS` yet. Simply set it to `*` then fix it later to something like `{{project_name}}-a1b2.onrender.com` and your domain name like `example.org`.

#### `ENABLE_DJANGO_COLLECTSTATIC`

Default is 1, meaning the build script will run collectstatic during deploys.

#### `AUTO_MIGRATE`

Default is 1, meaning the build script will run collectstatic during deploys.

### Build script

By default, the project will always run the `render_build.sh` script during deployments. This script does the following:

1.  Build the frontend
2.  Build the backend
3.  Run Django checks
4.  Run `collectstatic`
5.  Run Django migrations
6.  Push frontend source maps to Sentry

### Celery

As there aren't free plans for Workers in Render.com, the configuration for Celery workers/beat will be commented by default in the `render.yaml`. This means celery won't be available by default.

Uncommenting the worker configuration lines on `render.yaml` will imply in costs.

### SendGrid

To enable sending emails from your application you'll need to have a valid SendGrid account and also a valid verified sender identity. After finishing the validation process you'll be able to generate the API credentials and define the `SENDGRID_USERNAME` and `SENDGRID_PASSWORD` environment variables on Render.com.

These variables are required for your application to work on Render.com since it's pre-configured to automatically email admins when the application is unable to handle errors gracefully.

### Media storage

Media files integration with S3 or similar is not supported yet. Please feel free to contribute!

### Sentry

[Sentry](https://sentry.io) is already set up on the project. For production, add `SENTRY_DSN` environment variable on Render.com, with your Sentry DSN as the value.

You can test your Sentry configuration by deploying the boilerplate with the sample page and clicking on the corresponding button.

### Sentry source maps for JS files

The `render_build.sh` script has a step to push Javascript source maps to Sentry, however some environment variables need to be set on Render.com.

The environment variables that need to be set are:

-   `SENTRY_ORG` - Name of the Sentry Organization that owns your Sentry Project.
-   `SENTRY_PROJECT_NAME` - Name of the Sentry Project.
-   `SENTRY_API_KEY` - Sentry API key that needs to be generated on Sentry. [You can find or create authentication tokens within Sentry](https://sentry.io/api/).

After enabling dyno metadata and setting the environment variables, your next Render.com Deploys will create a release on Sentry where the release name is the commit SHA, and it will push the source maps to it.

## Linting

-   At pre-commit time (see below)
-   Manually with `poetry run ruff` and `npm run lint` on project root.
-   During development with an editor compatible with ruff and ESLint.

## Pre-commit hooks

### If you are using Docker:

-   Not supported yet. Please feel free to contribute!

### If you are not using Docker:

-   On project root, run `poetry run pre-commit install` to enable the hook into your git repo. The hook will run automatically for each commit.

## Opinionated Settings

Some settings defaults were decided based on Vinta's experiences. Here's the rationale behind them:

### `DATABASES["default"]["ATOMIC_REQUESTS"] = True`

- Using atomic requests in production prevents several database consistency issues. Check [Django docs for more details](https://docs.djangoproject.com/en/5.0/topics/db/transactions/#tying-transactions-to-http-requests).

- **Important:** When you are queueing a new Celery task directly from a Django view, particularly with little or no delay/ETA, it is essential to use `transaction.on_commit(lambda: my_task.delay())`. This ensures that the task is only queued after the associated database transaction has been successfully committed.
  - If `transaction.on_commit` is not utilized, or if a significant delay is not set, you risk encountering race conditions. In such scenarios, the Celery task might execute before the completion of the request's transaction. This can lead to inconsistencies and unexpected behavior, as the task might operate on a database state that does not yet reflect the changes made in the transaction. Read more about this problem on [this article](https://www.vinta.com.br/blog/database-concurrency-in-django-the-right-way).

### `CELERY_TASK_ACKS_LATE = True`

- We believe Celery tasks should be idempotent. So for us it's safe to set `CELERY_TASK_ACKS_LATE = True` to ensure tasks will be re-queued after a worker failure. Check Celery docs on ["Should I use retry or acks_late?"](https://docs.celeryq.dev/en/stable/faq.html#faq-acks-late-vs-retry) for more info.

### Django-CSP

Django-CSP helps implementing Content Security Policy (CSP) in Django projects to mitigate cross-site scripting (XSS) attacks by declaring which dynamic resources are allowed to load.

In this project, we have defined several CSP settings that define the sources from which different types of resources can be loaded. If you need to load external images, fonts, or other resources, you will need to add the sources to the corresponding CSP settings. For example:
- To load scripts from an external source, such as https://browser.sentry-cdn.com, you would add this source to `CSP_SCRIPT_SRC`.
- To load images from an external source, such as https://example.com, you would add this source to `CSP_IMG_SRC`.

Please note that you should only add trusted sources to these settings to maintain the security of your site. For more details, please refer to the [Django-CSP documentation](https://django-csp.readthedocs.io/en/latest/).

## Contributing

If you wish to contribute to this project, please first discuss the change you wish to make via an [issue](https://github.com/vintasoftware/django-react-boilerplate/issues).

Check our [contributing guide](https://github.com/vintasoftware/django-react-boilerplate/blob/main/CONTRIBUTING.md) to learn more about our development process and how you can test your changes to the boilerplate.

## Commercial Support

[![alt text](https://avatars2.githubusercontent.com/u/5529080?s=80&v=4 "Vinta Logo")](https://www.vinta.com.br/)

This project is maintained by [Vinta Software](https://www.vinta.com.br/) and is used in products of Vinta's clients. We are always looking for exciting work! If you need any commercial support, feel free to get in touch: contact@vinta.com.br


+---------------+-----------------+
| Service       | Master branch   |
+===============+=================+
| GitHub        | |ghactions_ci|  |
+---------------+-----------------+
| Appveyor      | |appveyor_ci|   |
+---------------+-----------------+
| Read the Docs | |read_the_docs| |
+---------------+-----------------+

.. |ghactions_ci| image:: https://github.com/PyWavelets/pywt/actions/workflows/tests.yml/badge.svg?branch=main
   :alt: Build Status
   :target: https://github.com/PyWavelets/pywt/actions/workflows/tests.yml?query=branch%3Amain

.. |appveyor_ci| image:: https://ci.appveyor.com/api/projects/status/github/PyWavelets/pywt
   :align: middle
   :target: https://ci.appveyor.com/project/PyWavelets/pywt
   :alt: Appveyor Status

.. |read_the_docs| image:: https://readthedocs.org/projects/pywavelets/badge/?version=latest
   :align: middle
   :target: https://pywavelets.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status

PyWavelets
==========

.. contents::

What is PyWavelets
------------------

PyWavelets is a free Open Source library for wavelet transforms in Python.
Wavelets are mathematical basis functions that are localized in both time and
frequency.  Wavelet transforms are time-frequency transforms employing
wavelets.  They are similar to Fourier transforms, the difference being that
Fourier transforms are localized only in frequency instead of in time and
frequency.

The main features of PyWavelets are:

  * 1D, 2D and nD Forward and Inverse Discrete Wavelet Transform (DWT and IDWT)
  * 1D, 2D and nD Multilevel DWT and IDWT
  * 1D and 2D Stationary Wavelet Transform (Undecimated Wavelet Transform)
  * 1D and 2D Wavelet Packet decomposition and reconstruction
  * 1D Continuous Wavelet Transform
  * Computing Approximations of wavelet and scaling functions
  * Over 100 `built-in wavelet filters`_ and support for custom wavelets
  * Single and double precision calculations
  * Real and complex calculations
  * Results compatible with Matlab Wavelet Toolbox (TM)

Documentation
-------------

Documentation with detailed examples and links to more resources is available
online at http://pywavelets.readthedocs.org.

For more usage examples see the `demo`_ directory in the source package.

Installation
------------

PyWavelets supports `Python`_ >=3.10, and is only dependent on `NumPy`_
(supported versions are currently ``>= 1.23.0``). To pass all of the tests,
`Matplotlib`_ is also required. `SciPy`_ is also an optional dependency. When
present, FFT-based continuous wavelet transforms will use FFTs from SciPy
rather than NumPy.

There are binary wheels for Intel Linux, Windows and macOS / OSX on PyPi.  If
you are on one of these platforms, you should get a binary (precompiled)
installation with::

    pip install PyWavelets

Users of the Anaconda_ Python distribution may wish to obtain pre-built
Windows, Intel Linux or macOS / OSX binaries from the conda-forge channel.
This can be done via::

    conda install -c conda-forge pywavelets

Several Linux distributions have their own packages for PyWavelets, but these
tend to be moderately out of date.  Query your Linux package manager tool for
``python-pywavelets``, ``python-wavelets``, ``python-pywt`` or a similar
package name.

If you want or need to install from source, you will need a working C compiler
(any common one will work) and a recent version of `Cython`_.  Navigate to the
PyWavelets source code directory (containing ``pyproject.toml``) and type::

    pip install .

The most recent *development* version can be found on GitHub at
https://github.com/PyWavelets/pywt.

The latest release, including source and binary packages for Intel Linux,
macOS and Windows, is available for download from the `Python Package Index`_.
You can find source releases at the `Releases Page`_.

State of development & Contributing
-----------------------------------

PyWavelets started in 2006 as an academic project for a master thesis
on `Analysis and Classification of Medical Signals using Wavelet Transforms`
and was maintained until 2012 by its `original developer`_.  In 2013
maintenance was taken over in a `new repo <https://github.com/PyWavelets/pywt>`_)
by a larger development team - a move supported by the original developer.
The repo move doesn't mean that this is a fork - the package continues to be
developed under the name "PyWavelets", and released on PyPi and Github (see
`this issue <https://github.com/nigma/pywt/issues/13>`_ for the discussion
where that was decided).

All contributions including bug reports, bug fixes, new feature implementations
and documentation improvements are welcome.  Moreover, developers with an
interest in PyWavelets are very welcome to join the development team!

As of 2019, PyWavelets development is supported in part by Tidelift.
`Help support PyWavelets with the Tidelift Subscription <https://tidelift.com/subscription/pkg/pypi-pywavelets?utm_source=pypi-pywavelets&utm_medium=referral&utm_campaign=readme>`_

Contact
-------

Use `GitHub Issues`_ or the `mailing list`_ to post your comments or questions.

**Report a security vulnerability:** https://tidelift.com/security

License
-------

PyWavelets is a free Open Source software released under the MIT license.

If you wish to cite PyWavelets in a publication, please use the following
JOSS publication.

.. image:: http://joss.theoj.org/papers/10.21105/joss.01237/status.svg
   :target: https://doi.org/10.21105/joss.01237

Specific releases can also be cited via Zenodo. The DOI below will correspond
to the most recent release. DOIs for past versions can be found by following
the link in the badge below to Zenodo:

.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1407171.svg
   :target: https://doi.org/10.5281/zenodo.1407171

.. _built-in wavelet filters: http://wavelets.pybytes.com/
.. _Cython: http://cython.org/
.. _demo: https://github.com/PyWavelets/pywt/tree/main/demo
.. _Anaconda: https://www.continuum.io
.. _GitHub: https://github.com/PyWavelets/pywt
.. _GitHub Issues: https://github.com/PyWavelets/pywt/issues
.. _NumPy: https://www.numpy.org
.. _SciPy: https://www.scipy.org
.. _original developer: http://en.ig.ma
.. _Python: http://python.org/
.. _Python Package Index: http://pypi.python.org/pypi/PyWavelets/
.. _mailing list: http://groups.google.com/group/pywavelets
.. _Releases Page: https://github.com/PyWavelets/pywt/releases
.. _Matplotlib: http://matplotlib.org


=======
udiskie
=======

|Version| |License| |Translations|

*udiskie* is a udisks2_ front-end that allows to manage removable media such
as CDs or flash drives from userspace.

|Screenshot|

Its features include:

- automount removable media
- notifications
- tray icon
- command line tools for manual un-/mounting
- LUKS encrypted devices
- unlocking with keyfiles (requires udisks 2.6.4)
- loop devices (mounting iso archives)
- password caching (requires python keyutils 0.3)

All features can be individually enabled or disabled.

**NOTE:** support for python2 and udisks1 have been removed. If you need a
version of udiskie that supports python2, please check out the ``1.7.X``
releases or the ``maint-1.7`` branch.

.. _udisks2: https://www.freedesktop.org/wiki/Software/udisks

- `Documentation`_

  - Usage_
  - Installation_
  - `Debug Info`_
  - Troubleshooting_
  - FAQ_

- `Man page`_
- `Source Code`_
- `Latest Release`_
- `Issue Tracker`_

.. _Documentation:      https://github.com/coldfix/udiskie/wiki
.. _Usage:              https://github.com/coldfix/udiskie/wiki/Usage
.. _Installation:       https://github.com/coldfix/udiskie/wiki/Installation
.. _Debug Info:         https://github.com/coldfix/udiskie/wiki/Debug-Info
.. _Troubleshooting:    https://github.com/coldfix/udiskie/wiki/Troubleshooting
.. _FAQ:                https://github.com/coldfix/udiskie/wiki/FAQ

.. _Man Page:       https://raw.githubusercontent.com/coldfix/udiskie/master/doc/udiskie.8.txt
.. _Source Code:    https://github.com/coldfix/udiskie
.. _Latest Release: https://pypi.python.org/pypi/udiskie/
.. _Issue Tracker:  https://github.com/coldfix/udiskie/issues
.. _Roadmap:        https://github.com/coldfix/udiskie/blob/master/HACKING.rst#roadmap

.. Badges:

.. |Version| image::   https://img.shields.io/pypi/v/udiskie.svg
   :target:            https://pypi.python.org/pypi/udiskie
   :alt:               Version

.. |License| image::   https://img.shields.io/pypi/l/udiskie.svg
   :target:            https://github.com/coldfix/udiskie/blob/master/COPYING
   :alt:               License: MIT

.. |Translations| image:: http://weblate.coldfix.de/widgets/udiskie/-/udiskie/svg-badge.svg
   :target:               http://weblate.coldfix.de/engage/udiskie/
   :alt:                  Translations

.. |Screenshot| image:: https://raw.githubusercontent.com/coldfix/udiskie/master/screenshot.png
   :target:             https://raw.githubusercontent.com/coldfix/udiskie/master/screenshot.png
   :alt:                Screenshot


m.css
#####

*A no-nonsense, no-JavaScript CSS framework, site and documentation theme for
content-oriented websites*

.. image:: https://badges.gitter.im/mosra/m.css.svg
    :alt: Join the chat at https://gitter.im/mosra/m.css
    :target: https://gitter.im/mosra/m.css?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. image:: https://circleci.com/gh/mosra/m.css.svg?style=shield
    :target: https://circleci.com/gh/mosra/m.css
.. image:: https://codecov.io/gh/mosra/m.css/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/mosra/m.css
.. image:: https://img.shields.io/badge/License-MIT-green.svg
    :alt: MIT License
    :target: https://opensource.org/licenses/MIT

Do you *hate* contemporary web development like I do? Do you also feel that
it's not right for a web page to take *seconds* and *megabytes* to render? Do
you want to write beautiful content but *can't* because the usual CMS tools
make your blood boil and so you rather stay silent? Well, I have something for
you.

Project website: https://mcss.mosra.cz

Pure CSS and HTML
=================

Everything you need is a few kilobytes of compressed CSS. This framework has
exactly 0 bytes of JavaScript because *nobody actually needs it*. Even for
responsive websites.

`Get the CSS <https://mcss.mosra.cz/css/>`_

Designed for content
====================

If you just want to write content with beautiful typography, you don't need
forms, progressbars, popups, dropdowns or other cruft. You want fast iteration
times.

`Use it with Pelican <https://mcss.mosra.cz/pelican/>`_

Authoring made easy
===================

Code snippets, math, linking to docs, presenting photography in a beautiful
way? Or making a complex page without even needing to touch HTML? Everything is
possible.

`Get Pelican plugins <https://mcss.mosra.cz/plugins/>`_

-------

*Still not convinced?* Head over to a `detailed explanation <https://mcss.mosra.cz/why/>`_
of this project goals and design decisions.

CONTRIBUTING
============

Head over to the `contribution guide <CONTRIBUTING.rst>`_ to see how to make
changes, verify the results and submit patches.

CONTACT
=======

Want to learn more about m.css? Found a bug or want to share an awesome idea?
Feel free to visit the project website or contact the author at:

-   Website --- https://mcss.mosra.cz
-   GitHub --- https://github.com/mosra/m.css
-   Gitter --- https://gitter.im/mosra/m.css
-   Twitter --- https://twitter.com/czmosra
-   E-mail --- mosra@centrum.cz

CREDITS
=======

See the `CREDITS.rst <CREDITS.rst>`_ file for details about contributors and
third-party code involved in this project. Big thanks to everyone involved!

LICENSE
=======

m.css is licensed under MIT/Expat license, see `COPYING <COPYING>`_ file for
details.


A Jupyter kernel base class in Python which includes core magic functions (including help, command and file path completion, parallel and distributed processing, downloads, and much more).

.. image:: https://badge.fury.io/py/metakernel.png/
    :target: http://badge.fury.io/py/metakernel

.. image:: https://coveralls.io/repos/Calysto/metakernel/badge.png?branch=main
  :target: https://coveralls.io/r/Calysto/metakernel

.. image:: https://github.com/Calysto/metakernel/actions/workflows/tests.yml/badge.svg?query=branch%3Amain++
  :target: https://github.com/Calysto/metakernel/actions/workflows/tests.yml

.. image:: https://anaconda.org/conda-forge/metakernel/badges/version.svg
    :target: https://anaconda.org/conda-forge/metakernel

.. image:: https://anaconda.org/conda-forge/metakernel/badges/downloads.svg
    :target: https://anaconda.org/conda-forge/metakernel

See Jupyter's docs on `wrapper kernels
<http://jupyter-client.readthedocs.io/en/stable/wrapperkernels.html>`_.

Additional magics can be installed within the new kernel package under a `magics` subpackage.

Features
-------------
- Basic set of line and cell magics for all kernels.
    - Python magic for accessing python interpreter.
    - Run kernels in parallel.
    - Shell magics.
    - Classroom management magics.
- Tab completion for magics and file paths.
- Help for magics using ? or Shift+Tab.
- Plot magic for setting default plot behavior.

Kernels based on Metakernel
---------------------------

- matlab_kernel, https://github.com/Calysto/matlab_kernel
- octave_kernel, https://github.com/Calysto/octave_kernel
- calysto_scheme, https://github.com/Calysto/calysto_scheme
- calysto_processing, https://github.com/Calysto/calysto_processing
- java9_kernel, https://github.com/Bachmann1234/java9_kernel
- xonsh_kernel, https://github.com/Calysto/xonsh_kernel
- calysto_hy, https://github.com/Calysto/calysto_hy
- gnuplot_kernel, https://github.com/has2k1/gnuplot_kernel
- spylon_kernel, https://github.com/mariusvniekerk/spylon-kernel
- wolfram_kernel, https://github.com/mmatera/iwolfram
- sas_kernel, https://github.com/sassoftware/sas_kernel
- pysysh_kernel, https://github.com/Jaesin/psysh_kernel
- calysto_bash, https://github.com/Calysto/calysto_bash

... and many others.

Installation
----------------
You can install Metakernel through ``pip``:

.. code:: bash

 pip install metakernel --upgrade

Installing `metakernel` from the `conda-forge` channel can be achieved by adding `conda-forge` to your channels with:

.. code:: bash

 conda config --add channels conda-forge

Once the `conda-forge` channel has been enabled, `metakernel` can be installed with:

.. code:: bash

 conda install metakernel

It is possible to list all of the versions of `metakernel` available on your platform with:

.. code:: bash

 conda search metakernel --channel conda-forge

Use MetaKernel Magics in IPython
--------------------------------

Although MetaKernel is a system for building new kernels, you can use a subset of the magics in the IPython kernel.

.. code:: python

 from metakernel import register_ipython_magics
 register_ipython_magics()

Put the following in your (or a system-wide) ``ipython_config.py`` file:

.. code:: python

 # /etc/ipython/ipython_config.py
 c = get_config()
 startup = [
    'from metakernel import register_ipython_magics',
    'register_ipython_magics()',
 ]
 c.InteractiveShellApp.exec_lines = startup

Use MetaKernel Languages in Parallel

To use a MetaKernel language in parallel, do the following:

1. Make sure that the Python module `ipyparallel` is installed. In the shell, type:

.. code:: bash

  pip install ipyparallel

2. To enable the extension in the notebook, in the shell, type:

.. code:: bash

  ipcluster nbextension enable

3. To start up a cluster, with 10 nodes, on a local IP address, in the shell, type:

.. code:: bash

  ipcluster start --n=10 --ip=192.168.1.108

4. Initialize the code to use the 10 nodes, inside the notebook from a host kernel ``MODULE`` and ``CLASSNAME`` (can be any metakernel kernel):

.. code:: bash

  %parallel MODULE CLASSNAME

For example:

.. code:: bash

  %parallel calysto_scheme CalystoScheme

5. Run code in parallel, inside the notebook, type:

Execute a single line, in parallel:

.. code:: bash

  %px (+ 1 1)

Or execute the entire cell, in parallel:

.. code:: bash

  %%px
  (* cluster_rank cluster_rank)

Results come back in a Python list (Scheme vector), in ``cluster_rank`` order. (This will be a JSON representation in the future).

Therefore, the above would produce the result:

.. code:: bash

  #10(0 1 4 9 16 25 36 49 64 81)

You can get the results back in any of the parallel magics (``%px``, ``%%px``, or ``%pmap``) in the host kernel by accessing the variable ``_`` (single underscore), or by using the ``--set_variable VARIABLE`` flag, like so:

.. code:: bash

  %%px --set_variable results
  (* cluster_rank cluster_rank)

Then, in the next cell, you can access ``results``.

Notice that you can use the variable ``cluster_rank`` to partition parts of a problem so that each node is working on something different.

In the examples above, use ``-e`` to evaluate the code in the host kernel as well. Note that ``cluster_rank`` is not defined on the host machine, and that this assumes the host kernel is the same as the parallel machines.

Configuration
-------------
``Metakernel`` subclasses can be configured by the user.  The
configuration file name is determined by the ``app_name`` property of the subclass.
For example, in the ``Octave`` kernel, it is ``octave_kernel``.  The user of the kernel can add an ``octave_kernel_config.py`` file to their
``jupyter`` config path.  The base ``MetaKernel`` class offers ``plot_settings`` as a configurable trait.  Subclasses can define other traits that they wish to make
configurable.

As an example:

.. code:: bash

    cat ~/.jupyter/octave_kernel_config.py
    # use Qt as the default backend for plots
    c.OctaveKernel.plot_settings = dict(backend='qt')

Documentation
-----------------------

Example notebooks can be viewed here_.

Documentation is available online_. Magics have interactive help_ (and online).

For version information, see the Changelog_.

.. _here: http://nbviewer.jupyter.org/github/Calysto/metakernel/tree/main/examples/

.. _help: https://github.com/Calysto/metakernel/blob/main/metakernel/magics/README.md

.. _online: http://Calysto.github.io/metakernel/

.. _Changelog: https://github.com/Calysto/metakernel/blob/main/CHANGELOG.md


# Braintree Python library

The Braintree Python library provides integration access to the Braintree Gateway.

## TLS 1.2 required
> **The Payment Card Industry (PCI) Council has [mandated](https://blog.pcisecuritystandards.org/migrating-from-ssl-and-early-tls) that early versions of TLS be retired from service.  All organizations that handle credit card information are required to comply with this standard. As part of this obligation, Braintree has updated its services to require TLS 1.2 for all HTTPS connections. Braintrees require HTTP/1.1 for all connections. Please see our [technical documentation](https://github.com/paypal/tls-update) for more information.**

## Dependencies

* [requests](http://docs.python-requests.org/en/latest/)

The Braintree Python SDK is tested against Python versions 3.5.3 and 3.12.0.

_The Python core development community has released [End-of-Life branches](https://devguide.python.org/devcycle/#end-of-life-branches) for Python versions 2.7 - 3.4, and are no longer receiving [security updates](https://devguide.python.org/#branchstatus). As a result, Braintree no longer supports these versions of Python._

## Versions

Braintree employs a deprecation policy for our SDKs. For more information on the statuses of an SDK check our [developer docs](https://developer.paypal.com/braintree/docs/reference/general/server-sdk-deprecation-policy).

| Major version number | Status | Released | Deprecated | Unsupported |
| -------------------- | ------ | -------- | ---------- | ----------- |
| 4.x.x | Active | March 2020 | TBA | TBA |
| 3.x.x | Inactive | June 2014 | March 2022 | March 2023 |

## Documentation

 * [Official documentation](https://developer.paypal.com/braintree/docs/start/hello-server/python)

Updating from an Inactive, Deprecated, or Unsupported version of this SDK? Check our [Migration Guide](https://developer.paypal.com/braintree/docs/reference/general/server-sdk-migration-guide/python) for tips.

## Quick Start Example

```python
import braintree

gateway = braintree.BraintreeGateway(
    braintree.Configuration(
        environment=braintree.Environment.Sandbox,
        merchant_id="your_merchant_id",
        public_key="your_public_key",
        private_key="your_private_key",
    )
)

result = gateway.transaction.sale({
    "amount": "1000.00",
    "payment_method_nonce": nonce_from_the_client,
    "options": {
        "submit_for_settlement": True
    }
})

if result.is_success:
    print("success!: " + result.transaction.id)
elif result.transaction:
    print("Error processing transaction:")
    print("  code: " + result.transaction.processor_response_code)
    print("  text: " + result.transaction.processor_response_text)
else:
    for error in result.errors.deep_errors:
        print("attribute: " + error.attribute)
        print("  code: " + error.code)
        print("  message: " + error.message)
```

## Developing

1. Create a [virtualenv](https://virtualenv.pypa.io/) called `venv`:

   ```
   virtualenv venv
   ```

2. Start the virtualenv:

   ```
   source venv/bin/activate
   ```

3. Install dependencies:

   ```
   pip3 install -r dev_requirements.txt
   ```

## Developing (Docker)

The `Makefile` and `Dockerfile` will build an image containing the dependencies and drop you to a terminal where you can run tests.

```
make
```

## Testing

Our friends at [Venmo](https://venmo.com) have [an open source library](https://github.com/venmo/btnamespace) designed to simplify testing of applications using this library.

If you wish to run the tests, make sure you are set up for development (see instructions above). The unit specs can be run by anyone on any system, but the integration specs are meant to be run against a local development server of our gateway code. These integration specs are not meant for public consumption and will likely fail if run on your system. To run unit tests use rake (`rake test:unit`) or unittest (`python3 -m unittest discover tests/unit`).

## License

See the [LICENSE](LICENSE) file for more info.


# BigBang

BigBang is a toolkit for studying communications data from collaborative projects. It currently supports analyzing mailing lists from Sourceforge, Mailman, ListServ (version 16.5 and 17), Pipermail (version 0.09), Hypermail (version 2.4.0) or [.mbox][mbox] files.

Complete documentation for BigBang can be found on [ReadTheDocs](https://bigbang-py.readthedocs.io/en/latest/).

[mbox]: http://tools.ietf.org/html/rfc4155

[![DOI](https://img.shields.io/badge/DIO-10.25080%2FMajora--7b98e3ed--01b-blue)](http://conference.scipy.org/proceedings/scipy2015/sebastian_benthall.html)
[![codecov](https://codecov.io/gh/datactive/bigbang/branch/main/graph/badge.svg?token=Nhyl6g4ZIO)](https://codecov.io/gh/datactive/bigbang)
[![Gitter](https://badges.gitter.im/datactive/bigbang.svg)](https://gitter.im/datactive/bigbang?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)

## Background

Many Standards Development Organizations (SDOs) have working groups that organize themselves through mailing lists. This mailing list data is a valuable source of research insights but can be challenging to gather and analyze. BigBang is an open source toolkit for studying processes of open collaboration and deliberation via analysis of the communications records. Its tools for collecting, analyzing, and visualizing mailing list data are used by a community of information policy researchers to study participation trends and interaction in these settings.

### Three things BigBang Does

- **Ingress**. Tools for collecting data from SDOs, especially their mailing lists.
- **Analysis**. Tools for (pre)processing the data to produce useful insights.
- **Usability/Visualization**. Tools for visualizing and interacting with data.

### Institutional Collaboration

BigBang has been developed by a growing team of researchers spread across many universities and institutions, including UC Berkeley, University of Amsterdam, and New York University. Its development has been funded by Article 19 and Germany's Prototype Fund.

In addition to its scholarly use, BigBang has been building relationships with SDOs themselves. In 2021, the Internet Architecture Board hosted a workshop on Analyzing IETF Data, in which BigBang was featured as a tool for IAB to develop insights into internet governance.

### BigBang as Research Software

BigBing is research software -- written by scholars for our research purposes.

It is part of Scientific Python ecosystem, drawing on many other open source scientific software libraries, such as NumPy, Matplotlib, Pandas, and Jupyter Notebook.

BigBang is a reflexive process. Several of the core developers are also qualitative scholars of socio-technical systems and institutions. Researchers commonly combine BigBang with participant observation in the SDOs they are studying. BigBang is governed by a steering committee of its core developers.

## Installation*

You need to have Git and Pip (for Python3) installed.

Clone the repository and create a virtualenv:

```sh
git clone https://github.com/datactive/bigbang.git
cd bigbang
python3 -m venv env
# activate the virtualenv
. env/bin/activate
```

Inside the virtualenv, install BigBang:

```sh
pip install ".[dev]"
```

When you're done, you can deactivate the virtualenv:

```sh
deactivate
```

This video tutorial shows how to install BigBang.
[![BigBang Video Tutorial](http://img.youtube.com/vi/JWimku8JVqE/0.jpg)](http://www.youtube.com/watch?v=JWimku8JVqE "BigBang Tutorial")

## Usage

There are serveral Jupyter notebooks in the `examples/` directory of this
repository. To open them and begin exploring, run the following commands in the root directory of this repository:

```bash
source activate bigbang
jupyter notebook --notebook-dir=examples/
```

BigBang contains scripts that make it easy to collect data from a variety of sources.
For example, to collect data from an open mailing list archive hosted by Mailman, use:

```bash
bigbang collect-mail --url https://mail.python.org/pipermail/scipy-dev/
```

You can also give this command a file with several urls, one per line. One of these is provided in the `examples/` directory.

```bash
bigbang collect-mail --file examples/urls.txt
```

Once the data has been collected, BigBang has functions to support analysis.

You can read more about data source supported by BigBang in the [documentation](https://bigbang-py.readthedocs.io/).

## Development

### Unit tests

To run the automated unit tests, use: `pytest tests/unit`.

Our current goal is code coverage of **60%**. Add new unit tests within `tests/unit`. Unit tests run quickly, without relying on network requests.

### Documentation

Docstrings are preferred, so that auto-generated web-based documentation will be possible ([#412](https://github.com/datactive/bigbang/issues/412)). You can follow the [Google style guide for docstrings](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings).

### Formatting

Run `pre-commit install` to get automated usage of `black`, `flake8` and `isort` to all Python code files for consistent formatting across developers. We try to follow the [PEP8 style guide](https://pep8.org/).

## Community

If you are interested in participating in BigBang development or would like support from the core development team, please subscribe to the [bigbang-dev mailing list](https://lists.ghserv.net/mailman/listinfo/bigbang-dev) and let us know your suggestions, questions, requests and comments. A [development chatroom](https://gitter.im/datactive/bigbang) is also available.

In the interest of fostering an open and welcoming environment, we as contributors and maintainers [pledge to make participation in our project and our community a harassment-free experience for everyone](CODE_OF_CONDUCT.md).

## Publications

These academic publications use BigBang as part of their methods:

- Becker, Christoph., ten Oever, Niels, and Riccardo Nanni. 2022 “The standardisation of lawful interception technologies in the 3GPP: interrogating 5G and surveillance amid US-China competition“, TPRC2022, Washington DC https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4167105
- Benthall, Sebastian. 2015. “Testing Generative Models of Online Collaboration with BigBang.” In , 182–89. https://conference.scipy.org/proceedings/scipy2015/sebastian_benthall.html.
- Doty, Nick. 2015. “Reviewing for Privacy in Internet and Web Standard-Setting.” In Security and Privacy Workshops (SPW), 2015 IEEE, 185–192. IEEE. https://ieeexplore.ieee.org/document/7163224/
- Milan, Stefania, and Niels ten Oever. 2017. “Coding and Encoding Rights in Internet Infrastructure.” Internet Policy Review 6 (1)
- ten Oever, Niels. 2018. “Productive Contestation, Civil Society, and Global Governance: Human Rights as a Boundary Object in ICANN.” Policy & Internet, June. https://doi.org/10.1002/poi3.172.
- Nanni, Riccardo. “Digital Sovereignty and Internet Standards: Normative Implications of Public-Private Relations among Chinese Stakeholders in the Internet Engineering Task Force.” Information, Communication & Society 0, no. 0 (October 1, 2022): 1–21. https://doi.org/10.1080/1369118X.2022.2129270.
- ten Oever, Niels. 2021. “‘This Is Not How We Imagined It’ -  Technological Affordances, Economic Drivers and the Internet Architecture Imaginary.” New Media & Society. https://journals.sagepub.com/doi/full/10.1177/1461444820929320
- ten Oever, N., Milan, S., & Beraldo, D. (2020). Studying Discourse in Internet Governance through Mailing-list Analysis. In D. L. Cogburn, L. DeNardis, N. S. Levinson, & F. Musiani (Eds.), Research Methods in Internet Governance. Cambridge, MA: MIT Press. https://direct.mit.edu/books/oa-monograph/4936/chapter/625914/Studying-Discourse-in-Internet-Governance-through

## License

MIT, see [LICENSE](LICENSE) for its text. This license may be changed at any time according to the principles of the project [Governance](https://github.com/datactive/bigbang/wiki/Governance).

## Acknowledgements

This project is funded by:


[![License: BSD 3](https://img.shields.io/pypi/l/powerapi.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/powerapi-ng/powerapi/build.yml)](https://github.com/powerapi-ng/powerapi/actions/workflows/build.yml)
[![PyPI](https://img.shields.io/pypi/v/powerapi)](https://pypi.org/project/powerapi/)
[![Codecov](https://codecov.io/gh/powerapi-ng/powerapi/branch/master/graph/badge.svg)](https://codecov.io/gh/powerapi-ng/powerapi)
[![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.11453194.svg)](https://doi.org/10.5281/zenodo.11453194)
[![JOSS paper](https://joss.theoj.org/papers/ff5876ca096c62cb6d243e56b5676c4e/status.svg)](https://joss.theoj.org/papers/ff5876ca096c62cb6d243e56b5676c4e)

PowerAPI is a middleware toolkit for building software-defined power meters.
Software-defined power meters are configurable software libraries that can estimate the power consumption of software in real-time.
PowerAPI supports the acquisition of raw metrics from a wide diversity of sensors (*eg.*, physical meters, processor interfaces, hardware counters, OS counters) and the delivery of power consumptions via different channels (including file system, network, web, graphical).
As a middleware toolkit, PowerAPI offers the capability of assembling power meters *«à la carte»* to accommodate user requirements.

# About

PowerAPI is an open-source project developed by the [Spirals project-team](https://team.inria.fr/spirals), a joint research group between the [University of Lille](https://www.univ-lille.fr) and [Inria](https://www.inria.fr).

The documentation of the project is available [here](http://powerapi.org).

## Mailing list
You can follow the latest news and asks questions by subscribing to our mailing list.

## Contributing
If you would like to contribute code you can do so through GitHub by forking the repository and sending a pull request.

When submitting code, please make every effort to [follow existing conventions and style](CONTRIBUTING.md) in order to keep the code as readable as possible.

## Publications
- **[PowerAPI: A Python framework for building software-defined power meters](https://joss.theoj.org/papers/10.21105/joss.06670)**: G. Fieni, D. Romero Acero, P. Rust, R. Rouvoy. _Journal of Open Source Software_ (JOSS). The Open Journal, June 2024.
- **[Evaluating the Impact of Java Virtual Machines on Energy Consumption](https://hal.inria.fr/hal-03275286v1)**: Z. Ournani, MC. Belgaid, R. Rouvoy, P. Rust, J. Penhoat. _15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ (ESEM). October 2021, Bari, Italy.
- **[SelfWatts: On-the-fly Selection of Performance Events to Optimize Software-defined Power Meters](https://hal.inria.fr/hal-03173410v1)**: G. Fieni, R. Rouvoy, L. Seiturier. _20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing_ (CCGRID 2021). May 2021, Melbourne, Australia.
- **[SmartWatts: Self-Calibrating Software-Defined Power Meter for Containers](https://hal.inria.fr/hal-02470128v1)**: G. Fieni, R. Rouvoy, L. Seiturier. _20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing_ (CCGRID 2020). May 2020, Melbourne, Australia.
- **[Taming Energy Consumption Variations in Systems Benchmarking](https://hal.inria.fr/hal-02403379v1)**: Z. Ournani, MC. Belgaid, R. Rouvoy, P. Rust, J. Penhoat, L. Seinturier. _11th ACM/SPEC International Conference on Performance Engineering_ (ICPE'2020). April 2020, Edmonton, Canada.
- **[The Next 700 CPU Power Models](https://hal.inria.fr/hal-01827132v2)**: M. Colmant, R. Rouvoy, M. Kurpicz, A. Sobe, P. Felber, L. Seinturier. _Elsevier Journal of Systems and Software_ (JSS). 144(10):382-396, Elsevier.
- **[WattsKit: Software-Defined Power Monitoring of Distributed Systems](https://hal.inria.fr/hal-01439889)**: M. Colmant, P. Felber, R. Rouvoy, L. Seinturier. _IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing_ (CCGrid). April 2017, Spain, France. pp.1-14.
- **[Process-level Power Estimation in VM-based Systems](https://hal.inria.fr/hal-01130030)**: M. Colmant, M. Kurpicz, L. Huertas, R. Rouvoy, P. Felber, A. Sobe. _European Conference on Computer Systems_ (EuroSys). April 2015, Bordeaux, France. pp.1-14.
- **[Monitoring Energy Hotspots in Software](https://hal.inria.fr/hal-01069142)**: A. Noureddine, R. Rouvoy, L. Seinturier. _Journal of Automated Software Engineering_, Springer, 2015, pp.1-42.
- **[Unit Testing of Energy Consumption of Software Libraries](https://hal.inria.fr/hal-00912613)**: A. Noureddine, R. Rouvoy, L. Seinturier. _International Symposium On Applied Computing_ (SAC), March 2014, Gyeongju, South Korea. pp.1200-1205.
- **[Informatique : Des logiciels mis au vert](http://www.jinnove.com/Actualites/Informatique-des-logiciels-mis-au-vert)**: L. Seinturier, R. Rouvoy. _J'innove en Nord Pas de Calais_, [NFID](http://www.jinnove.com), 2013.
- **[PowerAPI: A Software Library to Monitor the Energy Consumed at the Process-Level](http://ercim-news.ercim.eu/en92/special/powerapi-a-software-library-to-monitor-the-energy-consumed-at-the-process-level)**: A. Bourdon, A. Noureddine, R. Rouvoy, L. Seinturier. _ERCIM News, Special Theme: Smart Energy Systems_, 92, pp.43-44. [ERCIM](http://www.ercim.eu), 2013.
- **[Mesurer la consommation en énergie des logiciels avec précision](http://www.lifl.fr/digitalAssets/0/807_01info_130110_16_39.pdf)**: A. Bourdon, R. Rouvoy, L. Seinturier. _01 Business & Technologies_, 2013.
- **[A review of energy measurement approaches](https://hal.inria.fr/hal-00912996v2)**: A. Noureddine, R. Rouvoy, L. Seinturier. _ACM SIGOPS Operating Systems Review_, ACM, 2013, 47 (3), pp.42-49.
- **[Runtime Monitoring of Software Energy Hotspots](https://hal.inria.fr/hal-00715331)**: A. Noureddine, A. Bourdon, R. Rouvoy, L. Seinturier. _International Conference on Automated Software Engineering_ (ASE), September 2012, Essen, Germany. pp.160-169.
- **[A Preliminary Study of the Impact of Software Engineering on GreenIT](https://hal.inria.fr/hal-00681560)**: A. Noureddine, A. Bourdon, R. Rouvoy, L. Seinturier. _International Workshop on Green and Sustainable Software_ (GREENS), June 2012, Zurich, Switzerland. pp.21-27.

## Use Cases
PowerAPI is used in a variety of projects to address key challenges of GreenIT:
* [SmartWatts](https://github.com/powerapi-ng/smartwatts-formula) is a self-adaptive power meter that can estimate the energy consumption of software containers in real-time.
* [GenPack](https://hal.inria.fr/hal-01403486) provides a container scheduling strategy to minimize the energy footprint of cloud infrastructures.
* [VirtualWatts](https://github.com/powerapi-ng/virtualwatts-formula) provides process-level power estimation of applications running in virtual machines.
* [Web Energy Archive](http://webenergyarchive.com) ranks popular websites based on the energy footpring they imposes to browsers.
* [Greenspector](https://greenspector.com) optimises the power consumption of software by identifying potential energy leaks in the source code.

## Research Projects

Currently, PowerAPI is used in two research projects:

- [Distiller ANR Project](https://www.davidson.fr/blog/comment-reduire-limpact-environnemental-des-applications-cloud-davidson-consulting-inria-ovhcloud-orange-sassocient-pour-le-programme-de-recherche-distiller) that searches to reduce energy consumption of Cloud applications.
- [Défi Pulse](https://www.inria.fr/fr/pulse-defi-qarnot-computing-ademe-calcul-intensif-hpc-environnement) studies how to valorize emissions from High Performance Computing (HPC) using as use case [Qarnot Computing's](https://qarnot.com/en) offers.

## License
PowerAPI is licensed under the BSD-3-Clause License. See the [LICENSE](LICENSE) file for details.

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fpowerapi-ng%2Fpowerapi.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fpowerapi-ng%2Fpowerapi?ref=badge_large)


[![DOI](http://joss.theoj.org/papers/10.21105/joss.01361/status.svg)](https://doi.org/10.21105/joss.01361)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5553202.svg)](https://doi.org/10.5281/zenodo.5553202)
[![Documentation Status](https://readthedocs.org/projects/smact/badge/?version=latest)](http://smact.readthedocs.org/en/latest/?badge=latest)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
![python version](https://img.shields.io/pypi/pyversions/smact)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![PyPi](https://img.shields.io/pypi/v/smact)](https://pypi.org/project/SMACT/)
[![Conda](https://anaconda.org/conda-forge/smact/badges/version.svg)](https://anaconda.org/conda-forge/smact)
[![GitHub issues](https://img.shields.io/github/issues-raw/WMD-Group/SMACT)](https://github.com/WMD-group/SMACT/issues)
![dependencies](https://img.shields.io/librariesio/release/pypi/smact)
[![CI Status](https://github.com/WMD-group/SMACT/actions/workflows/ci.yml/badge.svg)](https://github.com/WMD-group/SMACT/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/WMD-group/SMACT/branch/master/graph/badge.svg?token=UtgVxjoYNP)](https://codecov.io/gh/WMD-group/SMACT)
![PyPI - Downloads](https://img.shields.io/pypi/dm/smact)

SMACT
=====

**Semiconducting Materials from Analogy and Chemical Theory** (SMACT) is a collection of rapid screening and informatics tools that uses data about chemical elements.

- **Documentation:** https://smact.readthedocs.io/en/latest/
- **Examples:** https://github.com/WMD-group/SMACT/tree/master/examples

![](SMACT.png)

*If you torture the data enough, nature will always confess* - Roland Coase (from 'How should economists choose?')

Statement of need
--------
There is a strong demand for functional materials across a wide range of technologies. The motivation can include cost reduction, performance enhancement, or to enable a new application. We have developed low-cost procedures for screening hypothetical materials. This framework can be used for simple calculations on your own computer. SMACT follows a top-down approach where a set of element combinations is generated and then screened using rapid chemical filters. It can be used as part of a multi-technique workflow or to feed machine learning models for materials.

![](smact_simple.gif)

Getting started
-----
Features are accessed through Python scripts, importing classes and functions as needed.
The best place to start is looking at [the docs](https://smact.readthedocs.io/en/latest/), which highlight some simple examples of how these classes and functions can be used.
Extended examples are available in [our examples folder](https://github.com/WMD-group/SMACT/tree/master/examples).

Code features
--------
- At the core of SMACT are [Element](https://smact.readthedocs.io/en/latest/smact.html#smact.Element) and [Species](https://smact.readthedocs.io/en/latest/smact.html#smact.Species) (element in a given oxidation state) classes that have various properties associated with them.

- Oxidation states that are accessible to each element are included in their properties.

- Element compositions can be screened through based on the heuristic filters of charge neutrality and electronegativity order. This is handled using the [screening module](https://smact.readthedocs.io/en/latest/smact.screening.html) and [this publication](https://www.cell.com/chem/fulltext/S2451-9294(16)30155-3) describes the underlying theory. An example procedure is [outlined in the docs](https://smact.readthedocs.io/en/latest/examples.html#neutral-combinations) and more examples can be found in the [counting examples subfolder](https://github.com/WMD-group/SMACT/tree/master/examples/Counting).

- Further filters can be applied to generated lists of compositions in order to screen for particular properties. These properties are either intrinsic properties of elements or are calculated for compositions using the [properties module](https://smact.readthedocs.io/en/latest/smact.properties.html). For example:
  - A use case is shown in [this publication](https://pubs.rsc.org/en/content/articlehtml/2018/sc/c7sc03961a), in which 160,000 chemical compositions are screened based on optical band gap calculated using the [solid-state energy scale](https://www.sciencedirect.com/science/article/pii/S0022459615300888).
  - The [oxidation_states module](https://smact.readthedocs.io/en/latest/smact.oxidation_states.html) can be used to filter out compositions containing metals in unlikely oxidation states according to [a data-driven model](https://pubs.rsc.org/en/content/articlelanding/2018/fd/c8fd00032h#!divAbstract).

- Compositions can also be filtered based on sustainability via the abundance of elements in the Earth's crust or via the [HHI scale](https://pubs.acs.org/doi/10.1021/cm400893e).

- Compositions can be converted for use in Pymatgen or for representation to machine learning algorithms ([see "next steps" in this example](https://github.com/WMD-group/SMACT/blob/master/examples/Counting/Generate_compositions_lists.ipynb)) and the related [ElementEmbeddings](https://github.com/WMD-group/ElementEmbeddings) package.

- The code also has tools for manipulating common crystal lattice types:
  - Certain structure types can be built using the [builder module](https://smact.readthedocs.io/en/latest/smact.builder.html)
  - Lattice parameters can be estimated using ionic radii of the elements for various common crystal structure types using the [lattice_parameters module](https://smact.readthedocs.io/en/latest/smact.lattice_parameters.html).
  - The [lattice module](https://smact.readthedocs.io/en/latest/smact.lattice.html) and [distorter module](https://smact.readthedocs.io/en/latest/smact.distorter.html) rely on the [Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase/) and can be used to generate unique atomic substitutions on a given crystal structure.
  - The [structure  prediction](https://smact.readthedocs.io/en/latest/smact.structure_prediction.html) module can be used to predict the structure of hypothetical compositions using species similarity measures.
  - The [dopant prediction](https://smact.readthedocs.io/en/latest/smact.dopant_prediction.html) module can be used to facilitate high-throughput predictions of p-type and n-type dopants of multicomponent solids.

List of modules
-------
* **smact** library containing:
  * **\_\_init\_\_.py** Contains the core `Element` and `Species` classes.
  *  **data_loader.py** Handles the loading of external data used to initialise the core `smact.Element` and `smact.Species` classes.
  *  **screening.py** Used for generating and applying filters to compositional search spaces.
  *  **properties.py** A collection of tools for estimating useful properties based on composition.
  * **lattice.py** Given the sites, multiplicities and possible oxidation states
    at those sites, this reads from the database and generates all possible
    stoichiometeries.
  * **builder.py** Builds some common lattice structures, given the chemical
    composition.
  * **lattice_parameters.py** Estimation of lattice parameters for various lattice types using covalent/ionic radii.
  * **distorter.py** A collection of functions for enumerating and then
    substituting on inequivalent sites of a sub-lattice.
  * **oxidation_states.py**: Used for predicting the likelihood of species coexisting in a compound based on a statistical model.
  * **structure_prediction**: A submodule which contains a collection of tools for facilitating crystal structure predictions via ionic substitutions
  * **dopant_prediction**: A submodule which contains a collections of tools for predicting dopants.

Requirements
------------
The main language is Python 3 and has been tested using Python 3.9+.
Basic requirements are Numpy and Scipy.
The [Atomic Simulation Environment](https://wiki.fysik.dtu.dk/ase) (ASE),  [spglib](http://atztogo.github.io/spglib), and [pymatgen](www.pymatgen.org) are also required for many components.

Installation
------------
The latest stable release can be installed via pip which will automatically setup other Python packages as required:

    pip install smact

SMACT is also available via conda through the conda-forge channel on Anaconda Cloud:

```
conda install -c conda-forge smact
```

Alternatively, the very latest version can be installed using:

    pip install git+git://github.com/WMD-group/SMACT.git

For developer installation SMACT can be installed from a copy of the source
repository (https://github.com/wmd-group/smact); this will be preferred if using experimental code branches.

To clone the project from Github and make a local installation:

    git clone https://github.com/wmd-group/smact.git
    cd smact
    pip install --user -e .

With -e pip will create links to the source folder so that that changes
to the code will be immediately reflected on the PATH.

License and attribution
-----------------------

Python code and original data tables are licensed under the MIT License.

Development notes
-----------------

### Bugs, features and questions
Please use the [Issue Tracker](https://github.com/WMD-group/smact/issues) to report bugs or request features in the first instance. While we hope that most questions can be answered by searching [the docs](https://smact.readthedocs.io/en/latest/), we welcome new questions on the issue tracker, especially if they helps us improve the docs! For other queries about any aspect of the code, please contact either Dan Davies (author) or Anthony Onwuli (maintainer) by e-mail: d.w.davies@imperial.ac.uk or anthony.onwuli16@imperial.ac.uk respectively.

### Code contributions
We are always looking for ways to make SMACT better and more useful to the wider community; contributions are welcome. Please use the ["Fork and Pull"](https://guides.github.com/activities/forking/) workflow to make contributions and stick as closely as possible to the following:

- Code style should comply with [PEP8](http://www.python.org/dev/peps/pep-0008) where possible. [Google's house style](https://google.github.io/styleguide/pyguide.html) is also helpful, including a good model for docstrings.
- Please use comments liberally when adding nontrivial features, and take the chance to clean up other people's code while looking at it.
- Add tests wherever possible, and use the test suite to check if you broke anything.

### Tests
Testing modules should be pass/fail and wrapped into **tests/test_core.py** or another **tests/test_something.py** file added, if appropriate.
Run the tests using `python -m pytest -v`.
(The final `-v` is optional and adds more detail to the output.)

We also use integrated testing on Github via [GitHub Actions](hhttps://github.com/features/actions).

References
----------

[H. Park et al,
"Mapping inorganic crystal chemical space" *Faraday Discuss.* (2024)](https://pubs.rsc.org/en/content/articlelanding/2024/fd/d4fd00063c)

[D. W. Davies et al,
"SMACT: Semiconducting Materials by Analogy and Chemical Theory" *JOSS* **4**, 1361 (2019)](https://joss.theoj.org/papers/7efd2f2ad60d25bdccee3fbd3fc11448)

[D. W. Davies et al,
"Materials discovery by chemical analogy: role of oxidation states in structure prediction" *Faraday Discuss.* **211**, 553 (2018)](https://pubs.rsc.org/en/Content/ArticleLanding/2018/FD/C8FD00032H)

[D. W. Davies et al,
"Computational screening of all stoichiometric inorganic materials" *Chem* **1**, 617 (2016)](http://www.cell.com/chem/abstract/S2451-9294(16)30155-3)

[B. R. Pamplin, "A systematic method of deriving new semiconducting
compounds by structural analogy", *J. Phys. Chem. Solids*
**25**, 675 (1964)](http://www.sciencedirect.com/science/article/pii/0022369764901763)


[![Release Notes](https://img.shields.io/github/release/labelbox/labelbox-python)](https://github.com/Labelbox/labelbox-python/releases)
[![CI)](https://github.com/Labelbox/labelbox-python/actions/workflows/python-package-develop.yml/badge.svg?branch=develop)](https://github.com/Labelbox/labelbox-python/actions/workflows/python-package-develop.yml)
[![Downloads](https://pepy.tech/badge/labelbox)](https://pepy.tech/project/labelbox)
[![Dependency Status](https://img.shields.io/librariesio/github/labelbox/labelbox-python)](https://libraries.io/github/labelbox/labelbox-python)
[![Open Issues](https://img.shields.io/github/issues-raw/labelbox/labelbox-python)](https://github.com/labelbox/labelbox-python/issues)
[![Changelog](https://img.shields.io/badge/Changelog-Recent%20Updates-blue.svg)](https://docs.labelbox.com/changelog)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Twitter Follow](https://img.shields.io/twitter/follow/labelbox.svg?style=social&label=Follow)](https://twitter.com/labelbox)
[![LinkedIn Follow](https://img.shields.io/badge/Follow-LinkedIn-blue.svg?style=flat&logo=linkedin)](https://www.linkedin.com/company/labelbox/)
[![Supported Python Versions](https://img.shields.io/pypi/pyversions/labelbox)](https://img.shields.io/pypi/pyversions/labelbox)
[![SLSA 3](https://slsa.dev/images/gh-badge-level3.svg)](https://slsa.dev)

# Labelbox

Labelbox is a powerful data-centric AI platform that empowers enterprises to develop, optimize, and leverage AI to solve complex problems and drive innovation in their products and services.

With Labelbox, enterprises can easily curate and annotate data, generate high-quality human feedback data for computer vision and language models, evaluate and improve model performance, and automate tasks by seamlessly combining AI and human-centric workflows. The academic and research community also relies on Labelbox for cutting-edge AI research and experimentation.

Visit [Labelbox](http://labelbox.com/) for more information.

## Table of Contents
- [Quick Start](#quick-start)
- [Contribution Guidelines](#contribution-guidelines)
- [Develop with AI Assistance](#develop-with-ai-assistance)
- [Documentation](#documentation)
- [Jupyter Notebooks](#jupyter-notebooks)

## Quick Start

### Sign Up
If you haven't already, create a free account at [Labelbox](http://app.labelbox.com/).

### Generate an API key
Log into Labelbox and navigate to [Account > API Keys](https://docs.labelbox.com/docs/create-an-api-key) to generate an API key.

### Install

To install the SDK, run the following command.

```bash
pip install labelbox
```

If you'd like to install the SDK with enhanced functionality, which additional optional capabilities surrounding data processing, run the following command.

```bash
pip install "labelbox[data]"
```

**Please note that if you prefer to build and install your own version of the SDK, it is important to be aware that only tagged commits have been thoroughly tested. Building from the head of develop branch carries some level of risk.**

### Validate Installation and API Key

After installing the SDK and getting an API Key, it's time to validate them both.

```python
import labelbox as lb

client = lb.Client(API_KEY) # API_KEY = API Key generated from labelbox.com
dataset = client.create_dataset(name="Test Dataset")
data_rows = [{"row_data": "My First Data Row", "global_key": "first-data-row"}]
task = dataset.create_data_rows(data_rows)
task.wait_till_done()
```

You should be set! Running the snippet above should create a dataset called `Test Dataset` with its content being `My First Data Row`. You can log into [Labelbox](http://labelbox.com/) to verify this. If you have any issues please file a Github Issue or contact [Labelbox Support](https://docs.labelbox.com/docs/contacting-customer-support) directly. For more advanced examples and information on the SDK, see [Documentation](#documentation) below.

## Contribution Guidelines
We encourage anyone to contribute to this repository to help improve it. Please refer to [Contributing Guide](CONTRIBUTING.md) for detailed information on how to contribute. This guide also includes instructions for how to build and run the SDK locally.

## Develop with AI assistance
### Use the codebase as context for large language models
Using the [GPT repository loader](https://github.com/mpoon/gpt-repository-loader), we have created `lbx_prompt.txt` that contains data from all `.py` and `.md` files. The file has about 730k tokens. We recommend using Gemini 1.5 Pro with 1 million context length window.

## Documentation
The SDK is well-documented to help developers get started quickly and use the SDK effectively. Here are links to that documentation:

- [Labelbox Official Documentation](https://docs.labelbox.com/docs/overview)
- [Jupyter Notebook Examples](https://github.com/Labelbox/labelbox-python/tree/develop/examples)
- [Python SDK Reference](https://labelbox-python.readthedocs.io/en/latest/)

## Jupyter Notebooks
We have samples in the `examples` directory to help you get started with the SDK.

Make sure your notebook will use your source code:
1. `ipython profile create`
2. `ipython locate` - will show where the config file is. This is the config file used by the Jupyter server, since it runs via ipython
3. Open the file (this should be ipython_config.py and it is usually located in ~/.ipython/profile_default) and add the following line of code:
```
c.InteractiveShellApp.exec_lines = [
  'import sys; sys.path.insert(0, "<labelbox-python root folder>")'
]
```
4. Go to the root of your project and run `jupyter notebook` to start the server.

## Provenance

To enhance the software supply chain security of Labelbox's users, as of v.3.72.2, every Labelbox SDK release contains a [SLSA Level 3 Provenance](https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/generic/README.md) document.
This document provides detailed information about the build process, including the repository and branch from which the package was generated.

By using the [SLSA framework's official verifier](https://github.com/slsa-framework/slsa-verifier), you can verify the provenance document to ensure that the package is from a trusted source. Verifying the provenance helps confirm that the package has not been tampered with and was built in a secure environment.

Example of usage for the v.3.72.2 release wheel:

```
export VERSION=3.72.2 # sdk release version
export TAG=v.3.72.2 # github tag
pip download --no-deps labelbox==${VERSION}

curl --location -O \
  https://github.com/Labelbox/labelbox-python/releases/download/${TAG}/multiple.intoto.jsonl

slsa-verifier verify-artifact --source-branch develop --builder-id 'https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.0.0' --source-uri "git+https://github.com/Labelbox/labelbox-python" --provenance-path multiple.intoto.jsonl ./labelbox-${VERSION}-py3-none-any.whl
```


# mypyvy [![Build Status](https://github.com/wilcoxjay/mypyvy/actions/workflows/build.yml/badge.svg)](https://github.com/wilcoxjay/mypyvy/actions)

A language for symbolic transition systems, inspired by Ivy.

## Dependencies

You need python version 3.8 or higher.

```
python3 --version
Python 3.8.2
```

Make sure z3 is installed and on your `PYTHONPATH`. Basic functionality should
work with any version from the past few years, but for best results, use a recent
release, such as 4.11.2.

```
z3 --version
Z3 version 4.11.2 - 64 bit
```

Importing z3 should work (not report any errors) as follows.

```
python3
...
>>> import z3
>>>
```

You need all the python packages given in `requirements.txt`, which you can
do by running the following:

```
python3 -m pip install -r requirements.txt
```

You may wish to set up a virtual environment if you're into that.

## Getting started

mypyvy takes an input file describing a symbolic transition system and can
perform various tasks such as inductive invariant checking and inference.  For
an example input, see `examples/lockserv.pyv`, which is written in an tutorial
style.

For users who are familiar with Ivy, the syntax of mypyvy is broadly similar to
Ivy.  The primary difference is that transitions are specified directly as a
double-vocabulary formula, using `new(R(x))` to refer to the post-state version
of relation `R`.

The mypyvy command line tool has several modes, all of which take a single `.pyv`
file.  See `mypyvy --help` for a list of modes and `mypyvy <mode> --help` for
a description of all command line options to a particular mode.
- `verify`: verifies that the invariants given in the file are inductive.
  For example, we can verify the lock service:
```
python3 src/mypyvy.py verify examples/lockserv.pyv
checking init:
  implies invariant mutex... ok. (0:00:00.000176)
  ...
checking transation send_lock:
  preserves invariant mutex... ok. (0:00:00.000120)
  preserves invariant on line 109... ok. (0:00:00.000107)
  ...
...
all ok!
```

  The `all ok!` message means success. If you delete one of the invariants and run again,
  you can see what the counterexamples look like.

- `updr`: search for a strengthening that proves the safety property.  For
  example, we can ask it to strengthen the mutex property of the lock service:

```
python3 src/mypyvy.py updr examples/lockserv.pyv
checking init:
  implies invariant mutex... ok. (0:00:00.000234)
frame is safe and inductive. done!
!(exists node0:node, node1:node. node0 != node1 & holds_lock(node0) & holds_lock(node1))
!(exists node0:node, node1:node. grant_msg(node1) & holds_lock(node0))
!(exists node0:node. holds_lock(node0) & server_holds_lock)
!(exists node0:node, node1:node. node0 != node1 & grant_msg(node0) & grant_msg(node1))
!(exists node0:node. grant_msg(node0) & server_holds_lock)
!(exists node0:node, node1:node. holds_lock(node0) & unlock_msg(node1))
!(exists node0:node, node1:node. grant_msg(node0) & unlock_msg(node1))
!(exists node0:node, node1:node. node0 != node1 & unlock_msg(node0) & unlock_msg(node1))
!(exists node0:node. server_holds_lock & unlock_msg(node0))
```

  The message `frame is safe and inductive. done!` means success, and then it
  prints out the inductive strengthening.  Note that even though the file
  `examples/lockserv.pyv` actually already contains an inductive invariant to prove
  `mutex`, the algorithm *does not* use the given strengthening, but looks only
  at `mutex`, which is marked as a `safety` property.  (You can see this by
  going and deleting all the other invariants in the file and re-running.)

- `bmc`: performs bounded model checking out to depth given by the `--depth=DEPTH`
  flag for a property given by the `--safety=NAME` flag. For example, we can check
  that `mutex` property is true for 5 steps as follows:
```
python3 src/mypyvy.py bmc --depth=5 --safety=mutex examples/lockserv.pyv
bmc checking the following property to depth 5
  forall N1:node, N2:node. holds_lock(N1) & holds_lock(N2) -> N1 = N2
ok. (0:00:00.062531)
```

  The `ok.` message means success.

  As another example, you could add the (false) invariant
```
invariant [bad] !holds_lock(N)
```
  to the file and then rerun.  `mypyvy` reports a counterexample trace demonstrating
  how to reach a state that violates the invariant. (Note the use of the option
  `--minimize-models` to find the trace with the fewest client nodes.)

```
python3 src/mypyvy.py bmc --depth=5 --safety=bad --minimize-models examples/lockserv.pyv
bmc checking the following property to depth 5
  forall N:node. !holds_lock(N)

sort node
  node0

state 0:
server_holds_lock()

transition send_lock

state 1:
lock_msg(node0)
server_holds_lock()

transition recv_lock

state 2:
grant_msg(node0)

transition send_lock

state 3:
grant_msg(node0)
lock_msg(node0)

transition stutter

state 4:
grant_msg(node0)
lock_msg(node0)

transition recv_grant

state 5:
holds_lock(node0)
lock_msg(node0)
error: found concrete trace violating safety
```

  The `error: ...` message means that a counterexample was found.  The trace is
  pretty readable!

- `theorem`: proves state-independent theorems about the background axioms of a model.
  Currently not documented and rarely used.

- `trace`: is an generalization of `bmc` that allows user-guided queries over executions.
  For example, at the bottom of the lockserv file, we see the following declaration:
```
sat trace {
  any transition
  any transition
  any transition
  assert exists N. holds_lock(N)
}
```

which asks mypyvy to find an execution with 3 steps that ends in a state where
some client holds the lock. (This is essentially the same query we used bmc to
answer above.)  A trace declaration says whether it is expected to be `sat` or
`unsat`, and mypyvy reports an error if it disagrees.  The syntax of trace queries
is under flux and currently undocumented.

- `typecheck`: This mode justs typechecks the input file and then exits. It is
  used by the emacs mode when the system's verification queries get too expensive
  to run on every keystroke.


[![Build Status](https://travis-ci.org/choderalab/pymbar.png)](https://travis-ci.org/choderalab/pymbar)
[![Anaconda Cloud Downloads Badge](https://anaconda.org/conda-forge/pymbar/badges/downloads.svg)](https://anaconda.org/conda-forge/pymbar)
[![Anaconda Cloud Badge](https://anaconda.org/conda-forge/pymbar/badges/installer/conda.svg)](https://anaconda.org/conda-forge/pymbar)
[![PyPI Version](https://badge.fury.io/py/pymbar.png)](https://pypi.python.org/pypi/pymbar)
[![DOI](https://zenodo.org/badge/9991771.svg)](https://zenodo.org/badge/latestdoi/9991771)

pymbar
======

Python implementation of the [multistate Bennett acceptance ratio (MBAR)](http://www.alchemistry.org/wiki/Multistate_Bennett_Acceptance_Ratio) method for estimating expectations and free energy differences from equilibrium samples from multiple probability densities.
See our [docs](http://pymbar.readthedocs.org/en/latest/).

Installation
------------

The easiest way to install the `pymbar` release is via [conda](http://conda.pydata.org):

```bash
conda install -c conda-forge pymbar
```
which will come with JAX to speed up the code. Or to get the non-JAX accelerated version:
```bash
conda install -c conda-forge pymbar-core
```

You can also install JAX accelerated `pymbar` from the [Python package index](https://pypi.python.org/pypi/pymbar)
using `pip`:
```bash
pip install pymbar[jax]
```
or the non-jax-accelerated version with
```bash
pip install pymbar
```
Whether you install the JAX accelerated or non-JAX-accelerated version does not
change any calls or how the code is run. The non-Jax version is smaller on disk due to smaller
dependencies, but may not run as fast.

The development version can be installed directly from github via `pip`:

```bash
# Get the compressed tarball
pip install https://github.com/choderalab/pymbar/archive/master.tar.gz
# Or obtain a temporary clone of the repo with git
pip install git+https://github.com/choderalab/pymbar.git
```

Usage
-----

Basic usage involves importing `pymbar` and constructing an `MBAR` object from the [reduced potential](http://www.alchemistry.org/wiki/Multistate_Bennett_Acceptance_Ratio#Reduced_potential) of simulation or experimental data.

Suppose we sample a 1D harmonic oscillator from a few thermodynamic states:
```python
>>> from pymbar import testsystems
>>> x_n, u_kn, N_k, s_n = testsystems.HarmonicOscillatorsTestCase().sample()
```
We have the `nsamples` sampled oscillator positions `x_n` (with samples from all states concatenated), [reduced potentials](http://www.alchemistry.org/wiki/Multistate_Bennett_Acceptance_Ratio#Reduced_potential) in the `(nstates,nsamples)` matrix `u_kn`, number of samples per state in the `nsamples` array `N_k`, and indices `s_n` denoting which thermodynamic state each sample was drawn from.

To analyze this data, we first initialize the `MBAR` object:

```python
>>> mbar = MBAR(u_kn, N_k)
```

Estimating dimensionless free energy differences between the sampled thermodynamic states and their associated uncertainties (standard errors) simply requires a call to `compute_free_energy_differences()`:

```python
>>> results = mbar.compute_free_energy_differences()
```

Here `results` is a dictionary with keys `Deltaf_ij`, `dDeltaf`, and `Theta`. `Deltaf_ij[i,j]` is the matrix of dimensionless free energy differences `f_j - f_i`, `dDeltaf_ij[i,j]` is the matrix of standard errors in this matrices estimate, and `Theta` is a covariance matrix that can be used to propagate error into quantities derived from the free energies.

Expectations and associated uncertainties can easily be estimated for observables `A(x)` for all states:

```python
>>> A_kn = x_kn # use position of harmonic oscillator as observable
>>> results = mbar.compute_expectations(A_kn)
```

where `results` is a dictionary with keys `mu`, `sigma`, and `Theta`, where `mu[i]` is the array of the estimate for the average of the observable for in state i, `sigma[i]` is the estimated standard deviation of the `mu` estimates,  and `Theta[i,j]` is the covariance matrix of the log weights.

See the docstring help for these individual methods for more information on exact usage; in Python or IPython, you can view the docstrings with `help()`.

JAX needs 64-bit mode
---------------------
PyMBAR needs 64-bit floats to provide reliable answers. JAX by default uses
[32-bit (Single) bitsize](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#double-64bit-precision).
PyMBAR will turn on JAX's 64-bit mode, which may cause issues with some separate uses of JAX in the same code as PyMBAR,
such as existing Neural Network (NN) Models for machine learning.

Authors
-------
* Kyle A. Beauchamp <kyle.beauchamp@choderalab.org>
* John D. Chodera <john.chodera@choderalab.org>
* Levi N. Naden <lnaden@vt.edu>
* Michael R. Shirts <michael.shirts@colorado.edu>

References
----------

* Please cite the original MBAR paper:

  > Shirts MR and Chodera JD. Statistically optimal analysis of samples from multiple equilibrium states. J. Chem. Phys. 129:124105 (2008).  [DOI](http://dx.doi.org/10.1063/1.2978177)

* Some timeseries algorithms can be found in the following reference:

  > Chodera JD, Swope WC, Pitera JW, Seok C, and Dill KA. Use of the weighted histogram analysis method for the analysis of simulated and parallel tempering simulations. J. Chem. Theor. Comput. 3(1):26-41 (2007).  [DOI](http://dx.doi.org/10.1021/ct0502864)

* The automatic equilibration detection method provided in `pymbar.timeseries.detectEquilibration()` is described here:

  > Chodera JD. A simple method for automated equilibration detection in molecular simulations. J. Chem. Theor. Comput. 12:1799, 2016.  [DOI](http://dx.doi.org/10.1021/acs.jctc.5b00784)

License
-------

`pymbar` is free software and is licensed under the MIT license.

Thanks
------
We would especially like to thank a large number of people for helping us identify issues
and ways to improve `pymbar`, including Tommy Knotts, David Mobley, Himanshu Paliwal,
Zhiqiang Tan, Patrick Varilly, Todd Gingrich, Aaron Keys, Anna Schneider, Adrian Roitberg,
Nick Schafer, Thomas Speck, Troy van Voorhis, Gupreet Singh, Jason Wagoner, Gabriel Rocklin,
Yannick Spill, Ilya Chorny, Greg Bowman, Vincent Voelz, Peter Kasson, Dave Caplan, Sam Moors,
Carl Rogers, Josua Adelman, Javier Palacios, David Chandler, Andrew Jewett, Stefano Martiniani, and Antonia Mey.

Notes
-----
* `alchemical-analysis.py` described in [this publication](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4420631/) has been [relocated here](https://github.com/MobleyLab/alchemical-analysis).


OpenAeroStruct
==============

![GitHub Actions Status](https://github.com/mdolab/OpenAeroStruct/workflows/OAS%20build/badge.svg)
[![codecov](https://codecov.io/gh/mdolab/OpenAeroStruct/branch/main/graph/badge.svg?token=yOxeH7rT2H)](https://codecov.io/gh/mdolab/OpenAeroStruct)
[![Documentation Status](https://readthedocs.com/projects/mdolab-openaerostruct/badge/?version=latest)](https://mdolab-openaerostruct.readthedocs-hosted.com/en/latest/?badge=latest)
[![PyPI](https://img.shields.io/pypi/v/openaerostruct)](https://pypi.org/project/openaerostruct/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/openaerostruct)](https://pypi.org/project/openaerostruct/)

OpenAeroStruct is a lightweight tool that performs aerostructural optimization using OpenMDAO.
It couples a vortex-lattice method (VLM) and a 6 degrees of freedom 3-dimensional spatial beam model to simulate aerodynamic and structural analyses using lifting surfaces.
These simulations are wrapped with an optimizer using NASA's OpenMDAO framework.
The analysis and optimization results can be visualized using included tools, producing figures such as these:

*With a tubular structure*
![Example](https://raw.githubusercontent.com/mdolab/OpenAeroStruct/main/openaerostruct/docs/figures/example.png)

*With a wingbox structure*
![Example2](https://raw.githubusercontent.com/mdolab/OpenAeroStruct/main/openaerostruct/docs/figures/wingbox_fine.png)

Please note that this repository is provided as is without any guaranteed support.
If you would like to highlight issues, ask questions, or make changes, please do so using GitHub Issues and Pull Requests.
The developers will address them at their discretion.

The easiest way to get started is by installing OpenAeroStruct via the Python Package Index with

`pip install openaerostruct`

If you'd like easier access to the examples and source code, install OpenAeroStruct by cloning this repository and entering the folder it generates.
Then do:

`pip install -e .`

Documentation
-------------

Please see the [documentation](https://mdolab-openaerostruct.readthedocs-hosted.com/en/latest/) for more installation details, walkthroughs, and examples.

Citation
--------

For more background, theory, and figures, see the [OpenAeroStruct journal article](https://mdolab.engin.umich.edu/bibliography/Jasa2018a.html).
Please cite this article when using OpenAeroStruct in your research or curricula.

John P. Jasa, John T. Hwang, and Joaquim R. R. A. Martins. "Open-source coupled aerostructural optimization using Python." Structural and Multidisciplinary Optimization 57.4 (2018): 1815-1827. DOI: 10.1007/s00158-018-1912-8

```
@article{Jasa2018a,
	Author = {John P. Jasa and John T. Hwang and Joaquim R. R. A. Martins},
	Doi = {10.1007/s00158-018-1912-8},
	Journal = {Structural and Multidisciplinary Optimization},
	Month = {April},
	Number = {4},
	Pages = {1815--1827},
	Publisher = {Springer},
	Title = {Open-source coupled aerostructural optimization using {Python}},
	Volume = {57},
	Year = {2018}}
```

If using the wingbox model, fuel-weight inertial loads, or structural-weight inertial loads, please cite the following [conference paper](https://www.researchgate.net/publication/327654423_Low-Fidelity_Aerostructural_Optimization_of_Aircraft_Wings_with_a_Simplified_Wingbox_Model_Using_OpenAeroStruct).

Shamsheer S. Chauhan and Joaquim R. R. A. Martins, “Low-Fidelity Aerostructural Optimization of Aircraft Wings with a Simplified Wingbox Model Using OpenAeroStruct,” Proceedings of the 6th International Conference on Engineering Optimization, EngOpt 2018, Springer, Lisbon, Portugal, September 2018, pp. 418–431. doi:10.1007/978-3-319-97773-7 38

```
@inproceedings{Chauhan2018b,
	Author = {Shamsheer S. Chauhan and Joaquim R. R. A. Martins},
	Address = {Lisbon, Portugal},
	Booktitle = {Proceedings of the 6th International Conference on Engineering Optimization, EngOpt 2018},
	Doi = {10.1007/978-3-319-97773-7_38},
	Pages = {418-431},
	Publisher = {Springer},
	Title = {Low-Fidelity Aerostructural Optimization of Aircraft Wings with a Simplified Wingbox Model Using {OpenAeroStruct}},
	Year = {2018}}
```

If using point-mass loads or thrust loads, please cite the following [conference paper](https://www.researchgate.net/publication/333806174_How_Certain_Physical_Considerations_Impact_Aerostructural_Wing_Optimization).

John P. Jasa, Shamsheer S. Chauhan, Justin S. Gray, and Joaquim R. R. A. Martins, “How Certain Physical Considerations Impact Aerostructural Wing Optimization,” AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Dallas, TX, 2019. doi:10.2514/6.2019-3242

```
@inproceedings{Jasa2019c,
	Author = {John P. Jasa and Shamsheer S. Chauhan and Justin S. Gray and Joaquim R. R. A. Martins},
	Address = {Dallas, TX},
	Booktitle = {AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference},
	Doi = {10.2514/6.2019-3242},
	Title = {How Certain Physical Considerations Impact Aerostructural Wing Optimization},
	Month = {June},
	Year = {2019}}
```

Version Information
-------------------
The oldest and latest versions of the dependencies that we test regularly are the following (other versions may work, but no guarantees):

| Dependency         | oldest | latest |
|--------------------|--------| ------ |
| Python             | 3.8    | 3.11   |
| NumPy              | 1.20   | latest |
| SciPy              | 1.6.0  | latest |
| OpenMDAO           | 3.15   | latest |
| MPhys              | 1.0.0  | latest |
| Matplotlib         | latest | latest |
| pyGeo (optional)   | 1.6.0  | latest |
| OpenVSP (optional) | 3.27.1 | 3.27.1 |

If you are looking to use the previous version of OpenAeroStruct which uses OpenMDAO 1.7.4, use OpenAeroStruct 1.0 from [here](https://github.com/mdolab/OpenAeroStruct/releases).

License
-------
Copyright 2018-2023 MDO Lab

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.