# Data Hub API

[![CircleCI](https://circleci.com/gh/uktrade/data-hub-api.svg?style=svg)](https://circleci.com/gh/uktrade/data-hub-api)
[![codecov](https://codecov.io/gh/uktrade/data-hub-api/branch/main/graph/badge.svg)](https://codecov.io/gh/uktrade/data-hub-api)

Data Hub API provides an API into Data Hub for Data Hub clients. Using Data Hub API you can search for entities and manage companies, contacts and interactions.

More guides can be found in the [docs](./docs/) folder.

To instantiate the development environment, please follow one of the three following sets of instructions depending on your use case:

## Installation with Docker

This project uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development.

**Note for Mac Users:** By default, docker on Mac will restrict itself to using just 2GB of memory. This [should be increased](https://docs.docker.com/docker-for-mac/#resources) to at least 4GB to avoid running in to unexpected problems.

1.  Clone the repository:

    ```shell
    git clone https://github.com/uktrade/data-hub-api
    cd data-hub-api
    ```

2.  Create `.env` files from `sample.env`

    ```shell
    cp sample.env .env
    cp config/settings/sample.env config/settings/.env
    ```

    If you're working with data-hub-frontend and mock-sso, `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` should be the same as
    `MOCK_SSO_EMAIL_USER_ID` in mock-sso environment definition
    in [data-hub-frontend/docker-compose.frontend.yml](https://github.com/uktrade/data-hub-frontend/blob/main/docker-compose.frontend.yml)

3.  Build and run the necessary containers for the required environment:

    ```shell
    docker-compose up
    ```

    or

    ```shell
    make start-dev
    ```

    - It will take time for the API container to come up - it will run
      migrations on both DBs, load initial data, sync opensearch etc. Watch
      along in the api container's logs.
    - **NOTE:**
      If you are using a linux system, the opensearch container may not
      come up successfully (`data-hub-api_es_1`) - it might be perpetually
      restarting.
      If the logs for that container mention something like
      `max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]`,
      you will need to run the following on your host machine:

      ```shell
      sudo sysctl -w vm.max_map_count=262144
      ```

      and append/modify the `vm.max_map_count` setting in `/etc/sysctl.conf` (so
      that this setting persists after restart):

      ```shell
      vm.max_map_count=262144
      ```

      For more information, [see the opensearch docs on vm.max_map_count](https://opensearch.org/docs/latest/opensearch/install/important-settings/).

4.  Optionally, you may want to run a local copy of the data hub frontend.
    By default, you can run both the API and the frontend under one docker-compose
    project. [See the instructions in the frontend readme to set it up](https://github.com/uktrade/data-hub-frontend/#running-project-within-docker).
    Alternatively use the `make` command documented below if you also want to bring up `dnb-service`

### Installation with docker of data-hub-api, data-hub-frontend and dnb-service on same network

There is now a `make` command to bring up the three environments on a single docker network, allowing the services to talk to each other effortlessly

1. Clone the repositories

   ```shell
   git clone https://github.com/uktrade/data-hub-api
   git clone https://github.com/uktrade/data-hub-frontend
   git clone https://github.com/uktrade/dnb-service
   cd data-hub-api
   ```

2. Create `.env` files from `sample.env`

   ```shell
   cp sample.env .env
   cp config/settings/sample.env config/settings/.env
   ```

   Ensure `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` is the same as `MOCK_SSO_EMAIL_USER_ID`
   in mock-sso environment definition in [data-hub-frontend/docker-compose.frontend.yml](https://github.com/uktrade/data-hub-frontend/blob/main/docker-compose.frontend.yml)
   and `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` the same as `DJANGO_SUPERUSER_EMAIL` in
   data-hub-api .env file otherwise the user may not exist

   The `DNB_SERVICE_BASE_URL` should match the dnb-service domain you are trying to access (localhost, staging, uat etc). The `DNB_SERVICE_TOKEN` is a token generated by dnb-service using the django-rest-framework token authentication. For localhost, you will need to generate a token on dnb-service for your user. For staging, uat or prod, the current tokens can be found in vault which are associated with an already existing api user. See the [dnb-service README](https://github.com/uktrade/dnb-service/blob/06fa20c7b594c4c01f0d23556379754c3b724ea2/README.md#creating-the-dnb_service_token-for-data-hub-api) for more information on how to find or generate new tokens.

3. Run make command

   ```shell
   make start-frontend-api-dnb
   ```

   - It will take time for the API container to come up - it will run
     migrations on both DBs, load initial data, sync opensearch etc. Watch
     along in the api container's logs.
   - **NOTE:**
     If you are using a linux system, the opensearch container may not
     come up successfully (`data-hub-api_es_1`) - it might be perpetually
     restarting.
     If the logs for that container mention something like
     `max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]`,
     you will need to run the following on your host machine:

   ```shell
   sudo sysctl -w vm.max_map_count=262144
   ```

   and append/modify the `vm.max_map_count` setting in `/etc/sysctl.conf` (so
   that this setting persists after restart):

   ```shell
   vm.max_map_count=262144
   ```

   For more information, [see the opensearch docs on vm.max_map_count](https://opensearch.org/docs/latest/opensearch/install/important-settings/).

4. If you want to stop all the services, run the following make command

   ```shell
   make stop-frontend-api-dnb
   ```

## Native installation (without Docker)

Dependencies:

- Python 3.10.x
- PostgreSQL 12
- redis 6.x
- OpenSearch 1.x

1.  Clone the repository:

    ```shell
    git clone https://github.com/uktrade/data-hub-api
    cd data-hub-api
    ```

2.  Install Python 3.10.

    [See this guide](https://docs.python-guide.org/starting/installation/) for detailed instructions for different platforms.

3.  Install system dependencies:

    On Ubuntu:

    ```shell
    sudo apt install build-essential libpq-dev python3.10-dev python3.10-venv
    ```

    On macOS:

    ```shell
    brew install libpq
    ```

4.  Install _postgres_, if not done already, as this is required by **psycopg2** in the requirements below

    On Ubuntu:

    ```shell
    sudo apt install postgresql postgresql-contrib
    ```

    On macOS:

    ```shell
    brew install postgresql
    ```

5.  Create and activate the virtualenv:

    ```shell
    python3.10 -m venv env
    source env/bin/activate
    pip install -U pip
    or
    formally to make sure you have the same version as what is used for cloudfoundry, use buildpack to install the same version e.g. https://github.com/cloudfoundry/python-buildpack/releases e.g. 22.1.2
    python -m pip install pip==22.1.2
    ```

6.  Install the dependencies:

    ```shell
    pip install -r requirements-dev.txt
    ```

7.  Create an `.env` settings file (it’s gitignored by default):

    ```shell
    cp config/settings/sample.env config/settings/.env
    ```

8.  Set `DOCKER_DEV=False` and `LOCAL_DEV=True` in `.env`

9.  Make sure you have OpenSearch running locally. If you don't, you can run one in Docker:

    ```shell
    docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "plugins.security.disabled=true" opensearchproject/opensearch:1.2.4
    ```

10. Make sure you have redis running locally and that the REDIS_BASE_URL in your `.env` is up-to-date.

11. Populate the databases and initialise OpenSearch:

    ```shell
    ./manage.py migrate
    ./manage.py migrate_search

    ./manage.py loadinitialmetadata
    ./manage.py createinitialrevisions
    ```

12. Optionally, you can load some test data:

    ```shell
    ./manage.py loaddata fixtures/test_data.yaml
    ```

    Note that this will queue RQ tasks to index the created records in OpenSearch,
    and hence the loaded records won‘t be returned by search endpoints until RQ is
    started and the queued tasks have run.

13. Create a superuser:

    ```shell
    ./manage.py createsuperuser
    ```

    (You can enter any valid email address as the username and SSO email user ID.)

14. Start the server:

    ```shell
    ./manage.py runserver
    ```

15. Start RQ (Redis Queue):

    ```shell
    python rq/rq-worker.py
    ```

    Note that in production the cron-scheduler:1/1, short-running-working:4/4, long-running-worker:4/4 are run in separate instances .

## API documentation

Automatically-generated API documentation is served at `/docs` (requires admin site credentials).

## Local development

If using Docker, prefix these commands with `docker-compose run api`.

To run the tests:

```shell
./tests.sh
```

To run the tests in parallel, pass `-n <number of processes>` to `./tests.sh`. For example, for four processes:

```shell
./tests.sh -n 4
```

By default all tests are executed. To skip tests that should not run under certain conditions ie. are dependent on external services, use a marker `excluded` or a marker that start with `excluded_`.

```python
import pytest

@pytest.mark.excluded
class TestClass:
    def test_1(self):
        ...

@pytest.mark.excluded
def test_1():
    ...

@pytest.mark.excluded_dependent_on_redis
def test_2():
    ...
```

To exclude such tests from running use one of the options below:

```shell
pytest --skip-excluded
pytest -m excluded_dependent_on_redis
pytest -m "excluded_x or excluded_y"
pytest -m "not (excluded_x or excluded_y)"
```

**NOTE:** When testing, using the `--reuse-db` can speed up test runs by excluding migrations

To run the linter:

```shell
flake8
```

There is also a pre-commit hook for Flake8. To install this, run:

```shell
make setup-flake8-hook
```

### Obtaining an API access token

You can obtain an access token for local development work in one of two ways:

- by running `./manage.py add_access_token <SSO email user ID>` with the SSO
  email user ID of an existing adviser (run
  `./manage.py add_access_token --help` for a list of options)
- using the form on `http://localhost:8000/admin/add-access-token/`

(If you’re using Docker, an access token will be created automatically if
certain environment variables are set. See `sample.env` for more details.)

This access token can be used with most endpoints by setting an
`Authorization` header value of `Bearer <access token>`.

Note that machine-to-machine endpoints (such as those under `/v4/metadata/`)
instead use Hawk authentication and request signing.

## Granting access to the front end

The [internal front end](https://github.com/uktrade/data-hub-frontend) uses single sign-on. You should configure the API as follows to use with the front end:

- `SSO_ENABLED`: `True`
- `STAFF_SSO_BASE_URL`: URL of a [Staff SSO](https://github.com/uktrade/staff-sso) or [Mock SSO](https://github.com/uktrade/mock-sso) instance. This should be the same server the front end is configured to use.
- `STAFF_SSO_AUTH_TOKEN`: Access token for Staff SSO.

## Granting access to machine-to-machine clients

Pure machine-to-machine clients use Hawk authentication with separate credentials for each client.

There are separate views for such clients as these views don’t expect `request.user` to be set.

Hawk credentials for each client are defined in settings below and each client is assigned scopes in `config/settings/common.py`.

These scopes define which views each client can access.

## Deployment

Data Hub API can run on any Heroku-style platform. Configuration is performed via the following environment variables:

| Variable name                                                         | Required                                    | Description                                                                                |
| --------------------------------------------------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ACTIVITY_STREAM_ACCESS_KEY_ID`                                       | No                                          | A non-secret access key ID, corresponding to `ACTIVITY_STREAM_SECRET_ACCESS_KEY`. The holder of the secret key can access the activity stream endpoint by Hawk authentication.   |
| `ACTIVITY_STREAM_SECRET_ACCESS_KEY`                                   | If `ACTIVITY_STREAM_ACCESS_KEY_ID` is set   | A secret key, corresponding to `ACTIVITY_STREAM_ACCESS_KEY_ID`. The holder of this key can access the activity stream endpoint by Hawk authentication.                           |
| `ACTIVITY_STREAM_OUTGOING_URL`                                        | No                                          | The URL used to read from activity stream                                                                                |
| `ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID`                              | No                                          | A non-secret access key ID, corresponding to `ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY`. This is used when reading from the activity stream at `ACTIVITY_STREAM_OUTGOING_URL`. |
| `ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY`                          | No                                          | A secret key, corresponding to `ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID`. This is used when reading from the activity stream at `ACTIVITY_STREAM_OUTGOING_URL`.                   |
| `ADMIN_OAUTH2_ENABLED`                                                | Yes                                         | Enables Django Admin SSO login when is True.                                                                                |
| `ADMIN_OAUTH2_BASE_URL`                                               | If `ADMIN_OAUTH2_ENABLED` is set            | A base URL of OAuth provider.                                                                                |
| `ADMIN_OAUTH2_TOKEN_FETCH_PATH`                                       | If `ADMIN_OAUTH2_ENABLED` is set            | OAuth fetch token path for Django Admin SSO login.                                                                                |
| `ADMIN_OAUTH2_USER_PROFILE_PATH`                                      | If `ADMIN_OAUTH2_ENABLED` is set            | OAuth user profile path for Django Admin SSO login.                                                                                |
| `ADMIN_OAUTH2_AUTH_PATH`                                              | If `ADMIN_OAUTH2_ENABLED` is set            | OAuth auth path for Django Admin SSO login.                                                                                |
| `ADMIN_OAUTH2_CLIENT_ID`                                              | If `ADMIN_OAUTH2_ENABLED` is set            | OAuth client ID for Django Admin SSO login.                                                                                |
| `ADMIN_OAUTH2_CLIENT_SECRET`                                          | If `ADMIN_OAUTH2_ENABLED` is set            | OAuth client secret for Django Admin SSO login.                                                                                |
| `AV_V2_SERVICE_URL`                                                   | Yes                                         | URL for ClamAV V2 service. If not configured, virus scanning will fail.                                                                                |
| `AWS_ACCESS_KEY_ID`                                                   | No                                          | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials).                                              |
| `AWS_DEFAULT_REGION`                                                  | No                                          | [Default region used by boto3.](http://boto3.readthedocs.io/en/latest/guide/configuration.html#environment-variable-configuration)                                               |
| `AWS_SECRET_ACCESS_KEY`                                               | No                                          | Used as part of [boto3 auto-configuration](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials).                                              |
| `COMPANY_MATCHING_SERVICE_BASE_URL`                                   | No                                          | The base url of the company matching service (default=None).                                                                                |
| `COMPANY_MATCHING_HAWK_ID`                                            | No                                          | The hawk id to use when making a request to the company matching service (default=None).                                                                                |
| `COMPANY_MATCHING_HAWK_KEY`                                           | No                                          | The hawk key to use when making a request to the company matching service (default=None).                                                                                |
| `CONSENT_SERVICE_BASE_URL`                                            | No                                          | The base url of the consent service, to post email consent preferences to (default=None).                                                                                |
| `CONSENT_SERVICE_HAWK_ID`                                             | No                                          | The hawk id to use when making a request to the consent service (default=None).                                                                                |
| `CONSENT_SERVICE_HAWK_KEY`                                            | No                                          | The hawk key to use when making a request to the consent service (default=None).                                                                                |
| `CSRF_COOKIE_HTTPONLY`                                                | No                                          | Whether to use HttpOnly flag on the CSRF cookie (default=False).                                                                                |
| `CSRF_COOKIE_SECURE`                                                  | No                                          | Whether to use a secure cookie for the CSRF cookie (default=False).                                                                                |
| `DATA_FLOW_API_ACCESS_KEY_ID`                                         | No                                          | A non-secret access key ID, corresponding to `DATA_FLOW_API_SECRET_ACCESS_KEY`. The holder of the secret key can access the omis-dataset endpoint by Hawk authentication.        |
| `DATA_FLOW_API_SECRET_ACCESS_KEY`                                     | If `DATA_FLOW_API_ACCESS_KEY_ID` is set     | A secret key, corresponding to `DATA_FLOW_API_ACCESS_KEY_ID`. The holder of this key can access the omis-dataset endpoint by Hawk authentication.                                |
| `DATABASE_CONN_MAX_AGE`                                               | No                                          | [Maximum database connection age (in seconds).](https://docs.djangoproject.com/en/2.0/ref/databases/)                                                                            |
| `DATABASE_URL`                                                        | Yes                                         | PostgreSQL server URL (with embedded credentials).                                                                                |
| `DATAHUB_FRONTEND_BASE_URL`                                           | Yes                                         |                                                                                |
| `DATAHUB_NOTIFICATION_API_KEY`                                        | No                                          | The GOVUK notify API key to use for the `datahub.notification` django app.                                                                                |
| `DATAHUB_SUPPORT_EMAIL_ADDRESS`                                       | No                                          | Email address for DataHub support team.                                                                                |
| `DATA_HUB_FRONTEND_ACCESS_KEY_ID`                                     | No                                          | A non-secret access key ID, corresponding to `DATA_HUB_FRONTEND_SECRET_ACCESS_KEY`. The holder of the secret key can access the metadata endpoints by Hawk authentication.       |
| `DATA_HUB_FRONTEND_SECRET_ACCESS_KEY`                                 | If `DATA_HUB_FRONTEND_ACCESS_KEY_ID` is set | A secret key, corresponding to `METADATA_ACCESS_KEY_ID`. The holder of this key can access the metadata endpoints by Hawk authentication.                                        |
| `DEBUG`                                                               | Yes                                         | Whether Django's debug mode should be enabled.                                                                                |
| `DIT_EMAIL_DOMAIN_*`                                                  | No                                          | An allowable DIT email domain for email ingestion along with it's allowed email authentication methods. Django-environ dict format e.g. example.com=dmarc:pass implementation either  spf:pass or dkim:pass               |
| `DIT_EMAIL_INGEST_BLOCKLIST`                                          | No                                          | A list of emails for which email ingestion is prohibited.                                                                                |
| `DJANGO_SECRET_KEY`                                                   | Yes                                         |                                                                                |
| `DJANGO_SENTRY_DSN`                                                   | Yes                                         |                                                                                |
| `DJANGO_SETTINGS_MODULE`                                              | Yes                                         |                                                                                |
| `DNB_AUTOMATIC_UPDATE_LIMIT`                                          | No                                          | Integer of the maximum number of updates the DNB automatic update task should ingest before exiting. This is unlimited if this setting is not set.                               |
| `DNB_SERVICE_BASE_URL`                                                | No                                          | The base URL of the DNB service.                                                                                |
| `DNB_SERVICE_TOKEN`                                                   | No                                          | Access token provided by dnb-service through it's django-rest-framework token authentication.                                                                                |
| `DEFAULT_BUCKET`                                                      | Yes                                         | S3 bucket for object storage.                                                                                |
| `DISABLE_PAAS_IP_CHECK`                                               | No                                          | Disable PaaS IP check for Hawk endpoints (default=False).                                                                                |
| `ENABLE_ADMIN_ADD_ACCESS_TOKEN_VIEW`                                  | No                                          | Whether to enable the add access token page for superusers in the admin site (default=True).                                                                                |
| `ENABLE_AUTOMATIC_REMINDER_ITA_USER_MIGRATIONS`                       | No                                          | Whether to enable automatic migration of ITA users to receive notification reminders                                                                                |
| `ENABLE_AUTOMATIC_REMINDER_POST_USER_MIGRATIONS`                      | No                                          | Whether to enable automatic migration of POST users to receive notification reminders                                                                                |
| `ENABLE_DAILY_OPENSEARCH_SYNC`                                        | No                                          | Whether to enable the daily OpenSearch sync (default=False).                                                                                |
| `ENABLE_INVESTMENT_NOTIFICATION`                                      | No                                          | True or False. Whether or not to activate the RQ task for sending investment notifications                                                                                |
| `ENABLE_ESTIMATED_LAND_DATE_REMINDERS`                                | No                                          | True or False. Whether or not to activate the RQ task for sending investment notifications                                                                                |
| `ENABLE_ESTIMATED_LAND_DATE_REMINDERS_EMAIL_DELIVERY_STATUS`          | No                                          | True or False. Whether or not to activate the RQ task for checking delivery status                                                                                |
| `ENABLE_NEW_EXPORT_INTERACTION_REMINDERS`                             | No                                          | True or False. Whether or not to activate the RQ task for sending new export interaction notifications                                                                           |
| `ENABLE_NEW_EXPORT_INTERACTION_REMINDERS_EMAIL_DELIVERY_STATUS`       | No                                          | True or False. Whether or not to activate the RQ task for checking delivery status                                                                                |
| `ENABLE_NO_RECENT_EXPORT_INTERACTION_REMINDERS`                       | No                                          | True or False. Whether or not to activate the RQ task for sending no recent interaction notifications                                                                            |
| `ENABLE_NO_RECENT_EXPORT_INTERACTION_REMINDERS_EMAIL_DELIVERY_STATUS` | No                                          | True or False. Whether or not to activate the RQ task for checking delivery status                                                                                |
| `ENABLE_NO_RECENT_INTERACTION_REMINDERS`                              | No                                          | True or False. Whether or not to activate the RQ task for sending investment notifications                                                                                |
| `ENABLE_MAILBOX_PROCESSING`                                           | No                                          | True or False. Whether or not to activate the RQ task for mailbox processing                                                                                |
| `ENABLE_SLACK_MESSAGING`                                              | No                                          | If present and truthy, enable the transmission of messages to Slack. Necessitates the specification of the other env vars `SLACK_API_TOKEN` and `SLACK_MESSAGE_CHANNEL`          |
| `ENABLE_SPI_REPORT_GENERATION`                                        | No                                          | Whether to enable daily SPI report (default=False).                                                                                |
| `ES_APM_ENABLED`                                                      | Yes                                         | Enables Elasticsearch APM agent when is True.                                                                                |
| `ES_APM_SERVICE_NAME`                                                 | Yes, if ES_APM_ENABLED                      | A name of the running service. Must match following regexp: ^[a-zA-Z0-9 _-]+$.                                                                                |
| `ES_APM_SECRET_TOKEN`                                                 | Yes, if ES_APM_ENABLED                      | A secret token used to authorise requests to the APM server.                                                                                |
| `ES_APM_SERVER_TIMEOUT`                                               | No                                          | A timeout for requests to the Elasticsearch APM server in duration format (default=20s).                                                                                |
| `ES_APM_SERVER_URL`                                                   | Yes, if ES_APM_ENABLED                      | The URL of the Elasticsearch APM server.                                                                                |
| `ES_APM_ENVIRONMENT`                                                  | Yes, if ES_APM_ENABLED                      | A name of the environment the service is running, for example: `develop`.                                                                                |
| `EXPORT_WINS_SERVICE_BASE_URL`                                        | No                                          | The base url of the Export Wins API (default=None).                                                                                |
| `EXPORT_WINS_HAWK_ID`                                                 | No                                          | The hawk id to use when making a request to the Export Wins API (default=None).                                                                                |
| `EXPORT_WINS_HAWK_KEY`                                                | No                                          | The hawk key to use when making a request to the Export Wins API (default=None).                                                                                |
| `EXTRA_DJANGO_APPS`                                                   | Yes                                         | Additional Django apps to load (comma-separated). Can be used to reverse the migrations of a removed third-party app (see comment in config/settings/common.py for more detail). |
| `INTERACTION_ADMIN_CSV_IMPORT_MAX_SIZE`                               | No                                          | Maximum file size in bytes for interaction admin CSV uploads (default=2MB).                                                                                |
| `INTERACTION_NOTIFICATION_API_KEY`                                    | Yes                                         |                                                                                |
| `INVESTMENT_DOCUMENT_AWS_ACCESS_KEY_ID`                               | No                                          | Same use as AWS_ACCESS_KEY_ID, but for investment project documents.                                                                                |
| `INVESTMENT_DOCUMENT_AWS_SECRET_ACCESS_KEY`                           | No                                          | Same use as AWS_SECRET_ACCESS_KEY, but for investment project documents.                                                                                |
| `INVESTMENT_DOCUMENT_AWS_REGION`                                      | No                                          | Same use as AWS_DEFAULT_REGION, but for investment project documents.                                                                                |
| `INVESTMENT_DOCUMENT_BUCKET`                                          | No                                          | S3 bucket for investment project documents storage.                                                                                |
| `INVESTMENT_NOTIFICATION_ADMIN_EMAIL`                                 | Yes                                         |                                                                                |
| `INVESTMENT_NOTIFICATION_API_KEY`                                     | Yes                                         |                                                                                |
| `INVESTMENT_NOTIFICATION_ESTIMATED_LAND_DATE_TEMPLATE_ID`             | Yes                                         | An ID of Notify Template for Estimated Land Date notifications                                                                                |
| `INVESTMENT_NOTIFICATION_ESTIMATED_LAND_DATE_SUMMARY_TEMPLATE_ID`     | Yes                                         | An ID of Notify Template for Estimated Land Date summary notifications                                                                                |
| `INVESTMENT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID`           | Yes                                         | An ID of Notify Template for No Recent Investment Interaction notifications                                                                                |
| `EXPORT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID`               | Yes                                         | An ID of Notify Template for No Recent Export company Interaction notifications                                                                                |
| `EXPORT_NOTIFICATION_NO_INTERACTION_TEMPLATE_ID`                      | Yes                                         | An ID of Notify Second template for No Recent Export company Interaction notifications                                                                                |
| `EXPORT_NOTIFICATION_NO_RECENT_INTERACTION_TEMPLATE_ID`               | Yes                                         | An ID of Notify Template for No Recent Export company Interaction notifications                                                                                |
| `EXPORT_WIN_NOTIFICATION_API_KEY`                                     | Yes                                         |                                                                                |
| `EXPORT_WIN_CLIENT_RECEIPT_TEMPLATE_ID`               | Yes                                         | An ID of Notify Template for Export Win Client Email Receipt notifications                                                                                |
| `EXPORT_WIN_LEAD_OFFICER_APPROVED_TEMPLATE_ID`               | Yes                                         | An ID of Notify Template for Export Win Lead Officer Approved notifications                                                                                |
| `EXPORT_WIN_LEAD_OFFICER_REJECTED_TEMPLATE_ID`               | Yes                                         | An ID of Notify Template for Export Win Lead Officer Rejected notifications                                                                                |
| `TASK_REMINDER_EMAIL_TEMPLATE_ID`                                    | No                                         | An ID of Notify Template for the generic Task reminder notifications                                                                                |
| `TASK_NOTIFICATION_FROM_OTHERS_TEMPLATE_ID`                           | Yes                                         | An ID of Notify Template for Task assigned by others notifications                                                                                |
| `MAILBOX_AWS_ACCESS_KEY_ID`                                           | No                                          | Same use as AWS_ACCESS_KEY_ID, but for mailbox.                                                                                |
| `MAILBOX_AWS_SECRET_ACCESS_KEY`                                       | No                                          | Same use as AWS_SECRET_ACCESS_KEY, but for mailbox.                                                                                |
| `MAILBOX_AWS_REGION`                                                  | No                                          | Same use as AWS_DEFAULT_REGION, but for mailbox.                                                                                |
| `MAILBOX_BUCKET`                                                      | No                                          | S3 bucket for mailbox storage.                                                                                |
| `MAILBOX_INGESTION_SUCCESS_TEMPLATE_ID`                               | No                                          | An ID of Notify Template for mailbox ingestion success                                                                                |
| `MAILBOX_INGESTION_FAILURE_TEMPLATE_ID`                               | No                                          | An ID of Notify Template for mailbox ingestion failure                                                                                |
| `MARKET_ACCESS_ACCESS_KEY_ID`                                         | No                                          | A non-secret access key ID used by the Market Access service to access Hawk-authenticated public company endpoints.                                                              |
| `MARKET_ACCESS_SECRET_ACCESS_KEY`                                     | If `MARKET_ACCESS_ACCESS_KEY_ID` is set     | A secret key used by the Market Access service to access Hawk-authenticated public company endpoints.                                                                            |
| `NOTIFICATION_SUMMARY_THRESHOLD`                                      | No                                          | Number of notification items that trigger sending a summary email. (default=5)                                                                                |
| `OMIS_PUBLIC_ACCESS_KEY_ID`                                           | No                                          | A non-secret access key ID, corresponding to `OMIS_PUBLIC_SECRET_ACCESS_KEY`. The holder of the secret key can access the OMIS public endpoints by Hawk authentication.          |
| `OMIS_NOTIFICATION_ADMIN_EMAIL`                                       | Yes                                         |                                                                                |
| `OMIS_NOTIFICATION_API_KEY`                                           | Yes                                         |                                                                                |
| `OMIS_NOTIFICATION_OVERRIDE_RECIPIENT_EMAIL`                          | No                                          |                                                                                |
| `OMIS_PUBLIC_BASE_URL`                                                | Yes                                         |                                                                                |
| `OMIS_PUBLIC_SECRET_ACCESS_KEY`                                       | If `OMIS_PUBLIC_ACCESS_KEY_ID` is set       | A secret key, corresponding to `OMIS_PUBLIC_ACCESS_KEY_ID`. The holder of this key can access the OMIS public endpoints by Hawk authentication.                                  |
| `OPENSEARCH_INDEX_PREFIX`                                             | Yes                                         | Prefix to use for indices and aliases                                                                                |
| `OPENSEARCH_SEARCH_REQUEST_TIMEOUT`                                   | No                                          | Timeout (in seconds) for searches (default=20).                                                                                |
| `OPENSEARCH_SEARCH_REQUEST_WARNING_THRESHOLD`                         | No                                          | Threshold (in seconds) for emitting warnings about slow searches (default=10).                                                                                |
| `OPENSEARCH_VERIFY_CERTS`                                             | No                                          |                                                                                |
| `PAAS_IP_ALLOWLIST`                                                   | No                                          | IP addresses (comma-separated) that can access the Hawk-authenticated endpoints.                                                                                |
| `REDIS_BASE_URL`                                                      | No                                          | redis base URL without the db                                                                                |
| `REDIS_CACHE_DB`                                                      | No                                          | redis db for django cache (default 0)                                                                                |
| `REPORT_AWS_ACCESS_KEY_ID`                                            | No                                          | Same use as AWS_ACCESS_KEY_ID, but for reports.                                                                                |
| `REPORT_AWS_SECRET_ACCESS_KEY`                                        | No                                          | Same use as AWS_SECRET_ACCESS_KEY, but for reports.                                                                                |
| `REPORT_AWS_REGION`                                                   | No                                          | Same use as AWS_DEFAULT_REGION, but for reports.                                                                                |
| `REPORT_BUCKET`                                                       | No                                          | S3 bucket for report storage.                                                                                |
| `SECTOR_ENVIRONMENT`                                                  | No                                         | Only set to "production", in production environment. Otherwise it can be left empty.                                                                                |
| `SENTRY_ENVIRONMENT`                                                  | Yes                                         | Value for the environment tag in Sentry.                                                                                |
| `SKIP_OPENSEARCH_MAPPING_MIGRATIONS`                                  | No                                          | If non-empty, skip applying OpenSearch mapping type migrations on deployment.                                                                                |
| `SLACK_API_TOKEN`                                                     | No                                          | (Required if `ENABLE_SLACK_MESSAGING` is truthy) Auth token for connection to Slack API for purposes of sending messages through the datahub.core.realtime_messaging module      |
| `SLACK_MESSAGE_CHANNEL`                                               | No                                          | (Required if `ENABLE_SLACK_MESSAGING` is truthy) Name (or preferably ID) of the channel into which datahub.core.realtime_messaging should send messages                          |
| `SSO_ENABLED`                                                         | Yes                                         | Whether single sign-on via RFC 7662 token introspection is enabled                                                                                |
| `STAFF_SSO_AUTH_TOKEN`                                                | If SSO enabled                              | Access token for the Staff SSO API.                                                                                |
| `STAFF_SSO_BASE_URL`                                                  | If SSO enabled                              | The base URL for the Staff SSO API.                                                                                |
| `STAFF_SSO_REQUEST_TIMEOUT`                                           | No                                          | Staff SSO API request timeout in seconds (default=5).                                                                                |
| `STATSD_HOST`                                                         | No                                          | StatsD host url.                                                                                |
| `STATSD_PORT`                                                         | No                                          | StatsD port number.                                                                                |
| `STATSD_PREFIX`                                                       | No                                          | Prefix for metrics being pushed to StatsD.                                                                                |
| `VCAP_SERVICES`                                                       | No                                          | Set by GOV.UK PaaS when using their backing services. Contains connection details for OpenSearch and Redis.                                                                      |

## Management commands

If using Docker, remember to run these commands inside your container by prefixing them with `docker-compose run api`.

### Database

#### Apply migrations

##### For the default database

```shell
./manage.py migrate
```

#### Create django-reversion initial revisions

If the database is freshly built or a new versioned model is added run:

```shell
./manage.py createinitialrevisions
```

#### Load initial metadata

These commands are generally only intended to be used on a blank database.

```shell
./manage.py loadinitialmetadata
```

### OpenSearch

#### Update indexes and mapping types

To create missing OpenSearch indexes and migrate modified mapping types:

```shell
./manage.py migrate_search
```

This will also resync data (using RQ) for any newly-created indexes.

See [docs/OpenSearch migrations.md](<docs/OpenSearch migrations.md>) for more detail about how the command works.

#### Resync all OpenSearch records

To resync all records using RQ:

```shell
./manage.py sync_search
```

To resync all records synchronously (without RQ running):

```shell
./manage.py sync_search --foreground
```

You can resync only specific models by using the `--model=` argument.

```shell
./manage.py sync_search --model=company --model=contact
```

For more details including all the available choices:

```shell
./manage.py sync_search --help
```

## Dependencies

See [Managing dependencies](<docs/Managing dependencies.md>) for information about installing,
adding and upgrading dependencies.

## Activity Stream

The `/v3/activity-stream/*` endpoints are protected by two mechanisms:

- IP address allowlisting via the `X-Forwarded-For` header, with a comma separated list of allowlisted IPs in the environment variable `PAAS_IP_ALLOWLIST`.

- Hawk authentication via the `Authorization` header, with the credentials in the environment variables `ACTIVITY_STREAM_ACCESS_KEY_ID` and `ACTIVITY_STREAM_SECRET_ACCESS_KEY`.

### IP address allowlisted

The authentication blocks requests that do not have a allowlisteded IP in the second-from-the-end IP in `X-Forwarded-For` header. In general, this cannot be trusted. However, in PaaS, this can be, and this is the only production environment. Ideally, this would be done at a lower level than HTTP, but this is not possible with the current architecture.

If making requests to this endpoint locally, you must manually add this header or disable the check using `DISABLE_PAAS_IP_CHECK` environment variable.

### Hawk authentication

In general, Hawk authentication hashing the HTTP payload and `Content-Type` header, and using a nonce, are both _optional_. Here, as with the Activity Stream endpoints in other DIT projects, both are _required_. `Content-Type` may be the empty string, and if there is no payload, then it should be treated as the empty string.

### Emails

Email templates are created in [GOV UK Notify](https://www.notifications.service.gov.uk/). You can create a personal account for testing new templates or template changes locally. To test locally you need to update the API Keys and template ID environment variables.

#### Updating emails

Only update the live email templates once your changes are pushed live. Extra parameters passed to email templates are ignored but not sending parameters required by an email template will cause the emails to error.

Example of environment variables for updating the interaction notification email template:

```
INTERACTION_NOTIFICATION_API_KEY=<noftify-api-key>
EXPORT_NOTIFICATION_NEW_INTERACTION_TEMPLATE_ID=<template-id>
```


# CP-Ansible

## Introduction

Ansible provides a simple way to deploy, manage, and configure the Confluent Platform services. This repository provides playbooks and templates to easily spin up a Confluent Platform installation. Specifically this repository:

* Installs Confluent Platform packages or archive.
* Starts services using systemd scripts.
* Provides configuration options for many security options including encryption, authentication, and authorization.

The services that can be installed from this repository are:

* ZooKeeper
* Kafka
* Schema Registry
* REST Proxy
* Confluent Control Center
* Kafka Connect (distributed mode)
* KSQL Server
* Replicator

## Documentation

You can find the documentation for running CP-Ansible at https://docs.confluent.io/current/installation/cp-ansible/index.html.

You can find supported configuration variables in [VARIABLES.md](docs/VARIABLES.md)

## Contributing

If you would like to contribute to the CP-Ansible project, please refer to the [CONTRIBUTE.md](docs/CONTRIBUTING.md)

## License

[Apache 2.0](docs/LICENSE.md)


# gflownet

gflownet is a library built upon [PyTorch](https://pytorch.org/) to easily train and extend [GFlowNets](https://arxiv.org/abs/2106.04399), also known as GFN or generative flow networks. GFlowNets are a machine learning framework for probabilistic and generative modelling, with a wide range of applications, especially in [scientific discovery](https://pubs.rsc.org/en/content/articlelanding/2023/dd/d3dd00002h) problems.

In a nutshell, GFlowNets can be regarded as a generative model designed to sample objects $x \in \mathcal{X}$ proportionally to a reward function $R(x)$. This results in the potential of sampling diverse objects with high rewards. For example, given the reward landscape depicted below, defined over a two-dimensional space, a well-trained GFlowNet will be able to sample from the four high-reward corners with high probability.

GFlowNets rely on the principle of **compositionality** to generate samples. A meaningful decomposition of samples $x$ into multiple intermediate states $s_0\rightarrow s_1 \rightarrow \dots \rightarrow x$ can yield generalisable patterns. These patterns can then be learned by neural networks trained to model the value of transitions $F_{\theta}(s_t \rightarrow s_{t+1})$.

Consider the problem of generating [Tetris](https://en.wikipedia.org/wiki/Tetris)-like boards. A natural decomposition of the sample generation process would be to add one piece at a time, starting from an empty board. For any state representing a board with pieces, we could identify its valid parents and children, as illustrated in the figure below.

We could define a reward function $R(x)$ as the number of cells occupied by pieces, for instance. The goal of training a GFlowNet on this task would be to discover (sample) diverse solutions (boards with pieces) with high rewards. This represents an intuitive yet complex problem where GFlowNets can be used, which is [implemented in this library](https://github.com/alexhernandezgarcia/gflownet/blob/ahg/313-update-readme-components/gflownet/envs/tetris.py). Many problems in scientific discoveries, such as the inverse design of proteins, molecules, or crystals share similarties with this intuitive task.

## Main Components of the GFlowNet Library

The GFlowNet library comprises four core components: environment, proxy, policy models (forward and backward), and GFlowNet agent.

### Environment

The environment defines the state space $\mathcal{S}$ and action space $\mathbb{A} of a particular problem, for example the Tetris task. To illustrate the environment, let's consider an even simpler environment currently implemented in the library: the [Scrabble](https://en.wikipedia.org/wiki/Scrabble) environment, inspired by the popular board game.

The Scrabble environment simulates a simple letter arrangement game where words are constructed by adding one letter at a time, up to a maximum sequence length (typically 7). Therefore, the action space is the set of all English letters plus a special end-of-sequence (EOS) action; and the state space is the set of all possible words with up to 7 letters. We can represent each `state` as a list of indices corresponding to the letters, padded with zeroes to the maximum length. For example, the state for the word "CAT" would be represented as `[3, 1, 20, 0, 0, 0, 0]`. Actions in the Scrabble environment are single-element tuples containing the index of the letter, plus the end-of-sequence (EOS) action `(-1,)`.

Using the gflownet library for a new task will typically require implementing your own environment. The library is particularly designed to make such extensions as easy as possible. In the documentation, we show how to do it step by step. You can also watch [this live-coding tutorial](https://www.youtube.com/watch?v=tMVJnzFqa6w&t=5h22m35s) on how to code the Scrabble environment.

### Proxy

We use the term "[proxy](gflownet/proxy/base.py)" to refer to the function or model that provides the rewards for the states of an environment. In other words, In the context of GFlowNets, the proxy can be thought of as a function $E(x)$ from which the reward is derived: $R(x) = g(E(x))$, where $g$ is a function that transforms the proxy values into non-zero rewards, that is "the higher the reward the better". For example, we can implement a proxy that simulates the scores of a word in the Scrabble game. That is, the [`ScrabbleScorer`](gflownet/proxy/scrabble.py) proxy computes the sum of the score of each letter of a word. For the word "CAT" that is $E(x) = 3 + 1 + 1 = 5$. While in many environments the proxy functions is a simple scorer, more complex settings like molecule or [crystal generation](gflownet/proxy/crystals/dave.py) may be use proxies that represent the energy or a property predicted by a pre-trained machine learning model.

Adapting the gflownet library for a new task will also likely require implementing your own proxy, which is usually fairly simple, as illustrated in the documentation.

### Policy models

The policy models are neural networks that model the forward and backward transitions between states, $F_{F_{\theta}}(s_t \rightarrow s_{t+1})$ (forward) and $F_{B_{\theta}}(s_{t+1} \rightarrow s_t)$ (backward). These models take a state as input and output a distribution over the actions in the action space. For continuous environments, the outputs are the parameters of a probability distribution to sample continuous-valued actions. For many tasks, simple multi-layer perceptrons with a few layers do the job, but technically any architecture could be used as policy model.

### GFlowNet Agent

The GFlowNet Agent is the central component that ties all others together. It orchestrates the interaction between the environment, policies, and proxy, as well as other auxiliary components such as the Evaluator and the Logger. The GFlowNet can construct training batches by sampling trajectories, optimise the policy models via gradient descent, compute evaluation metrics, log data to [Weights & Biases](https://wandb.ai/), etc. The agent can be configured to optimise any of the following loss functions implemented in the library: flow matching (FM), trajectory balance (TB), and detailed balance (TB) and forward-looking (FL).

#### Exploring the Scrabble Environment

To better understand the GFlowNet components, let us explore the Scrabble environment in more detail below.

When initializing any GFlowNet agent, it's useful to explore the properties of the environment. The library offers various functionalities for this purpose. Below are some detailed examples, among others:

1. Checking the Initial State

You can observe the initial state of the environment. For Scrabble environment, this would be an empty board or sequence:

```python
env.state
>>> [0, 0, 0, 0, 0, 0, 0]
```

2. Exploring the Action Space
```python
env.get_action_space()
>>> [(1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,), (11,), (12,), (13,), (14,), (15,), (16,), (17,), (18,), (19,), (20,), (21,), (22,), (23,), (24,), (25,), (26,), (-1,)]
```
For Scrabble environment, the action space is all english alphabet letters indexed from 1 to 26. The action (-1,) represents the end-of-sequence (EOS) action, indicating the termination of word formation.

3. Taking a Random Step

```python
new_state, action_taken, valid = env.step_random()
print("New State:", new_state)
print("Action Taken:", action_taken)
print("Action Valid:", valid)

>>> New State: [24, 0, 0, 0, 0, 0, 0]
>>> Action Taken: (24,)
>>> Action Valid: True
```

This function randomly selects a valid action (adding a letter or ending the sequence) and applies it to the environment. The output shows the new state, the action taken, and whether the action was valid.

4. Performing a Specific Action

```python
action = (1,)  # Action to add 'A'
new_state, performed_action, is_valid = env.step(action)
print("Updated State:", new_state)
print("Performed Action:", performed_action)
print("Was the Action Valid:", is_valid)
>>> Updated State: [24, 1, 0, 0, 0, 0, 0]
>>> Performed Action: (1,)
>>> Was the Action Valid: True
```

5. Displaying the State as a human readable

```python
env.state2readable(env.state)
>>> 'X A'
```

6. Interpreting Actions as a human readable

```python
print("Action Meaning:", env.idx2token[action[0]])
>>> Action Meaning: A
```

7. Sampling a Random Trajectory

```python
new_state, action_sequence = env.trajectory_random()
print("New State:", new_state)
print("Action Sequence:" action_sequence)

>>> New State: [16, 16, 17, 20, 11, 16, 0]
>>> Action Sequence: [(16,), (16,), (17,), (20,), (11,), (16,), (-1,)]
```

8. Reset enviroment

```python
env.reset()
env.state
>>> [0, 0, 0, 0, 0, 0, 0]
```

So far, we've discussed how to manually set actions or use random actions in the GFlowNet environment. This approach is useful for testing or understanding the basic mechanics of the environment. However, in practice, the goal of a GFlowNet agent is to learn from its experiences to take increasingly effective actions that are driven by a learned policy.

As the agent interacts with the environment, it collects data about the outcomes of its actions. This data is used to train a policy network, which models the probability distribution of possible actions given the current state. Over time, the policy network learns to favor actions that lead to more successful outcomes with higher reward, optimizing the agent's performance.

9. Sample a batch of trajectories from a trained agent

```python
batch, _ = gflownet.sample_batch(n_forward=3,  train=False)
batch.states
>>> [[20, 20, 21, 3, 0, 0, 0], [12, 16, 8, 6, 14, 11, 20], [17, 17, 16, 23, 20, 16, 24]]
```

We can convert the first state to human readable:

```python
env.state2readable(batch.states[0])
>>> 'T T U C'
```

We can also compute the rewards and the proxy for all states or single state.

```python
proxy(env.states2proxy(batch.states))
>>> tensor([ 6., 19., 39.])
```
Or single state

```python
proxy(env.state2proxy(batch.states[0]))
>>> tensor([6.])
```

The `state2proxy` and `states2proxy` are helper functions that transform the input to appropriate format. For example to tensor.

We can also compute the rewards, and since our transformation function `g` is the identity, the rewards should be equal to the proxy directly.

```python
proxy.rewards(env.states2proxy(batch.states))
>>> tensor([ 6., 19., 39.])
```

## Installation

**Quickstart: If you simply want to install everything, run `setup_all.sh`.**

+ This project **requires** `python 3.10` and `cuda 11.8`.
+ Setup is currently only supported on Ubuntu. It should also work on OSX, but you will need to handle the package dependencies.
+ The recommend installation is as follows:

```bash
python3.10 -m venv ~/envs/gflownet  # Initalize your virtual env.
source ~/envs/gflownet/bin/activate  # Activate your environment.
./prereq_ubuntu.sh  # Installs some packages required by dependencies.
./prereq_python.sh  # Installs python packages with specific wheels.
./prereq_geometric.sh  # OPTIONAL - for the molecule environment.
pip install .[all]  # Install the remaining elements of this package.
```

Aside from the base packages, you can optionally install `dev` tools using this tag, `materials` dependencies using this tag, or `molecules` packages using this tag. The simplest option is to use the `all` tag, as above, which installs all dependencies.

## How to train a GFlowNet model

The configuration is handled via the use of [Hydra](https://hydra.cc/docs/intro/). To train a GFlowNet model with the default configuration, simply run

```bash
python main.py user.logdir.root=<path/to/log/files/>
```

Alternatively, you can create a user configuration file in `config/user/<username>.yaml` specifying a `logdir.root` and run

```bash
python main.py user=<username>
```

Using Hydra, you can easily specify any variable of the configuration in the command line. For example, to train GFlowNet with the trajectory balance loss, on the continuous torus (`ctorus`) environment and the corresponding proxy:

```bash
python main.py gflownet=trajectorybalance env=ctorus proxy=torus
```

The above command will overwrite the `env` and `proxy` default configuration with the configuration files in `config/env/ctorus.yaml` and `config/proxy/torus.yaml` respectively.

Hydra configuration is hierarchical. For instance, You can seamlessly modify exisiting flag or variable in the configuration by setting `logger.do.online=False`. For more, feel free to read the [Hydra documentation](https://hydra.cc/docs/intro/).

Note that by default, PyTorch will operate on the CPU because we have not observed performance improvements by running on the GPU. You may run on GPU with `device=cuda`.

## GFlowNet loss functions

Currently, the implementation includes the following GFlowNet losses:

- [Flow-matching (FM)](https://arxiv.org/abs/2106.04399): `gflownet=flowmatch`
- [Trajectory balance (TB)](https://arxiv.org/abs/2201.13259): `gflownet=trajectorybalance`
- [Detailed balance (DB)](https://arxiv.org/abs/2201.13259): `gflownet=detailedbalance`
- [Forward-looking (FL)](https://arxiv.org/abs/2302.01687): `gflownet=forwardlooking`

## Logger

The library also has Logger class which helps to manage all logging activities during the training and evaluation of the network. It captures and stores logs to track the model's performance and debugging information. For instance, it logs details such as training progress, performance metrics, and any potential errors or warnings that occur. It also integrates to [wandb.ai](https://wandb.ai) providing a cloud-based platform for logging the train and evaluation metrics to [wandb.ai](https://wandb.ai). The WandB is disabled by default. In order to enable it, set the configuration variable `logger.do.online` to `True`.

## Contributors

Many wonderful scientists and developers have contributed to this repository: [Alex Hernandez-Garcia](https://github.com/alexhernandezgarcia), [Nikita Saxena](https://github.com/nikita-0209), [Alexandra Volokhova](https://github.com/AlexandraVolokhova), [Michał Koziarski](https://github.com/michalkoziarski), [Divya Sharma](https://github.com/sh-divya), [Pierre Luc Carrier](https://github.com/carriepl) and [Victor Schmidt](https://github.com/vict0rsch).

## Research

This repository has been used in at least the following research articles:

- Lahlou et al. [A theory of continuous generative flow networks](https://proceedings.mlr.press/v202/lahlou23a/lahlou23a.pdf). ICML, 2023.
- Hernandez-Garcia, Saxena et al. [Multi-fidelity active learning with GFlowNets](https://arxiv.org/abs/2306.11715). RealML at NeurIPS 2023.
- Mila AI4Science et al. [Crystal-GFN: sampling crystals with desirable properties and constraints](https://arxiv.org/abs/2310.04925). AI4Mat at NeurIPS 2023 (spotlight).
- Volokhova, Koziarski et al. [Towards equilibrium molecular conformation generation with GFlowNets](https://arxiv.org/abs/2310.14782). AI4Mat at NeurIPS 2023.

## Cite

Bibtex Format

```text
@misc{hernandez-garcia2024,
  author = {Hernandez-Garcia, Alex and Saxena, Nikita and Volokhova, Alexandra and Koziarski, Michał and Sharma, Divya and Viviano, Joseph D and Carrier, Pierre Luc and Schmidt, Victor},
  title  = {gflownet},
  url    = {https://github.com/alexhernandezgarcia/gflownet},
  year   = {2024},
}
```

Or [CFF file](./CITATION.cff)


# scielo.packtools

Python library and command-line utilities to handle SciELO PS XML files that
runs on python > 3.9.

## Build status

[![Build Status](https://travis-ci.com/gustavofonseca/packtools.svg?branch=master)](https://travis-ci.com/gustavofonseca/packtools)

[![Latest Documentation Status](https://readthedocs.org/projects/packtools/badge/?version=latest)](https://packtools.readthedocs.io/en/latest/)

## Installation

``packtools`` depends on [lxml](http://lxml.de/installation.html).

Python Package Index (recommended):

```bash
$ pip install packtools
```

or just:

```bash
pip install -r requirements.txt
```

Pip + git (versão de desenvolvimento):

```bash
$ pip install -e git+git://github.com/scieloorg/packtools.git#egg=packtools
```

Source-code:

```bash
$ git clone https://github.com/scieloorg/packtools.git
$ cd packtools
$ python setup.py install
```

Installation as a web application, where a graphical interface for `stylechecker`
and an HTML previewer is provided:

```bash
$ pip install packtools[webapp]
```

or

```bash
pip install -r requirements.txt && pip install -r optional-requirements.txt
```

## Tests

To run tests execute:

```bash
python setup.py test
```

To run a specific module of tests, type:

```bash
python setup.py test -s tests.test_htmlgenerator
```

Run tests with tox:

```
tox
```

## Command line execute

```bash
htmlgenerator example.xml --loglevel=error --nochecks --nonetwork
```

It will generate the result in the same path of the .xml file.

## Running the web application

Configuring the application:

| environment variable | default value                                    |
|----------------------|--------------------------------------------------|
| APP_SETTINGS         | packtools.webapp.config.default.ProductionConfig |

```bash
$ export APP_SETTINGS=packtools.webapp.config.default.ProductionConfig
$ export FLASK_APP=packtools.webapp.app.py
$ flask run
```

## Documentation

http://packtools.readthedocs.org/ (we need help!)

## Use license

Copyright 2013 SciELO <scielo-dev@googlegroups.com>. Licensed under the terms
of the BSD license. Please see LICENSE in the source code for more
information.

https://github.com/scieloorg/packtools/blob/master/LICENSE

## Changelog

https://github.com/scieloorg/packtools/blob/master/HISTORY.md


Ayon server backend
===================

This repository contains Python backend code for Ayon server.

Visit [AYON documentation](https://ayon.ynput.io/docs/system_introduction/) for more information
or check out [ayon-docker](https://github.com/ynput/ayon-docker) repository for the reference Docker setup.


<!--
 Copyright 2024 Google LLC

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

      https://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 -->

[![Unit Tests](https://github.com/google/maxtext/actions/workflows/UnitTests.yml/badge.svg)](https://github.com/google/maxtext/actions/workflows/UnitTests.yml)

# Overview

MaxText is a **high performance**, **highly scalable**, **open-source** LLM written in pure Python/Jax and targeting Google Cloud TPUs and GPUs for **training** and **inference**. MaxText achieves [high MFUs](#runtime-performance-results) and scales from single host to very large clusters while staying simple and "optimization-free" thanks to the power of Jax and the XLA compiler.

MaxText aims to be a launching off point for ambitious LLM projects both in research and production. We encourage users to start by experimenting with MaxText out of the box and then fork and modify MaxText to meet their needs.

We have used MaxText to [demonstrate high-performance, well-converging training in int8](https://cloud.google.com/blog/products/compute/accurate-quantized-training-aqt-for-tpu-v5e) and [scale training to ~51K chips](https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e).

Key supported features:
* TPUs and GPUs (in preview)
* Training and Inference (in preview)
* Models: Llama2, Mistral and Gemma

# Table of Contents

* [Getting Started](getting_started/First_run.md)
* [Runtime Performance Results](#runtime-performance-results)
* [Comparison To Alternatives](#comparison-to-alternatives)
* [Development](#development)
* [Features and Diagnostics](#features-and-diagnostics)

# Getting Started

For your first time running MaxText, we provide specific [instructions](getting_started/First_run.md).

MaxText supports training and inference of various open models. Follow user guides in the [getting started](getting_started) folder to know more.

Some extra helpful guides:
* [Gemma](https://ai.google.dev/gemma): a family of open-weights Large Language Model (LLM) by [Google DeepMind](https://deepmind.google/), based on Gemini research and technology. You can run decode and finetuning using [these instructions](end_to_end/gemma/Run_Gemma.md).
* [Llama2](https://llama.meta.com/llama2/): a family of open-weights Large Language Model (LLM) by Meta. You can run decode and finetuning using [these instructions](getting_started/Run_Llama2.md).

In addition to the getting started guides, there are always other MaxText capabilities that are being constantly being added! The full suite of end-to-end tests is in [end_to_end](end_to_end). We run them with a nightly cadence. They can be a good source for understanding MaxText Alternatively you can see the continuous [unit tests](.github/workflows/UnitTests.yml) which are run almost continuously.

# Runtime Performance Results

More details on reproducing these results can be found in [MaxText/configs/README.md](MaxText/configs/README.md).

## TPU v5p

| No. of params | Accelerator Type | TFLOP/chip/sec | Model flops utilization (MFU) |
|---|---|---|---|
| 32B | v5p-128 | 3.28e+02 | 71.47% |
| 64B | v5p-128 | 3.23e+02 | 70.31% |
| 128B | v5p-256 | 3.15e+02 | 68.68% |
| 128B | v5p-512 | 3.15e+02 | 68.53% |
| 256B | v5p-1024 | 3.16e+02 | 68.82% |
| 512B | v5p-1024 | 2.94e+02 | 63.99% |
| 1024B | v5p-2048 | 2.49e+02 | 64.05% |
| 1024B | v5p-4096 | 2.97e+02 | 64.80% |
| 1160B | v5p-7680 | 2.95e+02 | 64.27% |
| 1160B | v5p-12288 | 3.04e+02 | 66.23% |

## TPU v5e

For 16B, 32B, 64B, and 128B models. See full run configs in [MaxText/configs/v5e/](MaxText/configs/v5e/) as `16b.sh`, `32b.sh`, `64b.sh`, `128b.sh`.

| Hardware    | 16B TFLOP/sec/chip | 16B MFU | 32B TFLOP/sec/chip | 32B MFU | 64B TFLOP/sec/chip | 64B MFU | 128B TFLOP/sec/chip | 128B MFU |
| ----------- | -----------------: | ------- | -----------------: | ------- | -----------------: | ------- | ------------------: | -------- |
| 1x v5e-256  | 120                | 61.10%  | 132                | 66.86%  | 118                | 59.90%  | 110                 | 56.06%   |
| 2x v5e-256  | 117                | 59.37%  | 128                | 64.81%  | 112                | 56.66%  | 110                 | 55.82%   |
| 4x v5e-256  | 117                | 59.14%  | 126                | 64.10%  | 110                | 55.85%  | 108                 | 54.93%   |
| 8x v5e-256  | 115                | 58.27%  | 125                | 63.67%  | 108                | 54.96%  | 104                 | 52.93%   |
| 16x v5e-256 | 111                | 56.56%  | 123                | 62.26%  | 105                | 53.29%  | 100                 | 50.86%   |
| 32x v5e-256 | 108                | 54.65%  | 119                | 60.40%  | 99                 | 50.18%  | 91                  | 46.25%   |

# Comparison to Alternatives

MaxText is heavily inspired by [MinGPT](https://github.com/karpathy/minGPT)/[NanoGPT](https://github.com/karpathy/nanoGPT), elegant standalone GPT implementations written in PyTorch and targeting Nvidia GPUs. MaxText is more complex, supporting more industry standard models and scaling to tens of thousands of chips. Ultimately MaxText has an MFU more than three times the [17%](https://twitter.com/karpathy/status/1613250489097027584?cxt=HHwWgIDUhbixteMsAAAA) reported most recently with that codebase, is massively scalable and implements a key-value cache for efficient auto-regressive decoding.

MaxText is more similar to [Nvidia/Megatron-LM](https://github.com/NVIDIA/Megatron-LM), a very well tuned LLM implementation targeting Nvidia GPUs. The two implementations achieve comparable MFUs. The difference in the codebases highlights the different programming strategies. MaxText is pure Python, relying heavily on the XLA compiler to achieve high performance. By contrast, Megatron-LM is a mix of Python and CUDA, relying on well-optimized CUDA kernels to achieve high performance.

MaxText is also comparable to [Pax](https://github.com/google/paxml). Like Pax, MaxText provides high-performance and scalable implementations of LLMs in Jax. Pax focuses on enabling powerful configuration parameters, enabling developers to change the model by editing config parameters. By contrast, MaxText is a simple, concrete implementation of various LLMs that encourages users to extend by forking and directly editing the source code.

# Features and Diagnostics
## Collect Stack Traces
When running a Single Program, Multiple Data (SPMD) job on accelerators, the overall process can hang if there is any error or any VM hangs/crashes for some reason. In this scenario, capturing stack traces will help to identify and troubleshoot the issues for the jobs running on TPU VMs.

The following configurations will help to debug a fault or when a program is stuck or hung somewhere by collecting stack traces. Change the parameter values accordingly in `MaxText/configs/base.yml`:
1. Set `collect_stack_trace: True` to enable collection of stack traces on faults or when the program is hung. This setting will periodically dump the traces for the program to help in debugging. To disable this, set `collect_stack_trace: False`.
2. Set `stack_trace_to_cloud: False` to display stack traces on console. `stack_trace_to_cloud: True` will create a temporary file in `/tmp/debugging` in the TPUs to store the stack traces. There is an agent running on TPU VMs that will periodically upload the traces from the temporary directory to cloud logging in the gcp project. You can view the traces in Logs Explorer on Cloud Logging using the following query:
```
logName="projects/<project_name>/logs/tpu.googleapis.com%2Fruntime_monitor"
jsonPayload.verb="stacktraceanalyzer"
```
3. `stack_trace_interval_seconds` signifies the duration in seconds between each stack trace collection event. Setting `stack_trace_interval_seconds: 600` will collect the stack traces every 600 seconds (10 minutes).

Here is the related PyPI package: https://pypi.org/project/cloud-tpu-diagnostics.

## Ahead of Time Compilation (AOT, tpu-only)
To compile your training run ahead of time, we provide a tool `train_compile.py`. This tool allows you to compile the main `train_step` in `train.py` for target hardware (e.g. a large number of v5e devices) without using the target hardware, and instead you may use only a CPU or a single VM from a different family. This compilation helps with two main goals:

* It will flag any out of memory (OOM) information, such as when the `per_device_batch_size` is set too high, with an identical OOM stack trace as if it was compiled on the target hardware.

* The ahead of time compilation can be saved and then loaded for fast startup and restart times on the target hardware.

The tool `train_compile.py` is tightly linked to `train.py` and uses the same configuration file `configs/base.yml`. Although you don't need to run on a TPU, you do need to install `jax[tpu]` in addition to other dependencies, so we recommend running `setup.sh` to install these if you have not already done so.

### Example AOT 1: Compile ahead of time basics
After installing the dependencies listed above, you are ready to compile ahead of time:
```
# Run the below on a single machine, e.g. a CPU
python3 MaxText/train_compile.py MaxText/configs/base.yml compile_topology=v5e-256 compile_topology_num_slices=2 \
global_parameter_scale=16 per_device_batch_size=4
```

This will compile a 16B parameter MaxText model on 2 v5e pods.

### Example AOT 2: Save compiled function, then load and run it
Here is an example that saves then loads the compiled `train_step`, starting with the save:

**Step 1: Run AOT and save compiled function**
```
# Run the below on a single machine, e.g. a CPU
export LIBTPU_INIT_ARGS="--xla_enable_async_all_gather=true"
python3 MaxText/train_compile.py MaxText/configs/base.yml compile_topology=v5e-256 \
compile_topology_num_slices=2 \
compiled_trainstep_file=my_compiled_train.pickle global_parameter_scale=16 \
per_device_batch_size=4 steps=10000 learning_rate=1e-3
```

**Step 2: Run train.py and load the compiled function**

To load the compiled train_step, you just need to pass `compiled_trainstep_file=my_compiled_train.pickle` into `train.py`:
```
# Run the below on each host of the target hardware, e.g. each host on 2 slices of v5e-256
export LIBTPU_INIT_ARGS="--xla_enable_async_all_gather=true"
python3 MaxText/train.py MaxText/configs/base.yml run_name=example_load_compile \
compiled_trainstep_file=my_compiled_train.pickle \
global_parameter_scale=16  per_device_batch_size=4 steps=10000 learning_rate=1e-3 \
base_output_directory=gs://my-output-bucket dataset_path=gs://my-dataset-bucket
```

In the save step of example 2 above we included exporting the compiler flag `LIBTPU_INIT_ARGS` and `learning_rate` because those affect the compiled object `my_compiled_train.pickle.` The sizes of the model (e.g. `global_parameter_scale`, `max_sequence_length` and `per_device_batch`) are fixed when you initially compile via `compile_train.py`, you will see a size error if you try to run the saved compiled object with different sizes than you compiled with. However a subtle note is that the **learning rate schedule** is also fixed when you run `compile_train` - which is determined by both `steps` and `learning_rate`. The optimizer parameters such as  `adam_b1` are passed only as shaped objects to the compiler - thus their real values are determined when you run `train.py`, not during the compilation. If you do pass in different shapes (e.g. `per_device_batch`), you will get a clear error message reporting that the compiled signature has different expected shapes than what was input. If you attempt to run on different hardware than the compilation targets requested via `compile_topology`, you will get an error saying there is a failure to map the devices from the compiled to your real devices. Using different XLA flags or a LIBTPU than what was compiled will probably run silently with the environment you compiled in without error. However there is no guaranteed behavior in this case; you should run in the same environment you compiled in.

## Automatically Upload Logs to Vertex Tensorboard
MaxText supports automatic upload of logs collected in a directory to a Tensorboard instance in Vertex AI. Follow [user guide](getting_started/Use_Vertex_AI_Tensorboard.md) to know more.


============
osl-dynamics
============

See the read the docs page for a description of this project: `https://osl-dynamics.readthedocs.io <https://osl-dynamics.readthedocs.io>`_.

Citation
========

If you find this toolbox useful, please cite:

    **Chetan Gohil, Rukuang Huang, Evan Roberts, Mats WJ van Es, Andrew J Quinn, Diego Vidaurre, Mark W Woolrich (2024) osl-dynamics, a toolbox for modeling fast dynamic brain activity eLife 12:RP91949.**

Installation
============

Conda
-----

We recommend installing osl-dynamics within a virtual environment. You can do this with `Anaconda <https://docs.anaconda.com/free/anaconda/install/index.html>`_ (or `miniconda <https://docs.conda.io/projects/miniconda/en/latest/miniconda-install.html>`_).

Below we describe how to install osl-dynamics from source. We recommend using the conda environment files in ``/envs``.

Linux
-----

.. code-block:: shell

    git clone https://github.com/OHBA-analysis/osl-dynamics.git
    cd osl-dynamics
    conda env create -f envs/linux.yml
    conda activate osld
    pip install -e .

Mac
---

For a Mac, the installation of TensorFlow is slightly different to a Linux computer. We recommend using the lines above replacing the Linux environment file ``envs/linux.yml`` with the Mac environment file ``envs/mac.yml``.

Windows
-------

If you are using a Windows computer, we recommend first installing Linux (Ubuntu) as a Windows Subsystem by following the instructions `here <https://ubuntu.com/wsl>`_. Then following the instructions above in the Ubuntu terminal.

Within an osl environment
-------------------------

If you have already installed `OSL <https://github.com/OHBA-analysis/osl>`_ you can install osl-dynamics in the ``osl`` environment with:

.. code-block:: shell

    conda activate osl
    cd osl-dynamics
    pip install tensorflow==2.11.0
    pip install tensorflow-probability==0.19.0
    pip install -e .

Note, if you're using a Mac computer you need to install TensorFlow with ``pip install tensorflow-macos==2.11.0`` instead of ``tensorflow==2.11.0``.

Removing osl-dynamics
---------------------

Simply delete the conda environment and repository:

.. code-block:: shell

    conda env remove -n osld
    rm -rf osl-dynamics

Documentation
=============

The read the docs page should be automatically updated whenever there's a new commit on the ``main`` branch.

The documentation is included as docstrings in the source code. Please write docstrings to any classes or functions you add following the `numpy style <https://numpydoc.readthedocs.io/en/latest/format.html>`_. The API reference documentation will only be automatically generated if the docstrings are written correctly. The documentation directory ``/doc`` also contains ``.rst`` files that provide additional info regarding installation, development, the models, etc.

To compile the documentation locally you need to install the required packages (sphinx, etc.) in your conda environment:

.. code-block:: shell

    cd osl-dynamics
    pip install -r doc/requirements.txt

To compile the documentation locally use:

.. code-block:: shell

    python setup.py build_sphinx

The local build of the documentation webpage can be found in ``build/sphinx/html/index.html``.

Releases
========

A couple packages are needed to build and upload a project to PyPI, these can be installed in your conda environment with:

.. code-block:: shell

    pip install build twine

The following steps can be used to release a new version:

#. Update the version on line 5 of ``setup.cfg`` by removing ``dev`` from the version number.
#. Commit the updated setup.cfg to the ``main`` branch of the GitHub repo.
#. Delete any old distributions that have been built (if there are any): ``rm -r dist``.
#. Build a distribution in the osl-dynamics root directory with ``python -m build``. This will create a new directory called ``dist``.
#. Test the build by installing in a test conda environment with ``cd dist; pip install <build>.whl``.
#. Upload the distribution to PyPI with ``twine upload dist/*``. You will need to enter the username and password that you used to register with `https://pypi.org <https://pypi.org>`_.
#. Tag the commit uploaded to PyPI with the version number using the 'Create a new release' link on the right of the GitHub repo webpage.
#. Change the version to ``X.Y.devZ`` in ``setup.cfg`` and commit the new dev version to ``main``.

The uploaded distribution will then be available to be installed with:

.. code-block:: shell

    pip install osl-dynamics

Editing Source Code
===================

See `here <https://github.com/OHBA-analysis/osl-dynamics/blob/main/doc/using_bmrc.rst>`_ for useful info regarding how to use the BMRC cluster and how to edit the source code.