cfgrib: A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
======================================================================================================================

.. image:: https://img.shields.io/pypi/v/cfgrib.svg
   :target: https://pypi.python.org/pypi/cfgrib/

Python interface to map GRIB files to the
`Unidata's Common Data Model v4 <https://docs.unidata.ucar.edu/netcdf-java/current/userguide/common_data_model_overview.html>`_
following the `CF Conventions <http://cfconventions.org/>`_.
The high level API is designed to support a GRIB engine for `xarray <http://xarray.pydata.org/>`_
and it is inspired by `netCDF4-python <http://unidata.github.io/netcdf4-python/>`_
and `h5netcdf <https://github.com/shoyer/h5netcdf>`_.
Low level access and decoding is performed via the
`ECMWF ecCodes library <https://confluence.ecmwf.int/display/ECC/>`_ and
the `eccodes python package <https://pypi.org/project/eccodes>`_.

Features with development status **Beta**:

- enables the ``engine='cfgrib'`` option to read GRIB files with *xarray*,
- reads most GRIB 1 and 2 files including heterogeneous ones with ``cfgrib.open_datasets``,
- supports all modern versions of Python 3.9, 3.8, 3.7 and PyPy3,
- the 0.9.6.x series with support for Python 2 will stay active and receive critical bugfixes,
- works wherever *eccodes-python* does: *Linux*, *MacOS* and *Windows*
- conda-forge package on all supported platforms,
- reads the data lazily and efficiently in terms of both memory usage and disk access,
- allows larger-than-memory and distributed processing via *xarray* and *dask*,
- supports translating coordinates to different data models and naming conventions,
- supports writing the index of a GRIB file to disk, to save a full-file scan on open,
- accepts objects implementing a generic *Fieldset* interface as described in `ADVANCED_USAGE.rst`.

Work in progress:

- **Beta** install a ``cfgrib`` utility that can convert a GRIB file ``to_netcdf``
  with a optional conversion to a specific coordinates data model,
  see `#40 <https://github.com/ecmwf/cfgrib/issues/40>`_.
- **Alpha/Broken** support writing carefully-crafted ``xarray.Dataset``'s to a GRIB1 or GRIB2 file,
  see the *Advanced write usage* section below, `#18 <https://github.com/ecmwf/cfgrib/issues/18>`_
  and `#156 <https://github.com/ecmwf/cfgrib/issues/156>`_.

Limitations:

- relies on *ecCodes* for the CF attributes of the data variables,
- relies on *ecCodes* for anything related to coordinate systems / ``gridType``,
  see `#28 <https://github.com/ecmwf/cfgrib/issues/28>`_.

Installation
============

The easiest way to install *cfgrib* and all its binary dependencies is via `Conda <https://conda.io/>`_::

    $ conda install -c conda-forge cfgrib

alternatively, if you install the binary dependencies yourself, you can install the
Python package from *PyPI* with::

    $ pip install cfgrib

Binary dependencies
-------------------

*cfgrib* depends on the `eccodes python package <https://pypi.org/project/eccodes>`_
to access the ECMWF *ecCodes* binary library,
when not using *conda* please follow the *System dependencies* section there.

You may run a simple selfcheck command to ensure that your system is set up correctly::

    $ python -m cfgrib selfcheck
    Found: ecCodes v2.20.0.
    Your system is ready.

Usage
=====

First, you need a well-formed GRIB file, if you don't have one at hand you can download our
`ERA5 on pressure levels sample <https://get.ecmwf.int/repository/test-data/cfgrib/era5-levels-members.grib>`_::

    $ wget https://get.ecmwf.int/repository/test-data/cfgrib/era5-levels-members.grib

Read-only *xarray* GRIB engine
------------------------------

Most of *cfgrib* users want to open a GRIB file as a ``xarray.Dataset`` and
need to have *xarray* installed::

    $ pip install xarray

In a Python interpreter try:

.. code-block:: python

    >>> import xarray as xr
    >>> ds = xr.open_dataset('era5-levels-members.grib', engine='cfgrib')
    >>> ds
    <xarray.Dataset>
    Dimensions:        (number: 10, time: 4, isobaricInhPa: 2, latitude: 61,
                        longitude: 120)
    Coordinates:
    * number         (number) int64 0 1 2 3 4 5 6 7 8 9
    * time           (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00
        step           timedelta64[ns] ...
    * isobaricInhPa  (isobaricInhPa) float64 850.0 500.0
    * latitude       (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0
    * longitude      (longitude) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0
        valid_time     (time) datetime64[ns] ...
    Data variables:
        z              (number, time, isobaricInhPa, latitude, longitude) float32 ...
        t              (number, time, isobaricInhPa, latitude, longitude) float32 ...
    Attributes:
        GRIB_edition:            1
        GRIB_centre:             ecmf
        GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             European Centre for Medium-Range Weather Forecasts
        history:                 ...

The *cfgrib* ``engine`` supports all read-only features of *xarray* like:

* merge the content of several GRIB files into a single dataset using ``xarray.open_mfdataset``,
* work with larger-than-memory datasets with `dask <https://dask.org/>`_,
* allow distributed processing with `dask.distributed <http://distributed.dask.org>`_.

Read arbitrary GRIB keys
------------------------

By default *cfgrib* reads a limited set of ecCodes recognised *keys* from the GRIB files
and exposes them as ``Dataset`` or ``DataArray`` attributes with the ``GRIB_`` prefix.
It is possible to have *cfgrib* read additional keys to the attributes by adding the
``read_keys`` dictionary key to the ``backend_kwargs`` with values the list of desired GRIB keys:

.. code-block:: python

    >>> ds = xr.open_dataset('era5-levels-members.grib', engine='cfgrib',
    ...                      backend_kwargs={'read_keys': ['experimentVersionNumber']})
    >>> ds.t.attrs['GRIB_experimentVersionNumber']
    '0001'

Translate to a custom data model
--------------------------------

Contrary to netCDF the GRIB data format is not self-describing and several details of the mapping
to the *Unidata Common Data Model* are arbitrarily set by the software components decoding the format.
Details like names and units of the coordinates are particularly important because
*xarray* broadcast and selection rules depend on them.
``cf2cfm`` is a small coordinate translation module distributed with *cfgrib* that make it easy to
translate CF compliant coordinates, like the one provided by *cfgrib*, to a user-defined
custom data model with set ``out_name``, ``units`` and ``stored_direction``.

For example to translate a *cfgrib* styled ``xr.Dataset`` to the classic *ECMWF* coordinate
naming conventions you can:

.. code-block:: python

    >>> import cf2cdm
    >>> ds = xr.open_dataset('era5-levels-members.grib', engine='cfgrib')
    >>> cf2cdm.translate_coords(ds, cf2cdm.ECMWF)
    <xarray.Dataset>
    Dimensions:     (number: 10, time: 4, level: 2, latitude: 61, longitude: 120)
    Coordinates:
    * number      (number) int64 0 1 2 3 4 5 6 7 8 9
    * time        (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00
        step        timedelta64[ns] ...
    * level       (level) float64 850.0 500.0
    * latitude    (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0
    * longitude   (longitude) float64 0.0 3.0 6.0 9.0 ... 348.0 351.0 354.0 357.0
        valid_time  (time) datetime64[ns] ...
    Data variables:
        z           (number, time, level, latitude, longitude) float32 ...
        t           (number, time, level, latitude, longitude) float32 ...
    Attributes:
        GRIB_edition:            1
        GRIB_centre:             ecmf
        GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             European Centre for Medium-Range Weather Forecasts
        history:                 ...

To translate to the Common Data Model of the Climate Data Store use:

.. code-block:: python

    >>> import cf2cdm
    >>> cf2cdm.translate_coords(ds, cf2cdm.CDS)
    <xarray.Dataset>
    Dimensions:                  (realization: 10, forecast_reference_time: 4,
                                plev: 2, lat: 61, lon: 120)
    Coordinates:
    * realization              (realization) int64 0 1 2 3 4 5 6 7 8 9
    * forecast_reference_time  (forecast_reference_time) datetime64[ns] 2017-01...
        leadtime                 timedelta64[ns] ...
    * plev                     (plev) float64 8.5e+04 5e+04
    * lat                      (lat) float64 -90.0 -87.0 -84.0 ... 84.0 87.0 90.0
    * lon                      (lon) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0
        time                     (forecast_reference_time) datetime64[ns] ...
    Data variables:
        z                        (realization, forecast_reference_time, plev, lat, lon) float32 ...
        t                        (realization, forecast_reference_time, plev, lat, lon) float32 ...
    Attributes:
        GRIB_edition:            1
        GRIB_centre:             ecmf
        GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             European Centre for Medium-Range Weather Forecasts
        history:                 ...

Filter heterogeneous GRIB files
-------------------------------

``xr.open_dataset`` can open a GRIB file only if all the messages
with the same ``shortName`` can be represented as a single hypercube.
For example, a variable ``t`` cannot have both ``isobaricInhPa`` and ``hybrid`` ``typeOfLevel``'s,
as this would result in multiple hypercubes for the same variable.
Opening a non-conformant GRIB file will fail with a ``ValueError: multiple values for unique key...``
error message, see `#2 <https://github.com/ecmwf/cfgrib/issues/2>`_.

Furthermore if different variables depend on the same coordinate, for example ``step``,
the values of the coordinate must match exactly.
For example, if variables ``t`` and ``z`` share the same ``step`` coordinate,
they must both have exactly the same set of steps.
Opening a non-conformant GRIB file will fail with a ``ValueError: key present and new value is different...``
error message, see `#13 <https://github.com/ecmwf/cfgrib/issues/13>`_.

In most cases you can handle complex GRIB files containing heterogeneous messages by passing
the ``filter_by_keys`` key in ``backend_kwargs`` to select which GRIB messages belong to a
well formed set of hypercubes.

For example to open
`US National Weather Service complex GRIB2 files <http://ftpprd.ncep.noaa.gov/data/nccf/com/nam/prod/>`_
you can use:

.. code-block:: python

    >>> xr.open_dataset('nam.t00z.awp21100.tm00.grib2', engine='cfgrib',
    ...     backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface'}})
    <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] ...
        step        timedelta64[ns] ...
        surface     float64 ...
        latitude    (y, x) float64 ...
        longitude   (y, x) float64 ...
        valid_time  datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        gust        (y, x) float32 ...
        sp          (y, x) float32 ...
        orog        (y, x) float32 ...
        tp          (y, x) float32 ...
        acpcp       (y, x) float32 ...
        csnow       (y, x) float32 ...
        cicep       (y, x) float32 ...
        cfrzr       (y, x) float32 ...
        crain       (y, x) float32 ...
        cape        (y, x) float32 ...
        cin         (y, x) float32 ...
        unknown     (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP...
        history:                 ...
    >>> xr.open_dataset('nam.t00z.awp21100.tm00.grib2', engine='cfgrib',
    ...     backend_kwargs={'filter_by_keys': {'typeOfLevel': 'heightAboveGround', 'level': 2}})
    <xarray.Dataset>
    Dimensions:            (y: 65, x: 93)
    Coordinates:
        time               datetime64[ns] ...
        step               timedelta64[ns] ...
        heightAboveGround  float64 ...
        latitude           (y, x) float64 ...
        longitude          (y, x) float64 ...
        valid_time         datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        t2m                (y, x) float32 ...
        r2                 (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP...
        history:                 ...

Automatic filtering
-------------------

*cfgrib* also provides a function that automates the selection of appropriate ``filter_by_keys``
and returns a list of all valid ``xarray.Dataset``'s in the GRIB file.

.. code-block:: python

    >>> import cfgrib
    >>> cfgrib.open_datasets('nam.t00z.awp21100.tm00.grib2')
    [<xarray.Dataset>
    Dimensions:                (y: 65, x: 93)
    Coordinates:
        time                   datetime64[ns] 2018-09-17
        step                   timedelta64[ns] 00:00:00
        atmosphereSingleLayer  float64 0.0
        latitude               (y, x) float64 ...
        longitude              (y, x) float64 ...
        valid_time             datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        pwat                   (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        cloudBase   float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        pres        (y, x) float32 ...
        gh          (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        cloudTop    float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        pres        (y, x) float32 ...
        t           (y, x) float32 ...
        gh          (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:            (y: 65, x: 93)
    Coordinates:
        time               datetime64[ns] 2018-09-17
        step               timedelta64[ns] 00:00:00
        heightAboveGround  float64 10.0
        latitude           (y, x) float64 ...
        longitude          (y, x) float64 ...
        valid_time         datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        u10                (y, x) float32 ...
        v10                (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:            (y: 65, x: 93)
    Coordinates:
        time               datetime64[ns] 2018-09-17
        step               timedelta64[ns] 00:00:00
        heightAboveGround  float64 2.0
        latitude           (y, x) float64 12.19 12.39 12.58 ... 57.68 57.49 57.29
        longitude          (y, x) float64 226.5 227.2 227.9 ... 308.5 309.6 310.6
        valid_time         datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        t2m                (y, x) float32 ...
        r2                 (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:                 (heightAboveGroundLayer: 2, y: 65, x: 93)
    Coordinates:
        time                    datetime64[ns] 2018-09-17
        step                    timedelta64[ns] 00:00:00
    * heightAboveGroundLayer  (heightAboveGroundLayer) float64 1e+03 3e+03
        latitude                (y, x) float64 ...
        longitude               (y, x) float64 ...
        valid_time              datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        hlcy                    (heightAboveGroundLayer, y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:        (isobaricInhPa: 19, y: 65, x: 93)
    Coordinates:
        time           datetime64[ns] 2018-09-17
        step           timedelta64[ns] 00:00:00
    * isobaricInhPa  (isobaricInhPa) float64 1e+03 950.0 900.0 ... 150.0 100.0
        latitude       (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude      (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time     datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        t              (isobaricInhPa, y, x) float32 ...
        u              (isobaricInhPa, y, x) float32 ...
        v              (isobaricInhPa, y, x) float32 ...
        w              (isobaricInhPa, y, x) float32 ...
        gh             (isobaricInhPa, y, x) float32 ...
        r              (isobaricInhPa, y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:        (isobaricInhPa: 5, y: 65, x: 93)
    Coordinates:
        time           datetime64[ns] 2018-09-17
        step           timedelta64[ns] 00:00:00
    * isobaricInhPa  (isobaricInhPa) float64 1e+03 850.0 700.0 500.0 250.0
        latitude       (y, x) float64 ...
        longitude      (y, x) float64 ...
        valid_time     datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        absv           (isobaricInhPa, y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:       (y: 65, x: 93)
    Coordinates:
        time          datetime64[ns] 2018-09-17
        step          timedelta64[ns] 00:00:00
        isothermZero  float64 0.0
        latitude      (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude     (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time    datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        gh            (y, x) float32 ...
        r             (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        maxWind     float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        pres        (y, x) float32 ...
        u           (y, x) float32 ...
        v           (y, x) float32 ...
        gh          (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        meanSea     float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        prmsl       (y, x) float32 ...
        mslet       (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:                  (pressureFromGroundLayer: 2, y: 65, x: 93)
    Coordinates:
        time                     datetime64[ns] 2018-09-17
        step                     timedelta64[ns] 00:00:00
    * pressureFromGroundLayer  (pressureFromGroundLayer) float64 9e+03 1.8e+04
        latitude                 (y, x) float64 12.19 12.39 12.58 ... 57.49 57.29
        longitude                (y, x) float64 226.5 227.2 227.9 ... 309.6 310.6
        valid_time               datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        cape                     (pressureFromGroundLayer, y, x) float32 ...
        cin                      (pressureFromGroundLayer, y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:                  (pressureFromGroundLayer: 5, y: 65, x: 93)
    Coordinates:
        time                     datetime64[ns] 2018-09-17
        step                     timedelta64[ns] 00:00:00
    * pressureFromGroundLayer  (pressureFromGroundLayer) float64 3e+03 ... 1.5e+04
        latitude                 (y, x) float64 12.19 12.39 12.58 ... 57.49 57.29
        longitude                (y, x) float64 226.5 227.2 227.9 ... 309.6 310.6
        valid_time               datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        t                        (pressureFromGroundLayer, y, x) float32 ...
        u                        (pressureFromGroundLayer, y, x) float32 ...
        v                        (pressureFromGroundLayer, y, x) float32 ...
        r                        (pressureFromGroundLayer, y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:                  (y: 65, x: 93)
    Coordinates:
        time                     datetime64[ns] 2018-09-17
        step                     timedelta64[ns] 00:00:00
        pressureFromGroundLayer  float64 3e+03
        latitude                 (y, x) float64 ...
        longitude                (y, x) float64 ...
        valid_time               datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        pli                      (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:                  (y: 65, x: 93)
    Coordinates:
        time                     datetime64[ns] 2018-09-17
        step                     timedelta64[ns] 00:00:00
        pressureFromGroundLayer  float64 1.8e+04
        latitude                 (y, x) float64 ...
        longitude                (y, x) float64 ...
        valid_time               datetime64[ns] ...
    Dimensions without coordinates: y, x
    Data variables:
        4lftx                    (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        surface     float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        unknown     (y, x) float32 ...
        cape        (y, x) float32 ...
        sp          (y, x) float32 ...
        acpcp       (y, x) float32 ...
        cin         (y, x) float32 ...
        orog        (y, x) float32 ...
        tp          (y, x) float32 ...
        crain       (y, x) float32 ...
        cfrzr       (y, x) float32 ...
        cicep       (y, x) float32 ...
        csnow       (y, x) float32 ...
        gust        (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP, <xarray.Dataset>
    Dimensions:     (y: 65, x: 93)
    Coordinates:
        time        datetime64[ns] 2018-09-17
        step        timedelta64[ns] 00:00:00
        tropopause  float64 0.0
        latitude    (y, x) float64 12.19 12.39 12.58 12.77 ... 57.68 57.49 57.29
        longitude   (y, x) float64 226.5 227.2 227.9 228.7 ... 308.5 309.6 310.6
        valid_time  datetime64[ns] 2018-09-17
    Dimensions without coordinates: y, x
    Data variables:
        t           (y, x) float32 ...
        u           (y, x) float32 ...
        v           (y, x) float32 ...
        trpp        (y, x) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             kwbc
        GRIB_centreDescription:  US National Weather Service - NCEP...
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             US National Weather Service - NCEP]

Advanced usage
==============

Write support
=============

**Please note that write support is Alpha.**
Only ``xarray.Dataset``'s in *canonical* form,
that is, with the coordinates names matching exactly the *cfgrib* coordinates,
can be saved at the moment:

.. code-block:: python

    >>> from cfgrib.xarray_to_grib import to_grib
    >>> ds = xr.open_dataset('era5-levels-members.grib', engine='cfgrib').sel(number=0)
    >>> ds
    <xarray.Dataset>
    Dimensions:        (time: 4, isobaricInhPa: 2, latitude: 61, longitude: 120)
    Coordinates:
        number         int64 0
    * time           (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00
        step           timedelta64[ns] ...
    * isobaricInhPa  (isobaricInhPa) float64 850.0 500.0
    * latitude       (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0
    * longitude      (longitude) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0
        valid_time     (time) datetime64[ns] ...
    Data variables:
        z              (time, isobaricInhPa, latitude, longitude) float32 ...
        t              (time, isobaricInhPa, latitude, longitude) float32 ...
    Attributes:
        GRIB_edition:            1
        GRIB_centre:             ecmf
        GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             European Centre for Medium-Range Weather Forecasts
        history:                 ...
    >>> to_grib(ds, 'out1.grib', grib_keys={'edition': 2})
    >>> xr.open_dataset('out1.grib', engine='cfgrib')
    <xarray.Dataset>
    Dimensions:        (time: 4, isobaricInhPa: 2, latitude: 61, longitude: 120)
    Coordinates:
        number         ...
    * time           (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00
        step           timedelta64[ns] ...
    * isobaricInhPa  (isobaricInhPa) float64 850.0 500.0
    * latitude       (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0
    * longitude      (longitude) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0
        valid_time     (time) datetime64[ns] ...
    Data variables:
        z              (time, isobaricInhPa, latitude, longitude) float32 ...
        t              (time, isobaricInhPa, latitude, longitude) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             ecmf
        GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             European Centre for Medium-Range Weather Forecasts
        history:                 ...

Per-variable GRIB keys can be set by setting the ``attrs`` variable with key prefixed by ``GRIB_``,
for example:

.. code-block:: python

    >>> import numpy as np
    >>> import xarray as xr
    >>> ds2 = xr.DataArray(
    ...     np.zeros((5, 6)) + 300.,
    ...     coords=[
    ...         np.linspace(90., -90., 5),
    ...         np.linspace(0., 360., 6, endpoint=False),
    ...     ],
    ...     dims=['latitude', 'longitude'],
    ... ).to_dataset(name='skin_temperature')
    >>> ds2.skin_temperature.attrs['GRIB_shortName'] = 'skt'
    >>> to_grib(ds2, 'out2.grib')
    >>> xr.open_dataset('out2.grib', engine='cfgrib')
    <xarray.Dataset>
    Dimensions:     (latitude: 5, longitude: 6)
    Coordinates:
        time        datetime64[ns] ...
        step        timedelta64[ns] ...
        surface     float64 ...
    * latitude    (latitude) float64 90.0 45.0 0.0 -45.0 -90.0
    * longitude   (longitude) float64 0.0 60.0 120.0 180.0 240.0 300.0
        valid_time  datetime64[ns] ...
    Data variables:
        skt         (latitude, longitude) float32 ...
    Attributes:
        GRIB_edition:            2
        GRIB_centre:             consensus
        GRIB_centreDescription:  Consensus
        GRIB_subCentre:          0
        Conventions:             CF-1.7
        institution:             Consensus
        history:                 ...

Dataset / Variable API
----------------------

The use of *xarray* is not mandatory and you can access the content of a GRIB file as
an hypercube with the high level API in a Python interpreter:

.. code-block:: python

    >>> ds = cfgrib.open_file('era5-levels-members.grib')
    >>> ds.attributes['GRIB_edition']
    1
    >>> sorted(ds.dimensions.items())
    [('isobaricInhPa', 2), ('latitude', 61), ('longitude', 120), ('number', 10), ('time', 4)]
    >>> sorted(ds.variables)
    ['isobaricInhPa', 'latitude', 'longitude', 'number', 'step', 't', 'time', 'valid_time', 'z']
    >>> var = ds.variables['t']
    >>> var.dimensions
    ('number', 'time', 'isobaricInhPa', 'latitude', 'longitude')
    >>> var.data[:, :, :, :, :].mean()
    262.92133
    >>> ds = cfgrib.open_file('era5-levels-members.grib')
    >>> ds.attributes['GRIB_edition']
    1
    >>> sorted(ds.dimensions.items())
    [('isobaricInhPa', 2), ('latitude', 61), ('longitude', 120), ('number', 10), ('time', 4)]
    >>> sorted(ds.variables)
    ['isobaricInhPa', 'latitude', 'longitude', 'number', 'step', 't', 'time', 'valid_time', 'z']
    >>> var = ds.variables['t']
    >>> var.dimensions
    ('number', 'time', 'isobaricInhPa', 'latitude', 'longitude')
    >>> var.data[:, :, :, :, :].mean()
    262.92133

GRIB index file
---------------

By default *cfgrib* saves the index of the GRIB file to disk appending ``.idx``
to the GRIB file name.
Index files are an **experimental** and completely optional feature, feel free to
remove them and try again in case of problems. Index files saving can be disable passing
adding ``indexpath=''`` to the ``backend_kwargs`` keyword argument.

Geographic Coordinate Caching
-----------------------------

By default, *cfgrib* caches computed geography coordinates for each record in the GRIB
file when opening a dataset, which significantly speeds up dataset creation.
This cache can theoretically grow unboundedly in memory in long-lived
applications which read many different grid types. Should it be necessary,
caching can be disabled by passing `backend_kwargs=dict(cache_geo_coords=False)`
to `xarray.open_dataset()`, `cfgrib.open_dataset()`, or
`cfgrib.open_datasets()`.

Project resources
=================

============= =========================================================
Development   https://github.com/ecmwf/cfgrib
Download      https://pypi.org/project/cfgrib
User support  https://stackoverflow.com/search?q=cfgrib
Code quality  .. image:: https://codecov.io/gh/ecmwf/cfgrib/branch/master/graph/badge.svg
                :target: https://codecov.io/gh/ecmwf/cfgrib
                :alt: Coverage status on Codecov
============= =========================================================

Contributing
============

The main repository is hosted on GitHub,
testing, bug reports and contributions are highly welcomed and appreciated:

https://github.com/ecmwf/cfgrib

Please see the CONTRIBUTING.rst document for the best way to help.

Lead developers:

- `Iain Russell <https://github.com/iainrussell>`_ - `ECMWF <https://ecmwf.int>`_
- `Baudouin Raoult <https://github.com/b8raoult>`_ - ECMWF

Main contributors:

- `Alessandro Amici <https://github.com/alexamici>`_ - `B-Open <https://bopen.eu>`_
- `Aureliana Barghini <https://github.com/aurghs>`_ - B-Open
- `Leonardo Barcaroli <https://github.com/leophys>`_ - B-Open

See also the list of `contributors <https://github.com/ecmwf/cfgrib/contributors>`_ who participated in this project.

License
=======

Copyright 2017-2021 European Centre for Medium-Range Weather Forecasts (ECMWF).

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


# Wind Waker Randomizer

This is a randomizer for The Legend of Zelda: The Wind Waker.
It randomizes all the items in the game so that each playthrough is unique and you never know where a particular item will be.
It also makes the game completely open world from the start, removes most cutscenes from the game, and increases sailing speed and text speed.

If you're new here, check out the official website for details on how to get started playing: https://lagolunatic.github.io/wwrando/

Or you can download the randomizer here: https://github.com/LagoLunatic/wwrando/releases/latest

## Information

The randomizer only supports the North American GameCube version of Wind Waker. (MD5: d8e4d45af2032a081a0f446384e9261b)
The European and Japanese versions of Wind Waker won't work, and neither will Wind Waker HD.

The randomizer guarantees that every playthrough will be completable, and that you don't need to use any glitches or tricks to beat it.

All items are randomized, but because Wind Waker is such a large game, a single run of it would take a very long time if you had to check every single location. Therefore the randomizer has options to limit where progress items can appear based on the type of the location.
For example, you can limit progress items to appearing in dungeons and secret caves only, or secret caves sidequests and mail, or any other combination you want.
Location types that you don't select will only have unimportant items that you don't need to beat the game - like rupees, heart pieces, bomb bag upgrades, etc. So you can skip checking them entirely, unless you want some of those optional items.

### Randomizer won't launch?

If the randomizer gives an error saying "Failed to execute script wwrando", or if the executable disappears before you can launch it, then your antivirus software may be incorrectly detecting the randomizer as malware. You should add an exception/exclusion for the randomizer into your antivirus software so that it will ignore the randomizer if this happens.

### Got stuck in a seed?

If you seem to be stuck and can't find anywhere to progress, you should first read the Frequently Asked Questions page on the official website: https://lagolunatic.github.io/wwrando/faq/

If you've read the FAQ and consulted the spoiler log but you're still stuck, it's possible you've encountered a bug in the randomizer.
Please report bugs like that here: https://github.com/LagoLunatic/wwrando/issues
In the bug report be sure to include the permalink for the seed you encountered the bug on.

## Discord Server

If you have any questions or are looking for people to play/race with, why not join the official Wind Waker Randomizer Discord server?
https://discord.gg/r2963mt

## Credits

The randomizer was created and programmed by LagoLunatic, with help from:
Aelire (additional programming)
CryZe (event flag documentation)
EthanArmbrust (Mac and Linux support)
Fig (additional programming)
Gamma / SageOfMirrors (custom model conversion, file format documentation, additional programming)
Hypatia (textures)
JarheadHME (additional programming)
LordNed (file format documentation)
MelonSpeedruns (game design suggestions, graphic design)
nbouteme (starting items programming)
tanjo3 (CTMC and hint programming)
TrogWW (additional programming)
wooferzfg (additional programming)

## Running the randomizer from source

If you want to run the latest development version of the randomizer from source, follow the instructions below.
Note that this is only recommended for advanced users or those interested in helping develop the randomizer.

Download and install git from here: https://git-scm.com/downloads
Then clone this repository with git by running this in a command prompt:
`git clone --recurse-submodules https://github.com/LagoLunatic/wwrando.git`

Download and install Python 3.12 from here: https://www.python.org/downloads/release/python-3121/
"Windows installer (64-bit)" is the one you want if you're on Windows, "macOS 64-bit universal2 installer" if you're on Mac.
If you're on Linux, run this command instead: `sudo apt-get install python3.12`

Open the wwrando folder in a command prompt and install dependencies by running:
`py -3.12 -m pip install -r requirements.txt` (on Windows)
`python3 -m pip install -r requirements.txt` (on Mac)
`python3 -m pip install $(cat requirements.txt) --user` (on Linux)

Then run the randomizer with:
`py wwrando.py` (on Windows)
`python3 wwrando.py` (on Mac)
`python3 wwrando.py` (on Linux)

Optionally, you can also install `requirements_full.txt` with the same process you used for `requirements.txt` above.
`requirements_full.txt` will install additional libraries that speed up texture recoloring, as well as for building a distributable version of the randomizer. You can still run the randomizer from source without these.


# Jupyter Client

[![Build Status](https://github.com/jupyter/jupyter_client/workflows/CI/badge.svg)](https://github.com/jupyter/jupyter_client/actions)
[![Documentation Status](https://readthedocs.org/projects/jupyter-client/badge/?version=latest)](http://jupyter-client.readthedocs.io/en/latest/?badge=latest)

`jupyter_client` contains the reference implementation of the [Jupyter protocol].
It also provides client and kernel management APIs for working with kernels.

It also provides the `jupyter kernelspec` entrypoint
for installing kernelspecs for use with Jupyter frontends.

## Development Setup

The [Jupyter Contributor Guides](https://jupyter.readthedocs.io/en/latest/contributing/content-contributor.html) provide extensive information on contributing code or documentation to Jupyter projects. The limited instructions below for setting up a development environment are for your convenience.

## Coding

You'll need Python and `pip` on the search path. Clone the Jupyter Client git repository to your computer, for example in `/my/project/jupyter_client`

```bash
cd /my/projects/
git clone git@github.com:jupyter/jupyter_client.git
```

Now create an [editable install](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)
and download the dependencies of code and test suite by executing:

```bash
cd /my/projects/jupyter_client/
pip install -e ".[test]"
pytest
```

The last command runs the test suite to verify the setup. During development, you can pass filenames to `pytest`, and it will execute only those tests.

## Documentation

The documentation of Jupyter Client is generated from the files in `docs/` using Sphinx. Instructions for setting up Sphinx with a selection of optional modules are in the [Documentation Guide](https://jupyter.readthedocs.io/en/latest/contributing/docs-contributions/index.html). You'll also need the `make` command.
For a minimal Sphinx installation to process the Jupyter Client docs, execute:

```bash
pip install ".[doc]"
```

The following commands build the documentation in HTML format and check for broken links:

```bash
cd /my/projects/jupyter_client/docs/
make html linkcheck
```

Point your browser to the following URL to access the generated documentation:

_file:///my/projects/jupyter_client/docs/\_build/html/index.html_

## Contributing

`jupyter-client` has adopted automatic code formatting so you shouldn't
need to worry too much about your code style.
As long as your code is valid,
the pre-commit hook should take care of how it should look.
You can invoke the pre-commit hook by hand at any time with:

```bash
pre-commit run
```

which should run any autoformatting on your code
and tell you about any errors it couldn't fix automatically.
You may also install [black integration](https://black.readthedocs.io/en/stable/integrations/editors.html)
into your text editor to format code automatically.

If you have already committed files before setting up the pre-commit
hook with `pre-commit install`, you can fix everything up using
`pre-commit run --all-files`. You need to make the fixing commit
yourself after that.

Some of the hooks only run on CI by default, but you can invoke them by
running with the `--hook-stage manual` argument.

## About the Jupyter Development Team

The Jupyter Development Team is the set of all contributors to the Jupyter project.
This includes all of the Jupyter subprojects.

The core team that coordinates development on GitHub can be found here:
https://github.com/jupyter/.

## Our Copyright Policy

Jupyter uses a shared copyright model. Each contributor maintains copyright
over their contributions to Jupyter. But, it is important to note that these
contributions are typically only changes to the repositories. Thus, the Jupyter
source code, in its entirety is not the copyright of any single person or
institution. Instead, it is the collective copyright of the entire Jupyter
Development Team. If individual contributors want to maintain a record of what
changes/contributions they have specific copyright on, they should indicate
their copyright in the commit message of the change, when they commit the
change to one of the Jupyter repositories.

With this in mind, the following banner should be used in any source code file
to indicate the copyright and license terms:

```
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
```

[jupyter protocol]: https://jupyter-client.readthedocs.io/en/latest/messaging.html


# Censys Python Library

[![PyPI](https://img.shields.io/pypi/v/censys?color=orange&logo=pypi&logoColor=orange)](https://pypi.org/project/censys/)
[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue?logo=python)](https://www.python.org/downloads/)
[![Read the Docs (version)](https://img.shields.io/readthedocs/censys-python/latest?logo=read%20the%20docs)](https://censys-python.readthedocs.io/en/stable/?badge=stable)
[![GitHub Discussions](https://img.shields.io/badge/GitHub-Discussions-brightgreen?logo=github)](https://github.com/censys/censys-python/discussions)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-organge.svg?logo=git&logoColor=organge)](http://makeapullrequest.com)
[![License](https://img.shields.io/github/license/censys/censys-python?logo=apache)](https://github.com/censys/censys-python/blob/main/LICENSE)

An easy-to-use and lightweight API wrapper for Censys APIs ([censys.io](https://censys.io/)). Python 3.8+ is currently supported. This library has been tested on Python 3.8 and 3.x (Currently version 3.10).

> **Notice:** The Censys Search v1 endpoints are deprecated as of Nov. 30, 2021. Please begin using v2 endpoints to query hosts and certificates and check out our [support center](https://support.censys.io/hc/en-us/sections/360013076551-Censys-Search-2-0) for resources.

## Features

- [Search Censys data](https://censys-python.readthedocs.io/en/stable/usage-v2.html)
- [Bulk Certificate lookups](https://censys-python.readthedocs.io/en/stable/usage-v2.html#bulk-view)
- [Download Bulk Data](https://censys-python.readthedocs.io/en/stable/usage-v1.html#data)
- [Manage assets, events, and seeds in Censys ASM](https://censys-python.readthedocs.io/en/stable/usage-asm.html)
- [Command-line interface](https://censys-python.readthedocs.io/en/stable/cli.html)

## Getting Started

The library can be installed using `pip`.

```sh
pip install censys
```

To upgraded using `pip`.

```sh
pip install --upgrade censys
```

Alternatively, you can install the library using `poetry`.

```sh
git clone https://github.com/censys/censys-python.git
cd censys-python/
poetry install
```

Optionally, you can enable tab completion for the CLI by adding this line to your `~/.bashrc`, `~/.zshrc`, or equivalent.

> Please note that autocomplete is supported for field names in the `search` command.

```sh
eval "$(register-python-argcomplete censys)"
```

To configure your search credentials run `censys config` or set both `CENSYS_API_ID` and `CENSYS_API_SECRET` environment variables.

```sh
$ censys config

Censys API ID: XXX
Censys API Secret: XXX
Do you want color output? [y/n]: y

Successfully authenticated for your@email.com
```

If you have a Censys ASM account, you can configure your ASM credentials by running `censys asm config` or set both `CENSYS_ASM_API_KEY` environment variables.

```sh
$ censys asm config

Censys ASM API Key: XXX
Do you want color output? [y/n]: y

Successfully authenticated
```

## Examples

The examples located in the [`examples/`](examples/) directory are a great place to start. You can also find more examples in the [usage documentation](https://censys-python.readthedocs.io/en/stable/usage-v2.html).

## [Documentation](https://censys-python.readthedocs.io/)

[![Read the Docs](https://raw.githubusercontent.com/censys/censys-python/main/docs/_static/readthedocs.png)](https://censys-python.readthedocs.io/)

## Resources

- [Source](https://github.com/censys/censys-python)
- [Issue Tracker](https://github.com/censys/censys-python/issues)
- [Changelog](https://github.com/censys/censys-python/releases)
- [Documentation](https://censys-python.rtfd.io)
- [Discussions](https://github.com/censys/censys-python/discussions)
- [Censys Homepage](https://censys.io/)
- [Censys Search](https://search.censys.io/)

## Contributing

All contributions (no matter how small) are always welcome. See [Contributing to Censys Python](.github/CONTRIBUTING.md)

## Development

This project uses [poetry](https://python-poetry.org/) for dependency management. Please ensure you have [installed the latest version](https://python-poetry.org/docs/#installation).

```sh
git clone git@github.com:censys/censys-python.git
cd censys-python/
poetry install
```

## Testing

```sh
# Run tests
poetry run pytest
# With coverage report
poetry run pytest --cov-report html
```

## License

This software is licensed under [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)

- Copyright (C) 2024 Censys, Inc.


===============================================================================
The Public Utility Data Liberation Project (PUDL)
===============================================================================

.. readme-intro

.. |repo-status| image:: https://www.repostatus.org/badges/latest/active.svg
   :target: https://www.repostatus.org/#active
   :alt: Project Status: Active
.. |pytest| image:: https://github.com/catalyst-cooperative/pudl/workflows/pytest/badge.svg
   :target: https://github.com/catalyst-cooperative/pudl/actions?query=workflow%3Apytest
   :alt: PyTest Status
.. |codecov| image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/pudl?style=flat&logo=codecov
   :target: https://codecov.io/gh/catalyst-cooperative/pudl
   :alt: Codecov Test Coverage
.. |rtd| image:: https://img.shields.io/readthedocs/catalystcoop-pudl?style=flat&logo=readthedocs
   :target: https://catalystcoop-pudl.readthedocs.io/en/latest/
   :alt: Read the Docs Build Status
.. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
   :target: https://github.com/astral-sh/ruff
.. |pre-commit-ci| image:: https://results.pre-commit.ci/badge/github/catalyst-cooperative/pudl/main.svg
   :target: https://results.pre-commit.ci/latest/github/catalyst-cooperative/pudl/main
   :alt: pre-commit CI
.. |zenodo-doi| image:: https://zenodo.org/badge/80646423.svg
   :target: https://zenodo.org/badge/latestdoi/80646423
   :alt: Zenodo DOI
.. |office-hours| image:: https://img.shields.io/badge/calend.ly-officehours-darkgreen
   :target: https://calend.ly/catalyst-cooperative/pudl-office-hours
   :alt: Schedule a 1-on-1 chat with us about PUDL.
.. |mastodon| image:: https://img.shields.io/mastodon/follow/110855618428885893?domain=https%3A%2F%2Fmastodon.energy&style=social&color=%23000000&link=https%3A%2F%2Fmastodon.energy%2F%40catalystcoop
   :target: https://mastodon.energy/@catalystcoop
   :alt: Follow CatalystCoop on Mastodon

|repo-status| |pytest| |codecov| |rtd| |ruff| |pre-commit-ci| |zenodo-doi|
|office-hours| |mastodon|

What is PUDL?
-------------

The `PUDL <https://catalyst.coop/pudl/>`__ Project is an open source data processing
pipeline that makes US energy data easier to access and use programmatically.

Hundreds of gigabytes of valuable data are published by US government agencies, but it's
often difficult to work with. PUDL takes the original spreadsheets, CSV files, and
databases and turns them into a unified resource. This allows users to spend more time
on novel analysis and less time on data preparation.

The project is focused on serving researchers, activists, journalists, policy makers,
and small businesses that might not otherwise be able to afford access to this data from
commercial sources and who may not have the time or expertise to do all the data
processing themselves from scratch.

We want to make this data accessible and easy to work with for as wide an audience as
possible: anyone from a grassroots youth climate organizers working with Google sheets
to university researchers with access to scalable cloud computing resources and everyone
in between!

PUDL is comprised of three core components:

Raw Data Archives
^^^^^^^^^^^^^^^^^
PUDL `archives <https://github.com/catalyst-cooperative/pudl-archiver>`__ all our raw
inputs on `Zenodo
<https://zenodo.org/communities/catalyst-cooperative/?page=1&size=20>`__ to ensure
permanent, versioned access to the data. In the event that an agency changes how it
publishes data or deletes old files, the data processing pipeline will still have access
to the original inputs. Each of the data inputs may have several different versions
archived, and all are assigned a unique DOI (digital object identifier) and made
available through Zenodo's REST API.  You can read more about the Raw Data Archives in
the `docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/#raw-data-archives>`__.

Data Pipeline
^^^^^^^^^^^^^
The data pipeline (this repo) ingests raw data from the archives, cleans and integrates
it, and writes the resulting tables to `SQLite <https://sqlite.org>`__ and `Apache
Parquet <https://parquet.apache.org/>`__ files, with some accompanying metadata stored as
JSON.  Each release of the PUDL software contains a set of DOIs indicating which
versions of the raw inputs it processes. This helps ensure that the outputs are
replicable. You can read more about our ETL (extract, transform, load) process in the
`PUDL documentation <https://catalystcoop-pudl.readthedocs.io/en/nightly/#the-etl-process>`__.

Data Warehouse
^^^^^^^^^^^^^^
The SQLite, Parquet, and JSON outputs from the data pipeline, sometimes called "PUDL
outputs", are updated each night by an automated build process, and periodically
archived so that users can access the data without having to install and run our data
processing system. These outputs contain hundreds of tables and comprise a small
file-based data warehouse that can be used for a variety of energy system analyses.
Learn more about `how to access the PUDL data
<https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html>`__.

What data is available?
-----------------------

PUDL currently integrates data from:

* **EIA Form 860**: 2001-2022
  - `Source Docs <https://www.eia.gov/electricity/data/eia860/>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia860.html>`__
* **EIA Form 860m**: 2023-12
  - `Source Docs <https://www.eia.gov/electricity/data/eia860m/>`__
* **EIA Form 861**: 2001-2022
  - `Source Docs <https://www.eia.gov/electricity/data/eia861/>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia861.html>`__
* **EIA Form 923**: 2001-2023
  - `Source Docs <https://www.eia.gov/electricity/data/eia923/>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia923.html>`__
* **EPA Continuous Emissions Monitoring System (CEMS)**: 1995Q1-2023Q4
  - `Source Docs <https://campd.epa.gov/>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/epacems.html>`__
* **FERC Form 1**: 1994-2022
  - `Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/ferc1.html>`__
* **FERC Form 714**: 2006-2022 (mostly raw)
  - `Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric/data>`__
  - `PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/ferc714.html>`__
* **FERC Form 2**: 1996-2022 (raw only)
  - `Source Docs <https://www.ferc.gov/industries-data/natural-gas/industry-forms/form-2-2a-3-q-gas-historical-vfp-data>`__
* **FERC Form 6**: 2000-2022 (raw only)
  - `Source Docs <https://www.ferc.gov/general-information-1/oil-industry-forms/form-6-6q-historical-vfp-data>`__
* **FERC Form 60**: 2006-2022 (raw only)
  - `Source Docs <https://www.ferc.gov/form-60-annual-report-centralized-service-companies>`__
* **US Census Demographic Profile 1 Geodatabase**: 2010
  - `Source Docs <https://www.census.gov/geographies/mapping-files/2010/geo/tiger-data.html>`__

Thanks to support from the `Alfred P. Sloan Foundation Energy & Environment
Program <https://sloan.org/programs/research/energy-and-environment>`__, from
2021 to 2024 we will be cleaning and integrating the following data as well:

* `EIA Form 176 <https://www.eia.gov/dnav/ng/TblDefs/NG_DataSources.html#s176>`__
  (The Annual Report of Natural Gas Supply and Disposition)
* `FERC Electric Quarterly Reports (EQR) <https://www.ferc.gov/industries-data/electric/power-sales-and-markets/electric-quarterly-reports-eqr>`__
* `FERC Form 2 <https://www.ferc.gov/industries-data/natural-gas/overview/general-information/natural-gas-industry-forms/form-22a-data>`__
  (Annual Report of Major Natural Gas Companies)
* `PHMSA Natural Gas Annual Report <https://www.phmsa.dot.gov/data-and-statistics/pipeline/gas-distribution-gas-gathering-gas-transmission-hazardous-liquids>`__
* Machine Readable Specifications of State Clean Energy Standards

How do I access the data?
-------------------------

For details on how to access PUDL data, see the `data access documentation
<https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html>`__. A quick
summary:

* `Datasette <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#-access-datasette>`__
  provides browsable and queryable data from our nightly builds on the web:
  https://data.catalyst.coop
* `Kaggle <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-kaggle>`__
  provides easy Jupyter notebook access to the PUDL data, updated weekly:
  https://www.kaggle.com/datasets/catalystcooperative/pudl-project
* `Zenodo <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-zenodo>`__
  provides stable long-term access to our versioned data releases with a citeable DOI:
  https://doi.org/10.5281/zenodo.3653158
* `Nightly Data Builds <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-nightly-builds>`__
  push their outputs to the AWS Open Data Registry:
  https://registry.opendata.aws/catalyst-cooperative-pudl/
  See `the nightly build docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-nightly-builds>`__
  for direct download links.
* `The PUDL Development Environment <https://catalystcoop-pudl.readthedocs.io/en/nightly/dev/dev_setup.html>`__
  lets you run the PUDL data processing pipeline locally.

Contributing to PUDL
--------------------

Find PUDL useful? Want to help make it better? There are lots of ways to help!

* Check out our `contribution guide <https://catalystcoop-pudl.readthedocs.io/en/nightly/CONTRIBUTING.html>`__
  including our `Code of Conduct <https://catalystcoop-pudl.readthedocs.io/en/nightly/code_of_conduct.html>`__.
* You can file a bug report, make a feature request, or ask questions in the
  `Github issue tracker <https://github.com/catalyst-cooperative/pudl/issues>`__.
* Feel free to fork the project and make a pull request with new code, better
  documentation, or example notebooks.
* `Make a recurring financial contribution <https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=PZBZDFNKBJW5E&source=url>`__
  to support our work liberating public energy data.
* `Hire us to do some custom analysis <https://catalyst.coop/hire-catalyst/>`__ and
  allow us to integrate the resulting code into PUDL.

Licensing
---------

In general, our code, data, and other work are permissively licensed for use by anybody,
for any purpose, so long as you give us credit for the work we've done.

* The PUDL software is released under
  `the MIT License <https://opensource.org/licenses/MIT>`__.
* The PUDL data and documentation are published under the
  `Creative Commons Attribution License v4.0 <https://creativecommons.org/licenses/by/4.0/>`__
  (CC-BY-4.0).

Contact Us
----------

* For bug reports, feature requests, and other software or data issues please make a
  `GitHub Issue <https://github.com/catalyst-cooperative/pudl/issues>`__.
* For more general support, questions, or other conversations around the project
  that might be of interest to others, check out the
  `GitHub Discussions <https://github.com/catalyst-cooperative/pudl/discussions>`__
* If you'd like to get occasional updates about the project
  `sign up for our email list <https://catalyst.coop/updates/>`__.
* Want to schedule a time to chat with us one-on-one about your PUDL use case, ideas
  for improvement, or get some personalized support? Join us for
  `Office Hours <https://calend.ly/catalyst-cooperative/pudl-office-hours>`__
* `Follow us here on GitHub <https://github.com/catalyst-cooperative/>`__
* Follow us on Mastodon: `@CatalystCoop@mastodon.energy <https://mastodon.energy/@CatalystCoop>`__
* Follow us on BlueSky:  `@catalyst.coop <https://bsky.app/profile/catalyst.coop>`__
* `Follow us on LinkedIn <https://www.linkedin.com/company/catalyst-cooperative/>`__
* `Follow us on HuggingFace <https://huggingface.co/catalystcooperative>`__
* Follow us on Twitter: `@CatalystCoop <https://twitter.com/CatalystCoop>`__
* `Follow us on Kaggle <https://www.kaggle.com/catalystcooperative/>`__
* More info on our website: https://catalyst.coop
* Email us if you'd like to hire us to provide customized data extraction and analysis:
  `hello@catalyst.coop <mailto:hello@catalyst.coop>`__

About Catalyst Cooperative
--------------------------

`Catalyst Cooperative <https://catalyst.coop>`__ is a small group of data wranglers
and policy wonks organized as a worker-owned cooperative consultancy. Our goal is a
more just, livable, and sustainable world. We integrate public data and perform
custom analyses to inform public policy
(`Hire us! <https://catalyst.coop/hire-catalyst>`__). Our focus is primarily on
mitigating climate change and improving electric utility regulation in the United
States.


# Snowflake Connector for Python

[![Build and Test](https://github.com/snowflakedb/snowflake-connector-python/actions/workflows/build_test.yml/badge.svg)](https://github.com/snowflakedb/snowflake-connector-python/actions/workflows/build_test.yml)
[![codecov](https://codecov.io/gh/snowflakedb/snowflake-connector-python/branch/main/graph/badge.svg?token=MVKSNtnLr0)](https://codecov.io/gh/snowflakedb/snowflake-connector-python)
[![PyPi](https://img.shields.io/pypi/v/snowflake-connector-python.svg)](https://pypi.python.org/pypi/snowflake-connector-python/)
[![License Apache-2.0](https://img.shields.io/:license-Apache%202-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)
[![Codestyle Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

This package includes the Snowflake Connector for Python, which conforms to the [Python DB API 2.0](https://www.python.org/dev/peps/pep-0249/) specification.

The Snowflake Connector for Python provides an interface for developing Python
applications that can connect to Snowflake and perform all standard operations. It
provides a programming alternative to developing applications in Java or C/C++
using the Snowflake JDBC or ODBC drivers.

The connector has **no** dependencies on JDBC or ODBC.
It can be installed using ``pip`` on Linux, Mac OSX, and Windows platforms
where Python 3.8.0 (or higher) is installed.

Snowflake Documentation is available at:
https://docs.snowflake.com/

Feel free to file an issue or submit a PR here for general cases. For official support, contact Snowflake support at:
https://community.snowflake.com/s/article/How-To-Submit-a-Support-Case-in-Snowflake-Lodge

## How to build

### Locally

Install Python 3.8.0 or higher. Clone the Snowflake Connector for Python repository, then run the following commands
to create a wheel package using PEP-517 build:

```shell
git clone git@github.com:snowflakedb/snowflake-connector-python.git
cd snowflake-connector-python
python -m pip install -U pip setuptools wheel build
python -m build --wheel .
```

Find the `snowflake_connector_python*.whl` package in the `./dist` directory.

### In Docker
Or use our Dockerized build script `ci/build_docker.sh` and find the built wheel files in `dist/repaired_wheels`.

Note: `ci/build_docker.sh` can be used to compile only certain versions, like this: `ci/build_docker.sh "3.8 3.9"`

## Code hygiene and other utilities
These tools are integrated into `tox` to allow us to easily set them up universally on any computer.

* **fix_lint**: Runs `pre-commit` to check for a bunch of lint issues. This can be installed to run upon each
  time a commit is created locally, keep an eye out for the hint that this environment prints upon succeeding.
* **coverage**: Runs `coverage.py` to combine generated coverage data files. Useful when multiple categories were run
  and we would like to have an overall coverage data file created for them.
* **flake8**: (Deprecated) Similar to `fix_lint`, but only runs `flake8` checks.


# claripy
[![Latest Release](https://img.shields.io/pypi/v/claripy.svg)](https://pypi.python.org/pypi/claripy/)
[![Python Version](https://img.shields.io/pypi/pyversions/claripy)](https://pypi.python.org/pypi/claripy/)
[![PyPI Statistics](https://img.shields.io/pypi/dm/claripy.svg)](https://pypistats.org/packages/claripy)
[![License](https://img.shields.io/github/license/angr/claripy.svg)](https://github.com/angr/claripy/blob/master/LICENSE)

Claripy is an abstracted constraint-solving wrapper.

## Project Links
Project repository: https://github.com/angr/claripy

Documentation: https://api.angr.io/projects/claripy/en/latest/

## Usage

It is usable!

General usage is similar to Z3:

```python
>>> import claripy
>>> a = claripy.BVV(3, 32)
>>> b = claripy.BVS('var_b', 32)
>>> s = claripy.Solver()
>>> s.add(b > a)
>>> print(s.eval(b, 1)[0])
```


![Logo](https://raw.githubusercontent.com/piccolo-orm/piccolo/master/docs/logo_hero.png "Piccolo Logo")

![Tests](https://github.com/piccolo-orm/piccolo/actions/workflows/tests.yaml/badge.svg)
![Release](https://github.com/piccolo-orm/piccolo/actions/workflows/release.yaml/badge.svg)
[![Documentation Status](https://readthedocs.org/projects/piccolo-orm/badge/?version=latest)](https://piccolo-orm.readthedocs.io/en/latest/?badge=latest)
[![PyPI](https://img.shields.io/pypi/v/piccolo?color=%2334D058&label=pypi)](https://pypi.org/project/piccolo/)
[![codecov](https://codecov.io/gh/piccolo-orm/piccolo/branch/master/graph/badge.svg?token=V19CWH7MXX)](https://codecov.io/gh/piccolo-orm/piccolo)

Piccolo is a fast, user friendly ORM and query builder which supports asyncio. [Read the docs](https://piccolo-orm.readthedocs.io/en/latest/).

## Features

Some of it’s stand out features are:

- Support for sync and async.
- A builtin playground, which makes learning a breeze.
- Tab completion support - works great with iPython and VSCode.
- Batteries included - a User model, authentication, migrations, an [admin GUI](https://github.com/piccolo-orm/piccolo_admin), and more.
- Modern Python - fully type annotated.
- Make your codebase modular and scalable with Piccolo apps (similar to Django apps).

## Syntax

The syntax is clean and expressive.

You can use it as a query builder:

```python
# Select:
await Band.select(
    Band.name
).where(
    Band.popularity > 100
)

# Join:
await Band.select(
    Band.name,
    Band.manager.name
)

# Delete:
await Band.delete().where(
    Band.popularity < 1000
)

# Update:
await Band.update({Band.popularity: 10000}).where(
    Band.name == 'Pythonistas'
)
```

Or like a typical ORM:

```python
# To create a new object:
b = Band(name='C-Sharps', popularity=100)
await b.save()

# To fetch an object from the database, and update it:
b = await Band.objects().get(Band.name == 'Pythonistas')
b.popularity = 10000
await b.save()

# To delete:
await b.remove()
```

## Installation

Installing with PostgreSQL driver:

```bash
pip install 'piccolo[postgres]'
```

Installing with SQLite driver:

```bash
pip install 'piccolo[sqlite]'
```

Installing with all optional dependencies (easiest):

```bash
pip install 'piccolo[all]'
```

## Building a web app?

Let Piccolo scaffold you an ASGI web app, using Piccolo as the ORM:

```bash
piccolo asgi new
```

[Starlette](https://www.starlette.io/), [FastAPI](https://fastapi.tiangolo.com/), [BlackSheep](https://www.neoteroi.dev/blacksheep/), [Litestar](https://litestar.dev/), [Esmerald](https://esmerald.dev/) and [Lilya](https://lilya.dev) are currently supported.

## Are you a Django user?

We have a handy page which shows the equivalent of [common Django queries in Piccolo](https://piccolo-orm.readthedocs.io/en/latest/piccolo/query_types/django_comparison.html).

## Documentation

Our documentation is on [Read the docs](https://piccolo-orm.readthedocs.io/en/latest/piccolo/getting_started/index.html).

We also have some great [tutorial videos on YouTube](https://www.youtube.com/channel/UCE7x5nm1Iy9KDfXPNrNQ5lA).


Dask CUDA
=========

Various utilities to improve deployment and management of Dask workers on
CUDA-enabled systems.

This library is experimental, and its API is subject to change at any time
without notice.

Example
-------

```python
from dask_cuda import LocalCUDACluster
from dask.distributed import Client

cluster = LocalCUDACluster()
client = Client(cluster)
```

Documentation is available [here](https://docs.rapids.ai/api/dask-cuda/nightly/).

What this is not
----------------

This library does not automatically convert your Dask code to run on GPUs.

It only helps with deployment and management of Dask workers in multi-GPU
systems.  Parallelizing GPU libraries like [RAPIDS](https://rapids.ai) and
[CuPy](https://cupy.chainer.org) with Dask is an ongoing effort.  You may wish
to read about this effort at [blog.dask.org](https://blog.dask.org) for more
information.  Additional information about Dask-CUDA can also be found in the
[docs](https://docs.rapids.ai/api/dask-cuda/nightly/).


# INDRA

[![License](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](https://opensource.org/licenses/BSD-2-Clause)
[![Build](https://github.com/sorgerlab/indra/workflows/Tests/badge.svg)](https://github.com/sorgerlab/indra/actions)
[![Documentation](https://readthedocs.org/projects/indra/badge/?version=latest)](https://indra.readthedocs.io/en/latest/?badge=latest)
[![PyPI version](https://badge.fury.io/py/indra.svg)](https://badge.fury.io/py/indra)
[![Python 3](https://img.shields.io/pypi/pyversions/indra.svg)](https://www.python.org/downloads/release/python-357/)

INDRA (Integrated Network and Dynamical Reasoning Assembler) is an automated
model assembly system, originally developed for molecular systems biology and
then generalized to other domains (see [INDRA
World](https://github.com/indralab/indra_world)). INDRA draws on natural
language processing systems and structured databases to collect mechanistic and
causal assertions, represents them in a standardized form (INDRA Statements),
and assembles them into various modeling formalisms including causal graphs and
dynamical models.

At the core of INDRA are its knowledge-level assembly procedures, allowing
sources to be assembled into coherent models, a process that involves
correcting systematic input errors, finding and resolving redundancies,
inferring missing information, filtering to a relevant scope and assessing the
reliability of causal information.

The detailed INDRA documentation is available at
[http://indra.readthedocs.io](http://indra.readthedocs.io).

## Contents

- [INDRA Modules](#indra-modules)
    - [Knowledge sources](#knowledge-sources)
    - [Output model assemblers](#output-model-assemblers)
    - [Internal knowledge assembly](#internal-knowledge-assembly)
    - [Other modules](#other-modules)
- [Citation](#citation)
- [Installation](#installation)
- [INDRA REST API](#indra-rest-api)
- [INDRA Docker](#indra-docker)
- [Using INDRA](#using-indra)
- [Funding](#funding)

## INDRA Modules

### Knowledge sources

INDRA is currently integrated with the following natural language processing
systems and structured databases. These input modules (available in
`indra.sources`) all produce INDRA Statements.

Reading systems:

| Reader     | Module                                                                                | Reference                                       |
|------------|--------------------------------------------------------------------------------------------------------|-------------------------------------------------|
| TRIPS/DRUM | [`indra.sources.trips`](https://indra.readthedocs.io/en/latest/modules/sources/trips/index.html)       | http://trips.ihmc.us/parser/cgi/drum            |
| REACH      | [`indra.sources.reach`](https://indra.readthedocs.io/en/latest/modules/sources/reach/index.html)       | https://github.com/clulab/reach                 |
| Sparser    | [`indra.sources.sparser`](https://indra.readthedocs.io/en/latest/modules/sources/sparser/index.html#)  | https://github.com/ddmcdonald/sparser           |
| Eidos      | [`indra.sources.eidos`](https://indra.readthedocs.io/en/latest/modules/sources/eidos/index.html#)      | https://github.com/clulab/eidos                 |
| TEES       | [`indra.sources.tees`](https://indra.readthedocs.io/en/latest/modules/sources/tees/index.html)         | https://github.com/jbjorne/TEES                 |
| MedScan    | [`indra.sources.medscan`](https://indra.readthedocs.io/en/latest/modules/sources/medscan/index.html)   | https://doi.org/10.1093/bioinformatics/btg207   |
| RLIMS-P    | [`indra.sources.rlimsp`](https://indra.readthedocs.io/en/latest/modules/sources/rlimsp/index.html)     | https://research.bioinformatics.udel.edu/rlimsp |
| ISI/AMR    | [`indra.sources.isi`](https://indra.readthedocs.io/en/latest/modules/sources/isi/index.html)           | https://github.com/sgarg87/big_mech_isi_gg      |
| Geneways   | [`indra.sources.geneways`](https://indra.readthedocs.io/en/latest/modules/sources/geneways/index.html) | https://www.ncbi.nlm.nih.gov/pubmed/15016385    |
| GNBR       | [`indra.sources.gnbr`](https://indra.readthedocs.io/en/latest/modules/sources/gnbr/index.html)         | https://zenodo.org/record/3459420               |
| SemRep     | [`indra.sources.semrep`](https://indra.readthedocs.io/en/latest/modules/sources/semrep.html)     | https://github.com/lhncbc/SemRep                |

Biological pathway databases:

| Database / Exchange format | Module                     | Reference                                                       |
|----------------------------|----------------------------|-----------------------------------------------------------------|
| PathwayCommons / BioPax    | [`indra.sources.biopax`](https://indra.readthedocs.io/en/latest/modules/sources/biopax/index.html)     | http://pathwaycommons.org/
 http://www.biopax.org/         |
| Large Corpus / BEL         | [`indra.sources.bel`](https://indra.readthedocs.io/en/latest/modules/sources/bel/index.html)        | https://github.com/pybel/pybel
 https://github.com/OpenBEL |
| Signor                     | [`indra.sources.signor`](https://indra.readthedocs.io/en/latest/modules/sources/signor/index.html)     | https://signor.uniroma2.it/                                     |
| BioGRID                    | [`indra.sources.biogrid`](https://indra.readthedocs.io/en/latest/modules/sources/biogrid/index.html)    | https://thebiogrid.org/                                         |
| Target Affinity Spectrum   | [`indra.sources.tas`](https://indra.readthedocs.io/en/latest/modules/sources/tas/index.html#)        | https://doi.org/10.1101/358978                                  |
| HPRD                       | [`indra.sources.hprd`](https://indra.readthedocs.io/en/latest/modules/sources/hprd/index.html) | http://www.hprd.org                                             |                   |
| TRRUST                     | [`indra.sources.trrust`](https://indra.readthedocs.io/en/latest/modules/sources/trrust.html) | https://www.grnpedia.org/trrust/                                |                   |
| Phospho.ELM                | [`indra.sources.phosphoelm`](https://indra.readthedocs.io/en/latest/modules/sources/phosphoelm/index.html) | http://phospho.elm.eu.org/                                      |
| VirHostNet                | [`indra.sources.virhostnet`](https://indra.readthedocs.io/en/latest/modules/sources/virhostnet/index.html) | http://virhostnet.prabi.fr/                                     |
| CTD                  | [`indra.sources.ctd`](https://indra.readthedocs.io/en/latest/modules/sources/ctd/index.html) | http://ctdbase.org                                              |
| DrugBank                  | [`indra.sources.drugbank`](https://indra.readthedocs.io/en/latest/modules/sources/drugbank/index.html) | https://www.drugbank.ca/                                        |
| OmniPath                  | [`indra.sources.omnipath`](https://indra.readthedocs.io/en/latest/modules/sources/omnipath/index.html) | https://omnipathdb.org/                                         |
| DGI                  | [`indra.sources.dgi`](https://indra.readthedocs.io/en/latest/modules/sources/dgi/index.html) | https://www.dgidb.org/                                          |
| CRoG                  | [`indra.sources.crog`](https://indra.readthedocs.io/en/latest/modules/sources/crog/index.html) | https://github.com/chemical-roles/chemical-roles                |
| CREEDS                     | [`indra.sources.creeds`](https://indra.readthedocs.io/en/latest/modules/sources/creeds/index.html) | https://maayanlab.cloud/CREEDS/                                 |
| UbiBrowser                 | [`indra.sources.ubibrowser`](https://indra.readthedocs.io/en/latest/modules/sources/ubibrowser.html) | http://ubibrowser.ncpsb.org.cn/                |
| ACSN                       | [`indra.sources.acsn`](https://indra.readthedocs.io/en/latest/modules/sources/acsn.html) | https://acsn.curie.fr/ACSN2/ACSN2.html   |

Custom knowledge bases:

| Database / Exchange format | Module                        | Reference                            |
|----------------------------|-------------------------------|--------------------------------------|
| NDEx / CX                  | [`indra.sources.ndex_cx`](https://indra.readthedocs.io/en/latest/modules/sources/ndex_cx/index.html)       | http://ndexbio.org                   |
| INDRA DB / INDRA Statements| [`indra.sources.indra_db_rest`](https://indra.readthedocs.io/en/latest/modules/sources/indra_db_rest/index.html) | https://github.com/indralab/indra_db |
| Hypothes.is                  | [`indra.sources.hypothesis`](https://indra.readthedocs.io/en/latest/modules/sources/hypothesis/index.html)       | https://hypothes.is                   |
| Biofactoid                  | [`indra.sources.biofactoid`](https://indra.readthedocs.io/en/latest/modules/sources/biofactoid/index.html)       | https://biofactoid.org/                   |
| MINERVA                    | [`indra.sources.minerva`](https://indra.readthedocs.io/en/latest/modules/sources/minerva/index.html)         | https://covid19map.elixir-luxembourg.org/minerva/ |

### Output model assemblers

INDRA also provides several model output assemblers that take INDRA Statements
as input. The most sophisticated model assembler is the PySB Assembler, which
implements a policy-guided automated assembly procedure of a rule-based
executable model (that can then be further compiled into other formats such as
SBML, Kappa, BNGL and SBGN to connect to a vast ecosystem of downstream tools).
Several other model assembly modules target various network formats for
visualization, and graph/structural analysis (PyBEL, CyJS, Graphviz, SBGN,
CX, SIF, etc.) and curation (HTML, TSV, IndexCards).
Finally, the English Assembler produces English language descriptions of a set
of INDRA Statements.

INDRA also supports extension by outside model assembly tools which take
INDRA Statements as input and produce models. One such example is Delphi
(https://github.com/ml4ai/delphi), which is a Dynamic Bayesian Network
model assembler. Similarly, outside tools that support INDRA Statements
can implement custom visualization methods, such as CauseMos, developed
by Uncharted Software (https://uncharted.software/).

Assemblers aimed at model-driven discovery and analysis:

| Modeling formalism / Exchange format           | Purpose                                              | Module                  | Reference           |
|------------------------------------------------|------------------------------------------------------|-------------------------|---------------------|
| PySB (-> SBML, SBGN, BNGL, Kappa, etc.)        | Detailed, mechanistic modeling, simulation, analysis | [`indra.assemblers.pysb`](https://indra.readthedocs.io/en/latest/modules/assemblers/pysb_assembler.html#) | http://pysb.org     |
| PyBEL                                          | Causal analysis, visualization                       | [`indra.assemblers.pybel`](https://indra.readthedocs.io/en/latest/modules/assemblers/pybel_assembler.html)| https://github.com/pybel/pybel
 https://bel-commons.scai.fraunhofer.de/ |
| IndraNet                                       | Causal analysis, signed and unsigned                 | [`indra.assemblers.indranet`](https://indra.readthedocs.io/en/latest/modules/assemblers/indranet_assembler.html) |                  |
| SIF                                            | Network analysis, logic modeling, visualization      | [`indra.assemblers.sif`](https://indra.readthedocs.io/en/latest/modules/assemblers/sif_assembler.html)  | [SIF format](http://manual.cytoscape.org/en/stable/Supported_Network_File_Formats.html#sif-format) |
| KAMI                                           | Knowledge aggregation of protein sites/states and Kappa modeling | [`indra.assemblers.kami`](https://indra.readthedocs.io/en/latest/modules/assemblers/kami_assembler.html) | https://github.com/Kappa-Dev/KAMI |

Assemblers primarily aimed at visualization:

| Network / Exchange format                      | Purpose                                              | Module                  | Reference           |
|------------------------------------------------|------------------------------------------------------|-------------------------|---------------------|
| Causal Analysis Graph                          | General causal graph visualization                   | [`indra.assemblers.cag`](https://indra.readthedocs.io/en/latest/modules/assemblers/cag_assembler.html)  |                     |
| CX                                             | Network browsing, versioning on NDEx                 | [`indra.assemblers.cx`](https://indra.readthedocs.io/en/latest/modules/assemblers/cx_assembler.html)   | http://ndexbio.org  |
| Cytoscape JS                                   | Interactive Cytoscape JS network to embed in websites| [`indra.assemblers.cyjs`](https://indra.readthedocs.io/en/latest/modules/assemblers/cyjs_assembler.html) | http://js.cytoscape.org/ |
| Graphviz                                       | Static PDF/PNG visualization with powerful automated layout using Graphviz | [`indra.assemblers.graph`](https://indra.readthedocs.io/en/latest/modules/assemblers/graph_assembler.html) | https://www.graphviz.org/ |
| SBGN                                           | Visualization with Systems Biology Graphical Notation| [`indra.assemblers.sbgn`](https://indra.readthedocs.io/en/latest/modules/assemblers/sbgn_assembler.html) | http://sbgn.org     |

Assemblers primarily aimed at expert curation and browsing:

| Output format                                  | Purpose                                                | Module                  | Reference           |
|------------------------------------------------|------------------------------------------------------  |-------------------------|---------------------|
| English language                               | Human-readable descriptions, reports, dialogue         | [`indra.assemblers.english`](https://indra.readthedocs.io/en/latest/modules/assemblers/english_assembler.html) |                  |
| HTML                                           | Web-based browsing, linking out to provenance, curation| [`indra.assemblers.html`](https://indra.readthedocs.io/en/latest/modules/assemblers/html_assembler.html) | [Curation tutorial](https://indra.readthedocs.io/en/latest/tutorials/html_curation.html) |
| TSV (Tab/Comma Separated Values)               | Spreadsheet-based browsing and curation                | [`indra.assemblers.tsv`](https://indra.readthedocs.io/en/latest/modules/assemblers/tsv_assembler.html)  |                     |
| Index Cards                                    | Custom JSON format for curating biological mechanisms  | [`indra.assemblers.index_card`](https://indra.readthedocs.io/en/latest/modules/assemblers/index_card_assembler.html) |               |

### Internal knowledge assembly

A key feature of INDRA is providing internal knowledge-assembly modules
that operate on INDRA Statements and perform the following tasks:
- Redundancy/subsumption/generalization/contradiction finding and resolution
with respect to an ontology with the Preassembler
([`indra.preassembler.Preassembler`](https://indra.readthedocs.io/en/latest/modules/preassembler/index.html#module-indra.preassembler))
- Belief calculation based on evidence using the BeliefEngine
([`indra.belief`](https://indra.readthedocs.io/en/latest/modules/belief/index.html))
- Mapping grounding between multiple ontologies
([`indra.preassembler.ont_mapper.OntMapper`](https://indra.readthedocs.io/en/latest/modules/preassembler/index.html#module-indra.preassembler.ontology_mapper))
- Grounding override and disambiguation
([`indra.preassembler.grounding_mapper.GroundingMapper`](https://indra.readthedocs.io/en/latest/modules/preassembler/index.html#module-indra.preassembler.grounding_mapper))
- Protein sequence mapping ([`indra.preassembler.site_mapper.SiteMapper`](https://indra.readthedocs.io/en/latest/modules/preassembler/index.html#module-indra.preassembler.sitemapper))

The internal assembly steps of INDRA including the ones listed above, and also
a large collection of filters (filter by source, belief, the presence of
grounding information, semantic filters by entity role, etc.) are exposed
in the
[indra.tools.assemble_corpus](http://indra.readthedocs.io/en/latest/modules/tools/index.html#module-indra.tools.assemble_corpus)
submodule. This submodule contains functions that
take Statements as input and produce processed Statements as output. They can
be composed to form an assembly pipeline connecting knowledge collected from
sources with an output model.

This diagram illustrates the assembly pipeline process.

![assembly](doc/images/assembly.png)

The choice of assembly functions can vary depending on the domain (i.e,
biology or world modeling), the modeling goal (i.e., the type of model that
will be assembled and how that model will be used), desired features, and
confidence (e.g., filter to human genes only or apply a belief cutoff),
and any other user preferences.

An example of a typical assembly pipeline for biology statements is as follows.
Some of the below steps can be removed, rearranged, and other steps added
to change the assembly pipeline.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.tools import assemble_corpus as ac
stmts = <the collection of all raw statements to use>
stmts = ac.filter_no_hypothesis(stmts)  # Filter out hypothetical statements
stmts = ac.map_grounding(stmts)         # Map grounding
stmts = ac.filter_grounded_only(stmts)  # Filter out ungrounded agents
stmts = ac.filter_human_only(stmts)     # Filter out non-human genes
stmts = ac.map_sequence(stmts)          # Map sequence
stmts = ac.run_preassembly(stmts,       # Run preassembly
                           return_toplevel=False)
stmts = ac.filter_belief(stmts, 0.8)    # Apply belief cutoff of 0.8
```

An example of an assembly pipeline for statements in the world modeling domain
is as follows (note how biology-specific functions are not used, and a custom
belief_scorer and ontology is passed to `run_preassembly` here, while the
biology pipeline used default values). Note that this example requires
the `indra_world` package to be installed.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.tools import assemble_corpus as ac
from indra_world.belief.wm_scorer import get_eidos_scorer
from indra_world.ontology.world import world_ontology
stmts = <the collection of all raw statements to use>
stmts = ac.filter_grounded_only(stmts)  # Filter out ungrounded agents
belief_scorer = get_eidos_scorer()
stmts = ac.run_preassembly(stmts,       # Run preassembly
                           return_toplevel=False,
                           belief_scorer=belief_scorer,
                           ontology=world_ontology,
                           normalize_equivalences=True,     # Optional: rewrite equivalent groundings to one standard
                           normalize_opposites=True,        # Optional: rewrite opposite groundings to one standard
                           normalize_ns='WM')               # Use 'WM' namespace to normalize equivalences and opposites
stmts = ac.filter_belief(stmts, 0.8)    # Apply belief cutoff of e.g., 0.8
```
Assembled statements returned after running the assembly pipeline can be
passed into any of the output model assemblers.

### Other modules

INDRA also contains modules to access literature content (e.g., PubMed, Elsevier), available in [`indra.literature`](
https://indra.readthedocs.io/en/latest/modules/literature/index.html), and
access ontological information and convert between identifiers (e.g., UniProt,
HGNC), available in [`indra.databases`](
https://indra.readthedocs.io/en/latest/modules/databases/index.html).
A full list of further INDRA modules is available in the [`documentation`](
https://indra.readthedocs.io/en/latest/modules/index.html).

## Citation

Gyori B.M., Bachman J.A., Subramanian K., Muhlich J.L., Galescu L., Sorger P.K.
[From word models to executable models of signaling networks using automated
assembly](http://msb.embopress.org/content/13/11/954) (2017),
Molecular Systems Biology, 13, 954.

Bachman J.A., Gyori B.M., Sorger P.K.
[Automated assembly of molecular mechanisms at scale from text mining and
curated databases](https://doi.org/10.15252/msb.202211325) (2023),
Molecular Systems Biology, e11325.

## Installation

For detailed installation instructions,
[see the documentation](http://indra.readthedocs.io/en/latest/installation.html).

INDRA currently supports Python 3.8-3.10. The last release of INDRA compatible
with Python 2.7 is 1.10, the last release fully compatible with Python 3.5
is 1.17. Most usages of INDRA will work with other Python versions, however,
full compatibility is currently only tested with 3.8-3.10.

The preferred way to install INDRA is by pointing pip to the source repository
as

    $ pip install git+https://github.com/sorgerlab/indra.git

Releases of INDRA are also available on
[PyPI](https://pip.pypa.io/en/latest/installing/), you can install the latest
release as

    $ pip install indra

However, releases will usually be behind the latest code available in this
repository.

INDRA depends on a few standard Python packages. These packages are installed
by pip during setup.
For certain modules and use cases, other "extra" dependencies may be needed,
which are described in detail in the
[documentation](http://indra.readthedocs.io/en/latest/installation.html).

## INDRA REST API
A REST API for INDRA is available at http://api.indra.bio:8000.
Note that the REST API is ideal for prototyping and for building light-weight
web apps, but should not be used for large reading and assembly workflows.

## INDRA Docker
INDRA is available as a Docker image on Dockerhub and can be pulled as

```
docker pull labsyspharm/indra
```

You can run the INDRA REST API using the container as
```
docker run -id -p 8080:8080 --entrypoint python labsyspharm/indra /sw/indra/rest_api/api.py
```

The Dockerfile to build the image locally is available in a separate repository
at https://github.com/indralab/indra_docker.

## Using INDRA

In this example INDRA assembles a PySB model from the natural language
description of a mechanism via the [TRIPS reading web
service](http://trips.ihmc.us/parser/cgi/drum).

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.sources import trips
from indra.assemblers.pysb import PysbAssembler
pa = PysbAssembler()
# Process a natural language description of a mechanism
trips_processor = trips.process_text('MEK2 phosphorylates ERK1 at Thr-202 and Tyr-204')
# Collect extracted mechanisms in PysbAssembler
pa.add_statements(trips_processor.statements)
# Assemble the model
model = pa.make_model(policies='two_step')
```

INDRA also provides an interface for the
[REACH](http://github.com/clulab/reach) natural language
processor. In this example, a full paper from [PubMed
Central](http://www.ncbi.nlm.nih.gov/pmc/) is processed. The paper's PMC ID is
[PMC8511698](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8511698/). The example
assumest that a REACH server is running locally (see documentation at
[`indra.sources.reach`](https://indra.readthedocs.io/en/latest/modules/sources/reach/index.html)).
Note that REACH takes about 8 minutes to read this full-text paper.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.sources import reach
reach_processor = reach.process_pmc('PMC8511698', url=reach.local_nxml_url)
```
At this point, `reach_processor.statements` contains a list of INDRA statements
extracted from the PMC paper.

Next we look at an example of reading the 10 most recent PubMed abstracts on
BRAF and collecting the results in INDRA statements.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.sources import reach
from indra.literature import pubmed_client
# Search for 10 most recent abstracts in PubMed on 'BRAF'
pmids = pubmed_client.get_ids('BRAF', retmax=10)
all_statements = []
for pmid in pmids:
    abs = pubmed_client.get_abstract(pmid)
    if abs is not None:
        reach_processor = reach.process_text(abs, url=reach.local_text_url)
        if reach_processor is not None:
            all_statements += reach_processor.statements
```
At this point, the `all_statements` list contains all the statements
extracted from the 10 abstracts.

The next example shows querying the [BEL large
corpus](https://github.com/cthoyt/selventa-knowledge)
network for a neighborhood of a given list of proteins using their
HGNC gene names. This example performs the query via PyBEL.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.sources import bel
# Process the neighborhood of BRAF and MAP2K1
bel_processor = bel.process_pybel_neighborhood(['BRAF', 'MAP2K1'])
```
At this point, `bel_processor.statements` contains a list of INDRA statements
extracted from the neighborhood query.

Next, we look at an example of querying the [Pathway Commons
database](http://pathwaycommons.org) for paths between two lists of proteins.

[//]: # (If code is changed here, also update it in tests/test_docs_code.py)

```python
from indra.sources import biopax
# Process the neighborhood of BRAF and MAP2K1
biopax_processor = biopax.process_pc_pathsfromto(['BRAF', 'RAF1'], ['MAP2K1', 'MAP2K2'])
```
At this point, `biopax_processor.statements` contains a list of INDRA
Statements extracted from the paths-from-to query.

## Funding

The development of INDRA has been funded from the following sources:

| Program                                          | Grant number         |
|--------------------------------------------------|----------------------|
| DARPA Big Mechanism                              | W911NF-14-1-0397     |
| DARPA World Modelers                             | W911NF-18-1-0014     |
| DARPA Communicating with Computers               | W911NF-15-1-0544     |
| DARPA Automated Scientific Discovery Framework   | W911NF-18-1-0124     |
| DARPA Automating Scientific Knowledge Extraction | HR00111990009        |
| DARPA Panacea                                    | HR00111920022        |
| DARPA Young Faculty Award                        | W911NF-20-1-0255     |