# PVNet 2.1

[![Python Bump Version & release](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml/badge.svg)](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml)

This project is used for training PVNet and running PVNet on live data.

PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and
satellite data are sent through some neural network which encodes them down to
1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)
output history, the calculated solar coordinates (azimuth and elevation) and the
GSP ID which has been put through an embedding layer. This 1D concatenated
feature vector is put through an output network which outputs predictions of the
future GSP yield. National forecasts are made by adding all the GSP forecasts
together.

## Experiments

Our paper based on this repo was accepted into the Tackling Climate Change with Machine Learning workshop at ICLR 2024 and can be viewed [here](https://www.climatechange.ai/papers/iclr2024/46).

Some slightly more structured notes on deliberate experiments we have performed with PVNet are [here](https://docs.google.com/document/d/1VumDwWd8YAfvXbOtJEv3ZJm_FHQDzrKXR0jU9vnvGQg).

Some very rough, early working notes on this model are
[here](https://docs.google.com/document/d/1fbkfkBzp16WbnCg7RDuRDvgzInA6XQu3xh4NCjV-WDA). These are now somewhat out of date.

## Setup / Installation

```bash
git clone https://github.com/openclimatefix/PVNet.git
cd PVNet
pip install .
```

The commit history is extensive. To save download time, use a depth of 1:
```bash
git clone --depth 1 https://github.com/openclimatefix/PVNet.git
```
This means only the latest commit and its associated files will be downloaded.

Next, in the PVNet repo, install PVNet as an editable package:

```bash
pip install -e .
```

### Additional development dependencies

```bash
pip install ".[dev]"
```

## Getting started with running PVNet

Before running any code in PVNet, copy the example configuration to a
configs directory:

```
cp -r configs.example configs
```

You will be making local amendments to these configs. See the README in
`configs.example` for more info.

### Datasets

As a minimum, in order to create batches of data/run PVNet, you will need to
supply paths to NWP and GSP data. PV data can also be used. We list some
suggested locations for downloading such datasets below:

**GSP (Grid Supply Point)** - Regional PV generation data\
The University of Sheffield provides API access to download this data:
https://www.solar.sheffield.ac.uk/api/

Documentation for querying generation data aggregated by GSP region can be found
here:
https://docs.google.com/document/d/e/2PACX-1vSDFb-6dJ2kIFZnsl-pBQvcH4inNQCA4lYL9cwo80bEHQeTK8fONLOgDf6Wm4ze_fxonqK3EVBVoAIz/pub#h.9d97iox3wzmd

**NWP (Numerical weather predictions)**\
OCF maintains a Zarr formatted version of the German Weather Service's (DWD)
ICON-EU NWP model here:
https://huggingface.co/datasets/openclimatefix/dwd-icon-eu which includes the UK

**PV**\
OCF maintains a dataset of PV generation from 1311 private PV installations
here: https://huggingface.co/datasets/openclimatefix/uk_pv

### Connecting with ocf_datapipes for batch creation

Outside the PVNet repo, clone the ocf-datapipes repo and exit the conda env created for PVNet: https://github.com/openclimatefix/ocf_datapipes
```bash
git clone --depth 1 https://github.com/openclimatefix/ocf_datapipes.git
conda create -n ocf_datapipes python=3.10
```

Then go inside the ocf_datapipes repo to add packages

```bash
pip install ".[dev]"
```

Then exit this environment, and enter back into the pvnet conda environment and install ocf_datapies in editable mode (-e). This means the package is directly linked to the source code in the ocf_datapies repo.

```bash
pip install -e <PATH-TO-ocf_datapipes-REPO>
```

## Generating pre-made batches of data for training/validation of PVNet

PVNet contains a script for generating batches of data suitable for training the PVNet models. To run the script you will need to make some modifications to the datamodule configuration.

Make sure you have copied the example configs (as already stated above):
```
cp -r configs.example configs
```

### Set up and config example for batch creation

We will use the following example config file for creating batches: `/PVNet/configs/datamodule/configuration/example_configuration.yaml`. Ensure that the file paths are set to the correct locations in `example_configuration.yaml`: search for `PLACEHOLDER` to find where to input the location of the files. You will need to comment out or delete the parts of `example_configuration.yaml` pertaining to the data you are not using.

When creating batches, an additional datamodule config located in `PVNet/configs/datamodule` is passed into the batch creation script: `streamed_batches.yaml`. Like before, a placeholder variable is used when specifying which configuration to use:

```yaml
configuration: "PLACEHOLDER.yaml"
```

This should be given the whole path to the config on your local machine, for example:

```yaml
configuration: "/FULL-PATH-TO-REPO/PVNet/configs/datamodule/configuration/example_configuration.yaml"
```

Where `FULL-PATH-TO-REPO` represent the whole path to the PVNet repo on your local machine.

This is also where you can update the train, val & test periods to cover the data you have access to.

### Running the batch creation script

Run the `save_batches.py` script to create batches with the parameters specified in the datamodule config (`streamed_batches.yaml` in this example):

```bash
python scripts/save_batches.py
```
PVNet uses
[hydra](https://hydra.cc/) which enables us to pass variables via the command
line that will override the configuration defined in the `./configs` directory, like this:

```bash
python scripts/save_batches.py datamodule=streamed_batches datamodule.batch_output_dir="./output" datamodule.num_train_batches=10 datamodule.num_val_batches=5
```

`scripts/save_batches.py` needs a config under `PVNet/configs/datamodule`. You can adapt `streamed_batches.yaml` or create your own in the same folder.

If downloading private data from a GCP bucket make sure to authenticate gcloud (the public satellite data does not need authentication):

```
gcloud auth login
```

Files stored in multiple locations can be added as a list. For example, in the `example_configuration.yaml` file we can supply a path to satellite data stored on a bucket:

```yaml
satellite:
    satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
```

Or to satellite data hosted by Google:

```yaml
satellite:
    satellite_zarr_paths:
      - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr"
      - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"
```

Datapipes are currently set up to use 11 channels from the satellite data, the 12th of which is HRV and is not included in these.

### Training PVNet

How PVNet is run is determined by the extensive configuration in the config
files. The configs stored in `PVNet/configs.example` should work with batches created using the steps and batch creation config mentioned above.

Make sure to update the following config files before training your model:

1. In `configs/datamodule/local_premade_batches.yaml`:
    - update `batch_dir` to point to the directory you stored your batches in during batch creation
2. In `configs/model/local_multimodal.yaml`:
    - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
        - `in_channels`: number of variables your NWP source supplies
        - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `nwp_image_size_pixels_height` and/or `nwp_image_size_pixels_width` in `datamodule/example_configs.yaml`, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
3. In `configs/local_trainer.yaml`:
    - set `accelerator: 0` if running on a system without a supported GPU

If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
your customised config files:

```yaml
defaults:
  - trainer: local_trainer.yaml
  - model: local_multimodal.yaml
  - datamodule: local_premade_batches.yaml
  - callbacks: null
  - logger: csv.yaml
  - experiment: null
  - hparams_search: null
  - hydra: default.yaml
```

Assuming you ran the `save_batches.py` script to generate some premade train and
val data batches, you can now train PVNet by running:

```
python run.py
```

## Backtest

If you have successfully trained a PVNet model and have a saved model checkpoint you can create a backtest using this, e.g. forecasts on historical data to evaluate forecast accuracy/skill. This can be done by running one of the scripts in this repo such as [the UK GSP backtest script](scripts/backtest_uk_gsp.py) or the [the pv site backtest script](scripts/backtest_sites.py), further info on how to run these are in each backtest file.

## Testing

You can use `python -m pytest tests` to run tests


![Data Safe Haven cartoon by Scriberia for The Alan Turing Institute](docs/source/_static/scriberia_diagram.jpg)

# :eyes: What is the Turing Data Safe Haven?

The **Turing Data Safe Haven** is an open-source framework for creating secure environments to analyse sensitive data.
It provides a set of scripts and templates that will allow you to deploy, administer and use your own secure environment.
It was developed as part of the Alan Turing Institute's [Data Safe Havens in the Cloud](https://www.turing.ac.uk/research/research-projects/data-safe-havens-cloud) project.

[![Documentation](https://readthedocs.org/projects/data-safe-haven/badge/?version=latest)](https://data-safe-haven.readthedocs.io/en/latest/?badge=latest)
[![Lint code](https://github.com/alan-turing-institute/data-safe-haven/actions/workflows/lint_code.yaml/badge.svg)](https://github.com/alan-turing-institute/data-safe-haven/actions/workflows/lint_code.yaml)
[![Test code](https://github.com/alan-turing-institute/data-safe-haven/actions/workflows/test_code.yaml/badge.svg)](https://github.com/alan-turing-institute/data-safe-haven/actions/workflows/test_code.yaml)
[![Latest version](https://img.shields.io/github/v/release/alan-turing-institute/data-safe-haven?style=flat&label=Latest&color=%234B78E6)](https://github.com/alan-turing-institute/data-safe-haven/releases)
[![Slack](https://img.shields.io/badge/Join%20us!-yellow?style=flat&logo=slack&logoColor=white&labelColor=4A154B&label=Slack)](https://join.slack.com/t/turingdatasafehaven/signup)
![Licence](https://img.shields.io/github/license/alan-turing-institute/data-safe-haven)
[![Citation](https://img.shields.io/badge/citation-cite%20this%20project-informational)](https://github.com/alan-turing-institute/data-safe-haven/blob/develop/CITATION.cff)

[![All Contributors](https://img.shields.io/badge/all_contributors-48-orange.svg?style=flat-square)](#contributors-)

## :family: Community & support

- Visit the [Data Safe Haven website](https://data-safe-haven.readthedocs.io) for full documentation and useful links.
- Join our [Slack server](https://join.slack.com/t/turingdatasafehaven/shared_invite/zt-104oyd8wn-DyOufeaAQFiJDlG5dDGk~w) to ask questions, discuss features, and for general API chat.
- Open a [discussion on GitHub](https://github.com/alan-turing-institute/data-safe-haven/discussions) for general questions, feature suggestions, and help with our deployment scripts.
- Look through our [issues on GitHub](https://github.com/alan-turing-institute/data-safe-haven/issues) to see what we're working on and progress towards specific fixes.
- Subscribe to the [Data Safe Haven newsletter](https://tinyletter.com/turingdatasafehaven) for release announcements.

## :open_hands: Contributing

We are keen to transition our implementation from being a [Turing](https://www.turing.ac.uk/) project to being a community owned platform.
We have worked together with the community to develop the policy, processes and design decisions for the Data Safe Haven.

We welcome contributions from anyone who is interested in the project.
There are lots of ways to contribute, not just writing code!

See our [Code of Conduct](CODE_OF_CONDUCT.md) and our [Contributor Guide](CONTRIBUTING.md) to learn more about how we work together as a community and how you can contribute.

## Contributors

Alvaro Cabrejas Egea
💻 🖋

Callum Mole
🐛 💻

Carlos Gavidia-Calderon
🐛

Catalina Vallejos
🖋

Christopher Edsall
💻 📖 🐛

DDelbarre
🐛

Daniel
💻 🐛

David Beavan
📖 🖋

Diego Arenas
💻 🤔 🖋

Ed Chalstrey
💻 📖 🐛 🤔 📋 👀 ⚠

Evelina Gabasova
🖋

Federico Nanni
💻 🐛 📖 🤔

Franz Király
🖋

Guillaume Noell
📖 🐛 🤔

Helen D Little
🐛 👀

Helen Sherwood-Taylor
🤔 🖋

Jack Roberts
💻 🐛

James Cunningham
💻 📖 🐛 🤔 🖋

James Geddes
🖋

James Hetherington
📖 🐛 🤔 🔍 📆 📣 📢 🖋

James Robinson
💻 🖋 📖 🐛 🤔 🔍 📋 🚇 📆 📣 💬 👀 🛡 ⚠ 📢

Jim Madge
💻 📖 🐛 🤔 🔍 📋 🚇 📆 📣 💬 👀 🛡 ⚠

Josh Everett
🐛

Jules M
📖 🤔 🐛 🖋

Kirstie Whitaker
🖋 📖 🐛 🤔 🔍 📋 📆 📣 📢 📓

Martin O'Reilly
💻 🖋 📖 🐛 🤔 🔍 📋 🚇 📆 📣 💬 👀 🛡 ⚠ 📢

Matt Craddock
💻 📖 🐛 🤔 🔍 📋 📣 💬 👀 🛡 ⚠

Oscar T Giles
💻 📖 🤔

Radka Jersakova
🖋

Rob Clarke
🤔 🐛 💻 📖 🖋

Steven Carlysle-Davies
💻 🖋 🤔

Tim Hobson
💻 🐛 📖 🤔

Tom Doel
💻 📖 🐛 🤔 🖋

Tomas Lazauskas
💻 📖 🐛 🤔

arielle-bennett
🔍 🤔 📆

bw-faststream
📖 🤔 🐛 📆 📓

cathiest
🖋 📖 🐛 🤔

davsarper
📖 📋 🔍 🤔 📆 📣 📢

ens-brett-todd
💻 🤔

ens-george-holmes
💻 🤔

getcarter21
💻 🖋

harisood
📖 🐛 🤔 🔍 📋 📆 📣 💬 📢 🛡 📓

kevinxufs
📖 🤔 🛡

miguelmorin
💻 📖 🤔 ⚠

oforrest
📖 🤔 📆 📣 🖋

rwinstanley1
📖 🤔 📆 🛡

vollmersj
📖 🐛 🤔 🖋

warwick26
💻 🤔

## :cake: Releases

If you're new to the project, why not check out our [latest release](https://github.com/alan-turing-institute/data-safe-haven/releases/latest)?

You can also browse [all our releases](https://github.com/alan-turing-institute/data-safe-haven/releases).
Follow the link from any release to view and clone this repository as at that release.

Read our [versioning scheme](VERSIONING.md) for how we number and label releases, as well as details of releases that have been used in production and releases that have undergone formal security evaluation.

When making a new release, open an issue on GitHub and choose the `Release checklist` template, which can be used to track the completion of security checks for the release.

## :mailbox_with_mail: Vulnerability disclosure

We value those who take the time and effort to report security vulnerabilities.
If you believe you have found a security vulnerability, please report it as outlined in our [Security and vulnerability disclosure policy](SECURITY.md).

## :bow: Acknowledgements

We are grateful for the following support for this project:

- The Alan Turing Institute's core and additional EPSRC funding ([EP/N510129/1](https://gtr.ukri.org/projects?ref=EP%2FN510129%2F1), [EP/W001381/1](https://gtr.ukri.org/projects?ref=EP%2FW001381%2F1), [EP/W037211/1](https://gtr.ukri.org/projects?ref=EP%2FW037211%2F1), [EP/X03870X/1](https://gtr.ukri.org/projects?ref=EP%2FX03870X%2F1)).
- The UKRI Strategic Priorities Fund - AI for Science, Engineering, Health and Government programme ([EP/T001569/1](https://gow.epsrc.ukri.org/NGBOViewGrant.aspx?GrantRef=EP/T001569/1)), particularly the "Tools, Practices and Systems" theme within that grant.
- Microsoft's generous [donation of Azure credits](https://www.microsoft.com/en-us/research/blog/microsoft-accelerates-data-science-at-the-alan-turing-institute-with-5m-in-cloud-computing-credits/) to the Alan Turing Institute.

## :warning: Disclaimer

The Alan Turing Institute and its group companies ("we", "us", the "Turing") make no representations, warranties, or guarantees, express or implied, regarding the information contained in this repository, including but not limited to information about the use or deployment of the Data Safe Haven and/or related materials.
We expressly exclude any implied warranties or representations whatsoever including without limitation regarding the use of the Data Safe Haven and related materials for any particular purpose.
The Data Safe Haven and related materials are provided on an 'as is' and 'as available' basis and you use them at your own cost and risk.
To the fullest extent permitted by law, the Turing excludes any liability arising from your use of or inability to use this repository, any of the information or materials contained on it, and/or the Data Safe Haven.

Deployments of the Data Safe Haven code and/or related materials depend on their specific implementation into different environments and we cannot account for all of these variations.
Safe use of any Data Safe Haven code or materials also relies upon individuals' and their organisations' good and responsible data handling processes and protocols and we make no representations and give no guarantees regarding the safety, security or suitability of any instance(s) of the deployment of the Data Safe Haven.
The Turing assumes no responsibility for updating any of the content in this repository; however, the underlying code and related materials may change from time to time with updates and it is the user's responsibility to keep abreast of these updates.


# Vega-Altair

[![github actions](https://github.com/vega/altair/workflows/build/badge.svg)](https://github.com/vega/altair/actions?query=workflow%3Abuild)
[![typedlib_mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://www.mypy-lang.org)
[![JOSS Paper](https://joss.theoj.org/papers/10.21105/joss.01057/status.svg)](https://joss.theoj.org/papers/10.21105/joss.01057)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/altair)](https://pypi.org/project/altair)

**Vega-Altair** is a declarative statistical visualization library for Python. With Vega-Altair, you can spend more time understanding your data and its meaning. Vega-Altair's
API is simple, friendly and consistent and built on top of the powerful
[Vega-Lite](https://github.com/vega/vega-lite) JSON specification. This elegant
simplicity produces beautiful and effective visualizations with a minimal amount of code.

*Vega-Altair was originally developed by [Jake Vanderplas](https://github.com/jakevdp) and [Brian
Granger](https://github.com/ellisonbg) in close collaboration with the [UW
Interactive Data Lab](https://idl.cs.washington.edu/).*
*The Vega-Altair open source project is not affiliated with Altair Engineering, Inc.*

## Documentation

See [Vega-Altair's Documentation Site](https://altair-viz.github.io) as well as the [Tutorial Notebooks](https://github.com/altair-viz/altair_notebooks). You can
run the notebooks directly in your browser by clicking on one of the following badges:

[![Binder](https://beta.mybinder.org/badge.svg)](https://beta.mybinder.org/v2/gh/altair-viz/altair_notebooks/master)
[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/altair-viz/altair_notebooks/blob/master/notebooks/Index.ipynb)

## Example

Here is an example using Vega-Altair to quickly visualize and display a dataset with the native Vega-Lite renderer in the JupyterLab:

```python
import altair as alt

# load a simple dataset as a pandas DataFrame
from vega_datasets import data
cars = data.cars()

alt.Chart(cars).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color='Origin',
)
```

![Vega-Altair Visualization](https://raw.githubusercontent.com/altair-viz/altair/main/images/cars.png)

One of the unique features of Vega-Altair, inherited from Vega-Lite, is a declarative grammar of not just visualization, but _interaction_.
With a few modifications to the example above we can create a linked histogram that is filtered based on a selection of the scatter plot.

```python
import altair as alt
from vega_datasets import data

source = data.cars()

brush = alt.selection_interval()

points = alt.Chart(source).mark_point().encode(
    x='Horsepower',
    y='Miles_per_Gallon',
    color=alt.condition(brush, 'Origin', alt.value('lightgray'))
).add_params(
    brush
)

bars = alt.Chart(source).mark_bar().encode(
    y='Origin',
    color='Origin',
    x='count(Origin)'
).transform_filter(
    brush
)

points & bars
```

![Vega-Altair Visualization Gif](https://raw.githubusercontent.com/altair-viz/altair/main/images/cars_scatter_bar.gif)

## Features

* Carefully-designed, declarative Python API.
* Auto-generated internal Python API that guarantees visualizations are type-checked and
  in full conformance with the [Vega-Lite](https://github.com/vega/vega-lite)
  specification.
* Display visualizations in JupyterLab, Jupyter Notebook, Visual Studio Code, on GitHub and
  [nbviewer](https://nbviewer.jupyter.org/), and many more.
* Export visualizations to various formats such as PNG/SVG images, stand-alone HTML pages and the
[Online Vega-Lite Editor](https://vega.github.io/editor/#/).
* Serialize visualizations as JSON files.

## Installation

Vega-Altair can be installed with:
```bash
pip install altair
```

If you are using the conda package manager, the equivalent is:
```bash
conda install altair -c conda-forge
```

For full installation instructions, please see [the documentation](https://altair-viz.github.io/getting_started/installation.html).

## Getting Help

If you have a question that is not addressed in the documentation,
you can post it on [StackOverflow](https://stackoverflow.com/questions/tagged/altair) using the `altair` tag.
For bugs and feature requests, please open a [Github Issue](https://github.com/vega/altair/issues).

## Development

[![Hatch project](https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg)](https://github.com/pypa/hatch)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![pytest](https://img.shields.io/badge/logo-pytest-blue?logo=pytest&labelColor=5c5c5c&label=%20)](https://github.com/pytest-dev/pytest)

You can find the instructions on how to install the package for development in [the documentation](https://altair-viz.github.io/getting_started/installation.html).

To run the tests and linters, use

```
hatch run test
```

For information on how to contribute your developments back to the Vega-Altair repository, see
[`CONTRIBUTING.md`](https://github.com/vega/altair/blob/main/CONTRIBUTING.md)

## Citing Vega-Altair

[![JOSS Paper](https://joss.theoj.org/papers/10.21105/joss.01057/status.svg)](https://joss.theoj.org/papers/10.21105/joss.01057)

If you use Vega-Altair in academic work, please consider citing https://joss.theoj.org/papers/10.21105/joss.01057 as

```bib
@article{VanderPlas2018,
    doi = {10.21105/joss.01057},
    url = {https://doi.org/10.21105/joss.01057},
    year = {2018},
    publisher = {The Open Journal},
    volume = {3},
    number = {32},
    pages = {1057},
    author = {Jacob VanderPlas and Brian Granger and Jeffrey Heer and Dominik Moritz and Kanit Wongsuphasawat and Arvind Satyanarayan and Eitan Lees and Ilia Timofeev and Ben Welsh and Scott Sievert},
    title = {Altair: Interactive Statistical Visualizations for Python},
    journal = {Journal of Open Source Software}
}
```
Please additionally consider citing the [Vega-Lite](https://vega.github.io/vega-lite/) project, which Vega-Altair is based on: https://dl.acm.org/doi/10.1109/TVCG.2016.2599030

```bib
@article{Satyanarayan2017,
    author={Satyanarayan, Arvind and Moritz, Dominik and Wongsuphasawat, Kanit and Heer, Jeffrey},
    title={Vega-Lite: A Grammar of Interactive Graphics},
    journal={IEEE transactions on visualization and computer graphics},
    year={2017},
    volume={23},
    number={1},
    pages={341-350},
    publisher={IEEE}
}
```


# Coreax

[![Unit Tests](https://github.com/gchq/coreax/actions/workflows/unittests.yml/badge.svg)](https://github.com/gchq/coreax/actions/workflows/unittests.yml)
[![Coverage](https://img.shields.io/endpoint?url=https%3A%2F%2Fgist.githubusercontent.com%2Ftp832944%2F51dd332be75961a7dc903c67718028e1%2Fraw%2Fcoreax.json)](https://github.com/gchq/coreax/actions/workflows/coverage.yml)
[![Pre-commit Checks](https://github.com/gchq/coreax/actions/workflows/pre_commit_checks.yml/badge.svg)](https://github.com/gchq/coreax/actions/workflows/pre_commit_checks.yml)
[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)
[![Python version](https://img.shields.io/pypi/pyversions/coreax.svg)](https://pypi.org/project/coreax)
[![PyPI](https://img.shields.io/pypi/v/coreax)](https://pypi.org/project/coreax)
![Beta](https://img.shields.io/badge/pre--release-beta-red)

_© Crown Copyright GCHQ_

Coreax is a library for **coreset algorithms**, written in JAX for fast execution and GPU support.

## About Coresets

For $n$ points in $d$ dimensions, a coreset algorithm takes an $n \times d$ data set and
reduces it to $m \ll n$ points whilst attempting to preserve the statistical properties
of the full data set. The algorithm maintains the dimension of the original data set.
Thus the $m$ points, referred to as the **coreset**, are also $d$-dimensional.

The $m$ points need not be in the original data set. We refer to the special case where
all selected points are in the original data set as a **coresubset**.

Some algorithms return the $m$ points with weights, so that importance can be
attributed to each point in the coreset. The weights, $w_i$ for $i=1,...,m$, are often
chosen from the simplex. In this case, they are non-negative and sum to 1:
$w_i >0$ $\forall i$ and $\sum_{i} w_i =1$.

Please see [the documentation](https://coreax.readthedocs.io/en/latest/quickstart.html) for some in-depth examples.

##  Example applications

### Choosing pixels from an image

In the example below, we reduce the original 180x215
pixel image (38,700 pixels in total) to a coreset approximately 20% of this size.
(Left) original image.
(Centre) 8,000 coreset points chosen using Stein kernel herding, with point size a
function of weight.
(Right) 8,000 points chosen randomly.
Run `examples/david_map_reduce_weighted.py` to  replicate.

![](https://raw.githubusercontent.com/gchq/coreax/main/examples/data/david_coreset.png)

### Video event detection

Here we identify representative frames such that most of the
useful information in a video is preserved.
Run `examples/pounce.py` to replicate.

|                                 Original                                 |                                     Coreset                                      |
|:------------------------------------------------------------------------:|:--------------------------------------------------------------------------------:|
| ![](https://raw.githubusercontent.com/gchq/coreax/main/examples/pounce/pounce.gif) | ![](https://raw.githubusercontent.com/gchq/coreax/main/examples/pounce/pounce_coreset.gif) |

# Setup
Before installing coreax, make sure JAX is installed. Be sure to install the preferred
version of JAX for your system.

Install [JAX](https://jax.readthedocs.io/en/latest/installation.html) noting that there
are (currently) different setup paths for CPU and GPU use:
```shell
$ python3 -m pip install jax
```

Install Coreax:
```shell
$ python3 -m pip install coreax
```

Optionally, install additional dependencies required to run the examples:
```shell
$ python3 -m pip install coreax[test]
```

Should the installation fail, try again using stable pinned package versions. Note that
these versions may be rather outdated, although we endeavour to avoid versions with
known vulnerabilities. To install Coreax:
```shell
$ python3 -m pip install --no-dependencies -r requirements.txt
```

To run the examples, use `requirements-test.txt` instead.

# Release cycle
We anticipate two release types: feature releases and security releases. Security
releases will be issued as needed in accordance with the
[security policy](https://github.com/gchq/coreax/security/policy). Feature releases will
be issued as appropriate, dependent on the feature pipeline and development priorities.

# Coming soon
Some features coming soon include:
* Coordinate bootstrapping for high-dimensional data.
* Other coreset-style algorithms, including recombination, as means
to reducing a large dataset whilst maintaining properties of the underlying distribution.


# This repository is no longer maintained.

Issue reports and pull requests will not be attended.

Visit [identity-elk-integration](https://github.com/wso2-extensions/identity-elk-integration) for latest analytics solution with ELK.

# WSO2 Identity Server Analytics

WSO2 Identity Server Analytics can be used to monitor authentication related analytics for residence and federated authentication scenarios of [WSO2 Identity Server](http://wso2.com/products/identity-server/).

WSO2 IS Analytics is powered by [WSO2 SP](https://wso2.com/analytics-and-stream-processing), the SOA middleware component platform. An open source product, WSO2 Carbon is available under the [Apache Software License (v2.0)](http://www.apache.org/licenses/LICENSE-2.0.html).

# Capabilities

- View Authentication analytics for Federated Identity scenarios which includes viewing overall success and failure login attempts and authentication attempts views for multiple dimensions of Service Provider, Identity Providers,Users and also first time login of service Providers.

- View Authentication analytics for Federated scenarios which includes viewing
overall success and failure login attempts and authentication attempts views for multiple dimensions of service provider, roles , user and user-stores

# System Requirements

1. Memory   - 2 GB minimum
2. Disk     - 1 GB minimum for unpacking the distribution
3. Java(TM) - Oracle Java SE Development Kit 1.8 or later

# Contact us

You can talk to us via the following mailing lists:

WSO2 Developers List : dev@wso2.org
WSO2 Architecture List : architecture@wso2.org


|logo|

|Build Status|

Streamparse lets you run Python code against real-time streams of data via
Apache Storm.  With streamparse you can create Storm bolts and spouts in
Python without having to write a single line of Java.  It also provides handy
CLI utilities for managing Storm clusters and projects.

The Storm/streamparse combo can be viewed as a more robust alternative to Python
worker-and-queue systems, as might be built atop frameworks like Celery and RQ.
It offers a way to do "real-time map/reduce style computation" against live
streams of data. It can also be a powerful way to scale long-running, highly
parallel Python processes in production.

|Demo|

Documentation
-------------

* `HEAD <http://streamparse.readthedocs.org/en/master/>`_
* `Stable <http://streamparse.readthedocs.org/en/stable/>`_

User Group
----------

Follow the project's progress, get involved, submit ideas and ask for help via
our Google Group, `streamparse@googlegroups.com <https://groups.google.com/forum/#!forum/streamparse>`__.

Contributors
------------

Alphabetical, by last name:

-  Dan Blanchard (`@dsblanch <https://twitter.com/dsblanch>`__)
-  Keith Bourgoin (`@kbourgoin <https://twitter.com/kbourgoin>`__)
-  Arturo Filastò (`@hellais <https://github.com/hellais>`__)
-  Jeffrey Godwyll (`@rey12rey <https://twitter.com/rey12rey>`__)
-  Daniel Hodges (`@hodgesds <https://github.com/hodgesds>`__)
-  Wieland Hoffmann (`@mineo <https://github.com/mineo>`__)
-  Tim Hopper (`@tdhopper <https://twitter.com/tdhopper>`__)
-  Omer Katz (`@thedrow <https://github.com/thedrow>`__)
-  Aiyesha Ma (`@Aiyesha <https://github.com/Aiyesha>`__)
-  Andrew Montalenti (`@amontalenti <https://twitter.com/amontalenti>`__)
-  Rohit Sankaran (`@roadhead <https://twitter.com/roadhead>`__)
-  Viktor Shlapakov (`@vshlapakov <https://github.com/vshlapakov>`__)
-  Mike Sukmanowsky (`@msukmanowsky <https://twitter.com/msukmanowsky>`__)
-  Cody Wilbourn (`@codywilbourn <https://github.com/codywilbourn>`__)
-  Curtis Vogt (`@omus <https://github.com/omus>`__)

Changelog
---------

See the `releases <https://github.com/Parsely/streamparse/releases>`__ page on
GitHub.

Roadmap
-------

See the `Roadmap <https://github.com/Parsely/streamparse/wiki/Roadmap>`__.

.. |logo| image:: https://raw.githubusercontent.com/Parsely/streamparse/master/doc/source/images/streamparse-logo.png
.. |Build Status| image:: https://travis-ci.org/Parsely/streamparse.svg?branch=master
   :target: https://travis-ci.org/Parsely/streamparse
.. |Demo| image:: https://raw.githubusercontent.com/Parsely/streamparse/master/doc/source/images/quickstart.gif


    Developer Docs •
    Screenshots •
    Downloads •
    Marketplace •
    Code of Conduct

    "Life is short, eat more biscuits."

Biscuit is a lightweight, awesome code editor for any language. Biscuit supports essential language features such as Smart Code Completions, Syntax highlighting, etc. We wrote it in python, using only tkinter for GUI. It's a great alternative to [insert funny thing here]! Scroll down to see [some of the features](https://github.com/tomlin7/Biscuit/tree/main?tab=readme-ov-file#features).

- Explore all community made extensions & authors at [Biscuit Extensions Marketplace](https://tomlin7.github.io/biscuit-extensions/)
- For developer/user guides or API reference, read the [documentation](https://tomlin7.github.io/biscuit).

## Installing
- Download latest **stable builds** from [**releases page**](https://github.com/tomlin7/Biscuit/releases).
- Nightly builds from [github actions page](https://github.com/tomlin7/Biscuit/actions).

To compile Biscuit for your platform, see [compiling guide](https://github.com/tomlin7/Biscuit/tree/main/scripts).

## Contributing
Your contributions are greatly appreciated! please have a look at our [**developer docs**](https://github.com/tomlin7/Biscuit/blob/main/CONTRIBUTING.md) for an outline of the project and to setup the environment.

- For feature suggestions and bug reports, please check [issue tracker](https://github.com/tomlin7/Biscuit/issues)
- Community made Extensions can be published at: [**Biscuit Extensions Repository**](https://github.com/tomlin7/biscuit-extensions)

## Features
## Smart Auto-completions
When the opened file have a recognized and supported file type, and also have language extensions installed, the autocomplete widget will automatically turn on LSP mode, the completions will be handled by registered language server afterwards. LSP mode enables smart and context sensitive code suggestions, as demonstrated below.
When there are no language extensions available/installed, autocomplete widget goes to word mode, the completions will not be context sensitive and can be unexpected.

![Autocompletions](https://github.com/tomlin7/biscuit/assets/70792552/885ebf36-ce18-45a3-ae57-9a4b709331f7)

## Hover for Documentation
Identifiers when hovered pops up a floating window on top, containing the documentation (if there is) of that symbol.

![hover](https://github.com/tomlin7/biscuit/assets/70792552/4f0ba532-4ec0-49c3-ad08-19e8a622a416)

## Goto Definitions by `ctrl` + clicking on symbols
User can jump directly into symbol definitions by  `ctrl` + clicking on them, the definition containing script will open up in new/existing tab. If there are multiple definitions, window containing all the definitions will popup right below the symbol to pick from

![gotodef](https://github.com/tomlin7/biscuit/assets/70792552/fb3a012f-1e93-4c00-930c-843a9728b958)

## **Git Support**

![image](https://github.com/tomlin7/Biscuit/assets/70792552/ea231a77-7899-4560-ab97-95828bb96932)

## **Integrated Terminals, Extension center**

![image](https://github.com/tomlin7/Biscuit/assets/70792552/2531ea77-a1e0-4a81-96c9-66ad6b6b0c6d)

## **Split-Pane Markdown Editor**

![image](https://github.com/tomlin7/Biscuit/assets/70792552/2e58ff22-2412-4cb1-b183-673591200308)

## **Command Palette for Quick Access**

![ezgif com-video-to-gif](https://github.com/tomlin7/Biscuit/assets/70792552/e0868336-a15f-4b98-a62e-a822e2211e57)

## **PathView for the Breadcrumbs!**

![pathview](https://imgur.com/CztWtni.jpg)

# License
Biscuit uses the MIT License, please check [LICENSE](https://github.com/tomlin7/Biscuit/blob/main/LICENSE.md).


# InstructLab 🐶 (`ilab`)

![Lint](https://github.com/instructlab/instructlab/actions/workflows/lint.yml/badge.svg?branch=main)
![Tests](https://github.com/instructlab/instructlab/actions/workflows/test.yml/badge.svg?branch=main)
![Build](https://github.com/instructlab/instructlab/actions/workflows/pypi.yaml/badge.svg?branch=main)
![Release](https://img.shields.io/github/v/release/instructlab/instructlab)
![License](https://img.shields.io/github/license/instructlab/instructlab)

## 📖 Contents

- [Welcome to the InstructLab CLI](#welcome-to-the-instructlab-cli)
- [❓ What is `ilab`](#-what-is-ilab)
- [📋 Requirements](#-requirements)
- [✅ Getting started](#-getting-started)
  - [🧰 Installing `ilab`](#-installing-ilab)
    - [Install with no GPU acceleration and PyTorch without CUDA bindings](#install-using-pytorch-without-cuda-bindings-and-no-gpu-acceleration)
    - [Install with AMD ROCm](#install-with-amd-rocm)
    - [Install with Apple Metal on M1/M2/M3 Macs](#install-with-apple-metal-on-m1m2m3-macs)
    - [Install with Nvidia CUDA](#install-with-nvidia-cuda)
  - [🏗 Initialize `ilab`](#-initialize-ilab)
  - [📥 Download the model](#-download-the-model)
  - [🍴 Serving the model](#-serving-the-model)
  - [📣 Chat with the model (Optional)](#-chat-with-the-model-optional)
- [💻 Creating new knowledge or skills and training the model](#-creating-new-knowledge-or-skills-and-training-the-model)
  - [🎁 Contribute knowledge or compositional skills](#-contribute-knowledge-or-compositional-skills)
  - [📜 List and validate your new data](#-list-and-validate-your-new-data)
  - [🚀 Generate a synthetic dataset](#-generate-a-synthetic-dataset)
  - [👩🏫 Training the model](#-training-the-model)
    - [Train the model locally on Linux](#train-the-model-locally-on-linux)
    - [Train the model locally on M-series Macs](#train-the-model-locally-on-an-m-series-mac)
    - [Train the model locally with GPU acceleration](#train-the-model-locally-with-gpu-acceleration)
    - [Train the model in the cloud](#train-the-model-in-the-cloud)
  - [📜 Test the newly trained model](#-test-the-newly-trained-model)
  - [🍴 Serve the newly trained model](#-serve-the-newly-trained-model)
- [📣 Chat with the new model (not optional this time)](#-chat-with-the-new-model-not-optional-this-time)
- [🚀 Upgrade InstructLab to latest version](#-upgrade-instructlab-to-latest-version)
- [🎁 Submit your new knowledge or skills](#-submit-your-new-knowledge-or-skills)
- [📬 Contributing](#-contributing)

## Welcome to the InstructLab CLI

InstructLab 🐶 uses a novel synthetic data-based alignment tuning method for
Large Language Models (LLMs.) The "**lab**" in Instruct**Lab** 🐶 stands for
[**L**arge-Scale **A**lignment for Chat**B**ots](https://arxiv.org/abs/2403.01081) [1].

[1] Shivchander Sudalairaj*, Abhishek Bhandwaldar*, Aldo Pareja*, Kai Xu, David D. Cox, Akash Srivastava*. "LAB: Large-Scale Alignment for ChatBots", arXiv preprint arXiv: 2403.01081, 2024. (* denotes equal contributions)

## ❓ What is `ilab`

`ilab` is a Command-Line Interface (CLI) tool that allows you to perform the following actions:

1. Download a pre-trained Large Language Model (LLM).
1. Chat with the LLM.

To add new knowledge and skills to the pre-trained LLM, add information to the companion [taxonomy](https://github.com/instructlab/taxonomy.git) repository.

After you have added knowledge and skills to the taxonomy, you can perform the following actions:

1. Use `ilab` to generate new synthetic training data based on the changes in your local `taxonomy` repository.
1. Re-train the LLM with the new training data.
1. Chat with the re-trained LLM to see the results.

```mermaid
graph TD;
  download-->chat
  chat[Chat with the LLM]-->add
  add[Add new knowledge\nor skill to taxonomy]-->generate[generate new\nsynthetic training data]
  generate-->train
  train[Re-train]-->|Chat with\nthe re-trained LLM\nto see the results|chat
```

For an overview of the full workflow, see the [workflow diagram](./docs/workflow.png).

> [!IMPORTANT]
> We have optimized InstructLab so that community members with commodity hardware can perform these steps. However, running InstructLab on a laptop will provide a low-fidelity approximation of synthetic data generation
> (using the `ilab generate` command) and model instruction tuning (using the `ilab train` command, which uses QLoRA). To achieve higher quality, use more sophisticated hardware and configure InstructLab to use a
> larger teacher model [such as Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral).

## 📋 Requirements

- **🍎 Apple M1/M2/M3 Mac or 🐧 Linux system** (tested on Fedora). We anticipate support for more operating systems in the future.
- C++ compiler
- Python 3.9+ (<3.12 for PyTorch JIT)
- Approximately 60GB disk space (entire process)

> **NOTE:** PyTorch 2.2.1 does not support `torch.compile` with Python 3.12. On Fedora 39+, install `python3.11-devel` and create the virtual env with `python3.11` if you wish to use PyTorch's JIT compiler.
<!-- -->
> **NOTE:** When installing the `ilab` CLI on macOS, you may have to run the `xcode select --install` command, installing the required packages previously listed.

## ✅ Getting started

### 🧰 Installing `ilab`

1. When installing on Fedora Linux, install C++, Python 3.9+, and other necessary tools by running the following command:

   ```shell
   sudo dnf install g++ gcc make pip python3 python3-devel python3-GitPython
   ```

   Optional: If `g++` is not found, try `gcc-c++` by running the following command:

   ```shell
   sudo dnf install gcc-c++ gcc make pip python3 python3-devel python3-GitPython
   ```

   If you are running on macOS, this installation is not necessary and you can begin your process with the following step.

2. Create a new directory called `instructlab` to store the files the `ilab` CLI needs when running and `cd` into the directory by running the following command:

   ```shell
   mkdir instructlab
   cd instructlab
   ```

   > **NOTE:** The following steps in this document use [Python venv](https://docs.python.org/3/library/venv.html) for virtual environments. However, if you use another tool such as [pyenv](https://github.com/pyenv/pyenv) or [Conda Miniforge](https://github.com/conda-forge/miniforge) for managing Python environments on your machine continue to use that tool instead. Otherwise, you may have issues with packages that are installed but not found in `venv`.

3. There are a few ways you can locally install the `ilab` CLI. Select your preferred installation method from the following instructions. You can then install `ilab` and activate your `venv` environment.

   > **NOTE**: ⏳ `pip install` may take some time, depending on your internet connection. In case installation fails with error ``unsupported instruction `vpdpbusd'``, append `-C cmake.args="-DLLAMA_NATIVE=off"` to `pip install` command.

   See [the GPU acceleration documentation](./docs/gpu-acceleration.md) for how to
   to enable hardware acceleration for interaction and training on AMD ROCm,
   Apple Metal Performance Shaders (MPS), and Nvidia CUDA.

   #### Install using PyTorch without CUDA bindings and no GPU acceleration

      ```shell
      python3 -m venv --upgrade-deps venv
      source venv/bin/activate
      pip cache remove llama_cpp_python
      pip install instructlab --extra-index-url=https://download.pytorch.org/whl/cpu
      ```

      > **NOTE**: *Additional Build Argument for Intel Macs*
      >
      > If you have an Mac with an Intel CPU, you must add a prefix of
      > `CMAKE_ARGS="-DLLAMA_METAL=off"` to the `pip install` command to ensure
      > that the build is done without Apple M-series GPU support.
      >
      > `(venv) $ CMAKE_ARGS="-DLLAMA_METAL=off" pip install ...`

   #### Install with AMD ROCm

      ```shell
      python3 -m venv --upgrade-deps venv
      source venv/bin/activate
      pip cache remove llama_cpp_python
      pip install instructlab \
         --extra-index-url https://download.pytorch.org/whl/rocm6.0 \
         -C cmake.args="-DLLAMA_HIPBLAS=on" \
         -C cmake.args="-DAMDGPU_TARGETS=all" \
         -C cmake.args="-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang" \
         -C cmake.args="-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++" \
         -C cmake.args="-DCMAKE_PREFIX_PATH=/opt/rocm"
      ```

      On Fedora 40+, use `-DCMAKE_C_COMPILER=clang-17` and `-DCMAKE_CXX_COMPILER=clang++-17`.

   #### Install with Apple Metal on M1/M2/M3 Macs

      > **NOTE**: Make sure your system Python build is `Mach-O 64-bit executable arm64` by using `file -b $(command -v python)`.

      ```shell
      python3 -m venv --upgrade-deps venv
      source venv/bin/activate
      pip cache remove llama_cpp_python
      pip install instructlab
      ```

   #### Install with Nvidia CUDA

      ```shell
      python3 -m venv --upgrade-deps venv
      source venv/bin/activate
      pip cache remove llama_cpp_python
      pip install instructlab -C cmake.args="-DLLAMA_CUDA=on"
   ```

4. From your `venv` environment, verify `ilab` is installed correctly, by running the `ilab` command.

   ```shell
   ilab
   ```

   *Example output of the `ilab` command*

   ```shell
   (venv) $ ilab
   Usage: ilab [OPTIONS] COMMAND [ARGS]...

   CLI for interacting with InstructLab.

   If this is your first time running InstructLab, it's best to start with `ilab init` to create the environment.

   Options:
   --config PATH  Path to a configuration file.  [default: config.yaml]
   --version      Show the version and exit.
   --help         Show this message and exit.

   Commands:
   chat      Run a chat using the modified model
   check     (Deprecated) Check that taxonomy is valid
   convert   Converts model to GGUF
   diff      Lists taxonomy files that have changed since <taxonomy-base>...
   download  Download the model(s) to train
   generate  Generates synthetic data to enhance your example data
   init      Initializes environment for InstructLab
   list      (Deprecated) Lists taxonomy files that have changed since <taxonomy-base>.
   serve     Start a local server
   test      Runs basic test to ensure model correctness
   train     Takes synthetic data generated locally with `ilab generate`...
   ```

   > **IMPORTANT:** every `ilab` command needs to be run from within your Python virtual environment. To enter the Python environment, run the following command:

   ```shell
   source venv/bin/activate
   ```

5. Optional: You can enable tab completion for the `ilab` command.

   #### Bash (version 4.4 or newer)

   Enable tab completion in `bash` with the following command:

   ```sh
   eval "$(_ILAB_COMPLETE=bash_source ilab)"
   ```

   To have this enabled automatically every time you open a new shell,
   you can save the completion script and source it from `~/.bashrc`:

   ```sh
   _ILAB_COMPLETE=bash_source ilab > ~/.ilab-complete.bash
   echo ". ~/.ilab-complete.bash" >> ~/.bashrc
   ```

   #### Zsh

   Enable tab completion in `zsh` with the following command:

   ```sh
   eval "$(_ILAB_COMPLETE=zsh_source ilab)"
   ```

   To have this enabled automatically every time you open a new shell,
   you can save the completion script and source it from `~/.zshrc`:

   ```sh
   _ILAB_COMPLETE=zsh_source ilab > ~/.ilab-complete.zsh
   echo ". ~/.ilab-complete.zsh" >> ~/.zshrc
   ```

   #### Fish

   Enable tab completion in `fish` with the following command:

   ```sh
   _ILAB_COMPLETE=fish_source ilab | source
   ```

   To have this enabled automatically every time you open a new shell,
   you can save the completion script and source it from `~/.bashrc`:

   ```sh
   _ILAB_COMPLETE=fish_source ilab > ~/.config/fish/completions/ilab.fish
   ```

### 🏗 Initialize `ilab`

1. Initialize `ilab` by running the following command:

   ```shell
   ilab init
   ```

   *Example output*

   ```shell
   Welcome to InstructLab CLI. This guide will help you set up your environment.
   Please provide the following values to initiate the environment [press Enter for defaults]:
   Path to taxonomy repo [taxonomy]: <ENTER>
   ```

2. When prompted by the interface, press **Enter** to add a new default `config.yaml` file.

3. When prompted, clone the `https://github.com/instructlab/taxonomy.git` repository into the current directory by typing **y**.

   **Optional**: If you want to point to an existing local clone of the `taxonomy` repository, you can pass the path interactively or alternatively with the `--taxonomy-path` flag.

   *Example output after initializing `ilab`*

   ```shell
   (venv) $ ilab init
   Welcome to InstructLab CLI. This guide will help you set up your environment.
   Please provide the following values to initiate the environment [press Enter for defaults]:
   Path to taxonomy repo [taxonomy]: <ENTER>
   `taxonomy` seems to not exists or is empty. Should I clone https://github.com/instructlab/taxonomy.git for you? [y/N]: y
   Cloning https://github.com/instructlab/taxonomy.git...
   Generating `config.yaml` in the current directory...
   Initialization completed successfully, you're ready to start using `ilab`. Enjoy!
   ```

   `ilab` will use the default configuration file unless otherwise specified. You can override this behavior with the `--config` parameter for any `ilab` command.

### 📥 Download the model

- Run the `ilab download` command.

  ```shell
  ilab download
  ```

  `ilab download` downloads a compact pre-trained version of the [model](https://huggingface.co/instructlab/) (~4.4G) from HuggingFace and store it in a `models` directory:

  ```shell
  (venv) $ ilab download
  Downloading model from instructlab/merlinite-7b-lab-GGUF@main to models...
  (venv) $ ls models
  merlinite-7b-lab-Q4_K_M.gguf
  ```

  > **NOTE** ⏳ This command can take few minutes or immediately depending on your internet connection or model is cached. If you have issues connecting to Hugging Face, refer to the [Hugging Face discussion forum](https://discuss.huggingface.co/) for more details.

  #### Downloading a specific model from a Hugging Face repository

- Specify repository, model, and a Hugging Face token if necessary. More information about Hugging Face tokens can be found [here](https://huggingface.co/docs/hub/en/security-tokens)

  ```shell
  HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab download --repository=TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF --filename=mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
  ```

  #### Downloading an entire Hugging Face repository

- Specify repository, and a Hugging Face token if necessary. For example:

  ```shell
  HF_TOKEN=<YOUR HUGGINGFACE TOKEN GOES HERE> ilab download --repository=mistralai/Mixtral-8x7B-v0.1
  ```

### 🍴 Serving the model

- Serve the model by running the following command:

   ```shell
   ilab serve
   ```

- Serve a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):

   ```shell
   ilab serve --model-path models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
   ```

- Once the model is served and ready, you'll see the following output:

   ```shell
   (venv) $ ilab serve
   INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/ggml-merlinite-7b-lab-Q4_K_M.gguf' with -1 gpu-layers and 4096 max context size.
   Starting server process
   After application startup complete see http://127.0.0.1:8000/docs for API.
   Press CTRL+C to shut down the server.
   ```

   > **NOTE:** If multiple `ilab` clients try to connect to the same InstructLab server at the same time, the 1st will connect to the server while the others will start their own temporary server. This will require additional resources on the host machine.

### 📣 Chat with the model (Optional)

Because you're serving the model in one terminal window, you will have to create a new window and re-activate your Python virtual environment to run `ilab chat` command:

```shell
source venv/bin/activate
ilab chat
```

Chat with a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1):

```shell
source venv/bin/activate
ilab chat --model models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
```

Before you start adding new skills and knowledge to your model, you can check its baseline performance by asking it a question such as `what is the capital of Canada?`.

> **NOTE:** the model needs to be trained with the generated synthetic data to use the new skills or knowledge

```shell
(venv) $ ilab chat
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GGML-MERLINITE-7B-lab-Q4_K_M (type /h for help)                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>b> what is the capital of Canada                                                                                [S][default]
╭────────────────────────────────────────────────────────────────────────────────────────────────────── ggml-merlinite-7b-lab-Q4_K_M ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ The capital city of Canada is Ottawa. It is located in the province of Ontario, on the southern banks of the Ottawa River in the eastern portion of southern Ontario. The city serves as the political center for Canada, as it is home to │
│ Parliament Hill, which houses the House of Commons, Senate, Supreme Court, and Cabinet of Canada. Ottawa has a rich history and cultural significance, making it an essential part of Canada's identity.                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 12.008 seconds ─╯
>>>                                                                                [S][default]
```

## 💻 Creating new knowledge or skills and training the model

### 🎁 Contribute knowledge or compositional skills

1. Contribute new knowledge or compositional skills to your local [taxonomy](https://github.com/instructlab/taxonomy.git) repository.

Detailed contribution instructions can be found in the [taxonomy repository](https://github.com/instructlab/taxonomy/blob/main/README.md).

> [!IMPORTANT]
> There is a limit to how much content can exist in the question/answer pairs for the model to process. Due to this, only add a maximum of around 2300 words to your question and answer seed example pairs in the `qna.yaml` file.

### 📜 List and validate your new data

1. List your new data by running the following command:

   ```shell
   ilab diff
   ```

2. To ensure `ilab` is registering your new knowledge or skills, you can run the `ilab diff` command. The following is the expected result after adding the new compositional skill `foo-lang`:

   ```shell
   (venv) $ ilab diff
   compositional_skills/writing/freeform/foo-lang/foo-lang.yaml
   Taxonomy in /taxonomy/ is valid :)
   ```

### 🚀 Generate a synthetic dataset

Before following these instructions, ensure the existing model you are adding skills or knowledge to is still running.

1. To generate a synthetic dataset based on your newly added knowledge or skill set in [taxonomy](https://github.com/instructlab/taxonomy.git) repository, run the following command:

   ```shell
   ilab generate
   ```

   Use a non-default model (e.g. Mixtral-8x7B-Instruct-v0.1) to generate data, run the following command:

   ```shell
   ilab generate --model models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
   ```

   > **NOTE:** ⏳ This can take from 15 minutes to 1+ hours to complete, depending on your computing resources.

   *Example output of `ilab generate`*

   ```shell
   (venv) $ ilab generate
   INFO 2024-02-29 19:09:48,804 lab.py:250 Generating model 'ggml-merlinite-7b-lab-Q4_K_M' using 10 CPUs,
   taxonomy: '/home/username/instructlab/taxonomy' and seed 'seed_tasks.json'

   0%|##########| 0/100 Cannot find prompt.txt. Using default prompt.
   98%|##########| 98/100 INFO 2024-02-29 20:49:27,582 generate_data.py:428 Generation took 5978.78s
   ```

   The synthetic data set will be three files in the newly created `generated` directory named `generated*.json`, `test*.jsonl`, and `train*.jsonl`.

> [!NOTE]
> If you want to pickup from where a failed or canceled `ilab generate` left off, you can copy the
> `generated*.json` file into a file named `regen.json`. `regen.json` will be picked up at the start of `lab
> generate` when available. You should remove it when the process is completed.

2. Verify the files have been created by running the `ls generated` command.

   ```shell
   (venv) $ ls generated/
   'generated_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.json'   'train_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.jsonl'
   'test_ggml-merlinite-7b-lab-0226-Q4_K_M_2024-02-29T19 09 48.jsonl'
   ```

   **Optional**: It is also possible to run the generate step against a different model via an
   OpenAI-compatible API. For example, the one spawned by `ilab serve` or any remote or locally hosted LLM (e.g. via [`ollama`](https://ollama.com/), [`LM Studio`](https://lmstudio.ai), etc.). Run the following command:

   ```shell
   ilab generate --endpoint-url http://localhost:8000/v1
   ```

### 👩🏫 Training the model

There are many options for training the model with your synthetic data-enhanced dataset.

> **Note:** **Every** `ilab` command needs to run from within your Python virtual environment.

#### Train the model locally on Linux

```shell
ilab train
```

> **NOTE:** ⏳ This step can potentially take **several hours** to complete depending on your computing resources. Please stop `ilab chat` and `ilab serve` first to free resources.

`ilab train` outputs a brand-new model that can be served in the `models` directory called `ggml-model-f16.gguf`.

```shell
 (venv) $ ls models
 ggml-merlinite-7b-lab-Q4_K_M.gguf  ggml-model-f16.gguf
```

#### Train the model locally on an M-series Mac

To train the model locally on your M-Series Mac is as easy as running:

```shell
ilab train
```

> **Note:** ⏳ This process will take a little while to complete (time can vary based on hardware
and output of `ilab generate` but on the order of 5 to 15 minutes)

`ilab train` outputs a brand-new model that is saved in the `<model_name>-mlx-q` directory called `adapters.npz` (in `Numpy` compressed array format). For example:

```shell
(venv) $ ls instructlab-merlinite-7b-lab-mlx-q
adapters-010.npz        adapters-050.npz        adapters-090.npz        config.json             tokenizer.model
adapters-020.npz        adapters-060.npz        adapters-100.npz        model.safetensors       tokenizer_config.json
adapters-030.npz        adapters-070.npz        adapters.npz            special_tokens_map.json
adapters-040.npz        adapters-080.npz        added_tokens.json       tokenizer.jso
```

#### Train the model locally with GPU acceleration

Training has experimental support for GPU acceleration with Nvidia CUDA or AMD ROCm. Please see [the GPU acceleration documentation](./docs/gpu-acceleration.md) for more details. At present, hardware acceleration requires a data center GPU or high-end consumer GPU with at least 18 GB free memory.

```shell
ilab train --device=cuda
```

#### Train the model in the cloud

Follow the instructions in [Training](./notebooks/README.md).

⏳ Approximate amount of time taken on each platform:

- *Google Colab*: **5-10 minutes** with a T4 GPU
- *Kaggle*: **~30 minutes** with a P100 GPU.

After that's done, you can play with your model directly in the Google Colab or Kaggle notebook. Model trained on the cloud will be saved on the cloud.
The model can also be downloaded and served locally.

### 📜 Test the newly trained model

- Run the following command to test the model:

   ```shell
   ilab test
   ```

   > **NOTE:** 🍎 This step is only implemented for macOS with M-series chips (for now)

   The output from the command will consist of a series of outputs from the model before and after training.

### 🍴 Serve the newly trained model

1. Stop the server you have running by entering `ctrl+c` keys in the terminal running the server.

   > **IMPORTANT**:
   - 🍎 This step is only implemented for macOS with M-series chips (for now).

   - Before serving the newly trained model you must convert it to work with
   the `ilab` cli. The `ilab convert` command converts the new model into quantized [GGUF](https://medium.com/@sandyeep70/ggml-to-gguf-a-leap-in-language-model-file-formats-cd5d3a6058f9) format which is required by the server to host the model in the `ilab serve` command.

2. Convert the newly trained model by running the following command:

   ```shell
   ilab convert
   ```

3. Serve the newly trained model locally via `ilab serve` command with the `--model-path`
argument to specify your new model:

   ```shell
   ilab serve --model-path <new model path>
   ```

   Which model should you select to serve? After running the `ilab convert` command, some files and a directory are generated. The model you will want to serve ends with an extension of `.gguf`
   and exists in a directory with the suffix `trained`. For example:
   `instructlab-merlinite-7b-lab-trained/instructlab-merlinite-7b-lab-Q4_K_M.gguf`.

## 📣 Chat with the new model (not optional this time)

- Try the fine-tuned model out live using the chat interface, and see if the results are better than the untrained version of the model with chat by running the following command:

   ```shell
   ilab chat -m <New model name>
   ```

   If you are interested in optimizing the quality of the model's responses, please see [`TROUBLESHOOTING.md`](./TROUBLESHOOTING.md#model-fine-tuning-and-response-optimization)

## 🚀 Upgrade InstructLab to latest version

- To upgrade InstructLab to the latest version, use the following command:

   ```shell
   pip install instructlab --upgrade
   ```

## 🎁 Submit your new knowledge or skills

Of course, the final step is, if you've improved the model, to open a pull-request in the [taxonomy repository](https://github.com/instructlab/taxonomy) that includes the files (e.g. `qna.yaml`) with your improved data.

## 📬 Contributing

Check out our [contributing](CONTRIBUTING/CONTRIBUTING.md) guide to learn how to contribute.


<picture>
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/neurad_logo_with_text_dark.png" />

</picture>

Neural Rendering for Autonomous Driving

CVPR 2024 highlight

[Quickstart](#quickstart) ·
[Learn more](#learn-more) ·
[Planned Features](#planned-featurestodos) ·
[Project page](https://research.zenseact.com/publications/neurad/)

# About

This is the official code release of the CVPR 2024 [paper](https://arxiv.org/abs/2311.15260) _NeuRAD: Neural Rendering for Autonomous Driving_, building on top of [Nerfstudio](https://github.com/nerfstudio-project/nerfstudio). Besides releasing the code for our NeuRAD model, we hope that this can lay the ground-work for research on applying neural rendering methods in autonomous driving.

In line with Nerfstudio's mission, this is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.

Do you have feature requests or want to add **your** new AD-NeRF model? Or maybe provide structures for a new dataset? **We welcome contributions!**

<picture style="padding-left: 10px; padding-right: 10px;">
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/ZEN_Vertical_logo_white.svg" />

</picture>

<picture style="padding-left: 10px; padding-right: 10px; padding-bottom: 10px;">
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/EN_Avancez_CH_white.png" />

</picture>

<picture style="padding-left: 10px; padding-right: 10px;">
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/LundUniversity_C2line_NEG.png" />

</picture>

<picture style="padding-left: 10px; padding-right: 10px;">
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/LiU_secondary_1_white-PNG.png" />

</picture>

<picture>
    <source media="(prefers-color-scheme: dark)" srcset="docs/_static/imgs/WASP-logotype-white.png" />

</picture>

# Quickstart

The quickstart will help you get started with the NeuRAD model on a PandaSet sequence.
For more complex changes (e.g., running with your own data/setting up a new NeRF graph), please refer to our [references](#learn-more).

## 1. Installation: Setup the environment

### Prerequisites

Our installation steps largely follow Nerfstudio, with some added dataset-specific dependencies. You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDA [here](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html).

### Create environment

NeuRAD requires `python >= 3.8`. We recommend using conda to manage dependencies. Make sure to install [Conda](https://docs.conda.io/miniconda.html) before proceeding.

```bash
conda create --name neurad -y python=3.8
conda activate neurad
pip install --upgrade pip
```

### Dependencies

Install PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn).
`cuda-toolkit` is required for building `tiny-cuda-nn`.

For CUDA 11.8:

```bash
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit

# Some need to upgrade dill prior to tiny-cuda-nn install
pip install dill --upgrade

pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
```

We refer to [Nerfstudio](https://github.com/nerfstudio-project/nerfstudio/blob/v1.0.3/docs/quickstart/installation.md) for more installation support.

### Installing NeuRAD
```bash
git clone https://github.com/georghess/neurad-studio.git
cd neurad-studio
pip install --upgrade pip setuptools
pip install -e .
```

**OR** if you want to skip all installation steps and directly start using NeuRAD, use the provided docker image or apptainer recipe:

[Dockerfile](Dockerfile) or [Apptainer recipe](apptainer_recipe).

## 2. Training your first model!

The following will train a _NeuRAD_ model, our recommended model for real world AD scenes.

### Data preparation

Begin by downloading [PandaSet](https://huggingface.co/datasets/georghess/pandaset) and unzip it under ```data/pandaset```.
The dataset is no longer hosted by Scale but can be downloaded from the provided huggingface link.

### Training

Training models is done the same way as in nerfstudio, i.e.,

```bash
# Train model
python nerfstudio/scripts/train.py neurad pandaset-data
```

If everything works, you should see training progress like the following:

Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).

### Troubleshooting

If you run into issues, it could be due to the training taking up too much memory. You can try to adjust the model parameters according to the neurad-tiny [vscode launch config](https://github.com/georghess/neurad-studio/blob/master/.vscode/launch.json#L51).

### Resume from checkpoint / visualize existing run

It is possible to load a pretrained model by running

```bash
pyhton nerfstudio/scripts/train.py neurad pandaset-data --load-dir {outputs/.../nerfstudio_models}
```

## Visualize existing run

Given a pretrained model checkpoint, you can start the viewer by running

```bash
python nerfstudio/scripts/viewer/run_viewer.py --load-config {outputs/.../config.yml}
```

## 3. Exporting Results

Once you have a NeRF model you can render its output. There are multiple different renders, more info available using

```bash
python nerfstudio/scripts/render.py --help
```

## 4. Advanced Options

### Training models other than NeuRAD

Besides NeuRAD, we will provide a reimplementation of [UniSim](https://arxiv.org/abs/2308.01898) as well. Once this is released it can be trained using

```bash
# Train model
python nerfstudio/scripts/train.py unisim pandaset-data
```

Further, as we build on top of nerfstudio, models such as _nerfacto_ or _splatfacto_ are available as well, see nerfstudio for details. However, note that these are made for static scenes.

For a full list of included models run `python nerfstudio/scripts/train.py --help`.

### Modify Configuration

Each model contains many parameters that can be changed, too many to list here. Use the `--help` command to see the full list of configuration options.

```bash
python nerfstudio/scripts/train.py neurad --help
```

### Tensorboard / WandB / Comet / Viewer

There are four different methods to track training progress, using the viewer, [tensorboard](https://www.tensorflow.org/tensorboard), [Weights and Biases](https://wandb.ai/site), and [Comet](https://comet.com/?utm_source=nerf&utm_medium=referral&utm_content=github). You can specify which visualizer to use by appending `--vis {viewer, tensorboard, wandb, comet viewer+wandb, viewer+tensorboard, viewer+comet}` to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps.

# Learn More

And that's it for getting started with the basics of NeuRAD. If you are missing some features, have a look at [Planned Features](#planned-featurestodos) to see if we have plans on implementing this. Otherwise, feel free to open an issue, or even better implement it yourself and open a PR!

If you want to add a dataset, look [here](#adding-datasets). If you want to add a method, have a look [here](#adding-methods).

## Adding Datasets

We have provided dataparsers for multiple autonomous driving dataset, see below for a complete list. However, your favorite AD dataset might still be missing.

To add a dataset, create `nerfstudio/data/dataparsers/mydataset.py` containing one dataparsers config class `MyADDataParserConfig` and one dataparser class `MyADData`. Preferrably, these inherit from `ADDataParserConfig` and `ADDataParser`, as these provide common functionality and streamline the expected format of AD data. For most datasets, it should then be sufficient to overwrite `_get_cameras`, `_get_lidars`, `_read_lidars`, `_get_actor_trajectories`, and `_generate_dataparser_outputs`.

| Data                                                                                | Cameras | Lidars                                                      |
| --------------------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------- |
| 🚗 [nuScenes](https://www.nuscenes.org/)          | 6 cameras            |  32-beam lidar                   |
| 🚗 [ZOD](https://zod.zenseact.com/) ([Annotations](https://github.com/user-attachments/files/15773566/auto_annotations.zip))          | 1 camera            | 128-beam + 2 x 16-beam lidars                   |
| 🚗 [Argoverse 2](https://www.argoverse.org/av2.html)   | 7 ring cameras + 2 stereo cameras            | 2 x 32-beam lidars                   |
| 🚗 [PandaSet](https://pandaset.org/) ([huggingface download](https://huggingface.co/datasets/georghess/pandaset))         | 6 cameras | 64-beam lidar                                  |
| 🚗 [KITTIMOT](https://www.cvlibs.net/datasets/kitti/eval_tracking.php) ([Timestamps](https://www.cvlibs.net/datasets/kitti/raw_data.php)) | 2 stereo cameras | 64-beam lidar

## Adding Methods

Nerfstudio has made it easy to add new methods, see [here](https://docs.nerf.studio/developer_guides/new_methods.html) for details. We have added [our UniSim reimplementation](https://github.com/carlinds/unisim) as a plugin, which can be run as any other method using the `ns-train` command:
```bash
ns-train unisim pandaset-data --data data/pandaset
```
and follow the instructions in the terminal.

See [our UniSim repo](https://github.com/carlinds/unisim) for reference on how to add a new method as a plugin.

# Key features
- Dataparser for multiple autonomous driving datasets including
    - Dataparsing of lidar data (3D+intensity+time)
    - Dataparsing of annotations
- Datamanager for lidar+image data
- Rolling shutter handling for ray generation
- Viewer improvements
    - Lidar rendering
    - Dynamic actor modifications
- NeuRAD - SOTA neural rendering method for dynamic AD scenes

# Planned Features/TODOs

- [ ] 3DGS implementation supporting dynamic objects
- [x] UniSim plug-in
- [x] Release code

# Built On

<picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/nerfstudio-project/nerfstudio/main/docs/_static/imgs/logo-dark.png" />

</picture>

- Collaboration friendly studio for NeRFs

# Citation

You can find our paper on [arXiv](https://arxiv.org/abs/2311.15260).

If you use this code or find our paper useful, please consider citing:

```bibtex
@article{neurad,
  title={NeuRAD: Neural Rendering for Autonomous Driving},
  author={Tonderski, Adam and Lindstr{\"o}m, Carl and Hess, Georg and Ljungbergh, William and Svensson, Lennart and Petersson, Christoffer},
  journal={arXiv preprint arXiv:2311.15260},
  year={2023}
}
```

# Contributors

\+ [nerfstudio contributors](https://github.com/nerfstudio-project/nerfstudio/graphs/contributors)


# NeMo Framework Launcher

The NeMo Framework Launcher is a cloud-native tool for launching end-to-end NeMo Framework training jobs.

See the [NeMo Framework User Guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/index.html) for
the most up-to-date information and how to get started quickly.

The NeMo Framework focuses on foundation model training for generative AI models.
Large language model (LLM) pretraining typically requires a lot of compute and model parallelism to efficiently scale training.
NeMo Framework includes the latest in large-scale training techniques including:

- Model parallelism
  * Tensor
  * Pipeline
  * Sequence
- Distributed Optimizer
- Mixed precision training
  * FP8
  * BF16
- Distributed Checkpointing
- Community Models
  * LLAMA-2

NeMo Framework model training scales to 1000's of GPUs and can be used for training LLMs on trillions of tokens.

The Launcher is designed to be a simple and easy to use tool for launching NeMo FW training jobs
on CSPs or on-prem clusters. The launcher is typically used from a head node and only requires
a minimal python installation.

The Launcher will generate and launch submission scripts for the cluster scheduler and will also organize
and store jobs results. Tested configuration files are included with the launcher but anything
in a configuration file can be easily modified by the user.

The NeMo FW Launcher is tested with the [NeMo FW Container](https://registry.ngc.nvidia.com/orgs/ea-bignlp/teams/ga-participants/containers/nemofw-training) which can be applied for [here](https://developer.nvidia.com/nemo-framework).
Access is automatic.
Users may also easily configure the launcher to use any container image that they want to provide.

The NeMo FW launcher supports:
- Cluster setup and configuration
- Data downloading, curating, and processing
- Model parallel configuration
- Model training
- Model fine-tuning (SFT and PEFT)
- Model evaluation
- Model export and deployment

Some of the models that we support include:
- GPT
  * Pretraining, Fine-tuning, SFT, PEFT
- BERT
- T5/MT5
  * PEFT, MoE (non-expert)

See the [Feature Matrix](https://docs.nvidia.com/nemo-framework/user-guide/latest/featurematrix.html#gpt-models) for more details.

## Installation

The NeMo Framework Launcher should be installed on a head node or a local machine in a virtual python environment.

```bash
git clone https://github.com/NVIDIA/NeMo-Framework-Launcher.git
cd NeMo-Framework-Launcher
pip install -r requirements.txt
```

## Usage

The best way to get started with the NeMo Framework Launcher is go through
the [NeMo Framework Playbooks](https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/index.html)

After everything is configured in the `.yaml` files, the Launcher can be run with:

```bash
python main.py
```

Since the Launcher uses [Hydra](https://hydra.cc/docs/intro/),
any configuration can be overridden directly in the `.yaml` file or via the command line.
See Hydra's [override grammar](https://hydra.cc/docs/advanced/override_grammar/basic/) for more information.

## Contributing

Contributions are welcome!

To contribute to the NeMo Framework Launcher, simply create a pull request with the changes on GitHub.
After the pull request is reviewed by a NeMo FW Developer, approved, and passes the unit and CI tests,
then it will be merged.

## License

The NeMo Framework Launcher is licensed under the [Apache 2.0 License](https://github.com/NVIDIA/NeMo-Megatron-Launcher/blob/master/LICENSE)


#

[![GitHub release (latest by date)](https://img.shields.io/github/v/release/Kometa-Team/Kometa?style=plastic)](https://github.com/Kometa-Team/Kometa/releases)
[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/kometateam/kometa?label=docker&sort=semver&style=plastic)](https://hub.docker.com/r/kometateam/kometa)
[![Docker Pulls](https://img.shields.io/docker/pulls/kometateam/kometa?style=plastic)](https://hub.docker.com/r/kometateam/kometa)
[![Develop GitHub commits since latest stable release (by SemVer)](https://img.shields.io/github/commits-since/Kometa-Team/Kometa/latest/develop?label=Commits%20in%20Develop&style=plastic)](https://github.com/Kometa-Team/Kometa/tree/develop)
[![Nightly GitHub commits since latest stable release (by SemVer)](https://img.shields.io/github/commits-since/Kometa-Team/Kometa/latest/nightly?label=Commits%20in%20Nightly&style=plastic)](https://github.com/Kometa-Team/Kometa/tree/nightly)

[![Discord](https://img.shields.io/discord/822460010649878528?color=%2300bc8c&label=Discord&style=plastic)](https://kometa.wiki/en/latest/discord/)
[![Reddit](https://img.shields.io/badge/%2Fr%2Fkometa-e05d44?style=plastic&logo=Reddit&logoColor=white&labelColor=0e8a6a&color=00bc8c)](https://www.reddit.com/r/kometa/)
[![Wiki](https://img.shields.io/readthedocs/kometa?color=%2300bc8c&style=plastic)](https://kometa.wiki)
[![Translations](https://img.shields.io/weblate/progress/kometa?color=00bc8c&server=https%3A%2F%2Ftranslations.kometa.wiki&style=plastic)](https://translations.kometa.wiki/projects/kometa/#languages)
[![GitHub Sponsors](https://img.shields.io/github/sponsors/meisnate12?color=%238a2be2&style=plastic)](https://github.com/sponsors/meisnate12)
[![Sponsor or Donate](https://img.shields.io/badge/-Sponsor%2FDonate-blueviolet?style=plastic)](https://github.com/sponsors/meisnate12)
[![Feature Requests](https://img.shields.io/badge/Feature%20Requests-blueviolet?style=plastic)](https://features.kometa.wiki/)

Kometa is a powerful tool designed to give you complete control over your media libraries. With Kometa, you can take your customization to the next level, with granular control over metadata, collections, overlays, and much more.

Transform your media library with Kometa and discover its full potential! Connect to third-party services like TMDb, Trakt, and IMDb, among others, to create one-of-a-kind collections, overlays and more. Your media library will stand out and be tailored to your specific needs.

## What Can Kometa Do?

### Overhaul Your Media Libraries

-  Elevate your library with beautifully crafted metadata - customize artwork, titles, summaries, and more to create a stunning library.

### Kometa Defaults

-  Take advantage of pre-made modular Collections & Overlays to reduce the manual effort and get to the good stuff with less effort!

### Third-Party Integrations

-  Harness the power of Trakt, TMDb, IMDb and more to create collections and overlays!
-  Integrate with Sonarr and Radarr to automate your library growth.

### And More!

-  We're constantly working on new features to take your library management experience to the next level.
-  Consider sponsoring the project to allow us to continue building great features for you!

## Demo Video

The below YouTube video has been created by one of our community members to showcase some of the things that Kometa can do for you.

[![Kometa](https://img.youtube.com/vi/nTfCUtKWTYI/0.jpg)](https://www.youtube.com/watch?v=nTfCUtKWTYI "Kometa")

## Example Kometa Libraries

Here are some examples of the things you can achieve using Kometa!

**Example Movie Collections using the [Kometa Defaults](https://kometa.wiki/en/latest/defaults/collections/)** (click to enlarge):

![Movie Collection Preview](https://kometa.wiki/en/latest/images/movie-collection-preview.png)

**Example Show Overlays using the [Kometa Defaults](https://kometa.wiki/en/latest/defaults/overlays)** (click to enlarge):

![Show Library Preview](https://kometa.wiki/en/latest/images/show-library-preview.png)

## Kometa Defaults

Want your library to look like the above images?  With the [Kometa Defaults](https://kometa.wiki/en/latest/defaults/guide/) you can! These powerful and modular files were designed by the Kometa team to make it simple to create a personalized, one-of-a-kind media collection without the hassle of manually defining each one.

Want to see what the community has to offer? Check out the [Kometa Community Configs](https://github.com/Kometa-Team/Community-Configs) repository on GitHub to see user-submitted configuration files, or even add your own to the mix!

With Kometa, you can also manage metadata for all your media types, from movies and shows to music and more. And since your metadata is managed outside your libraries, you'll never have to worry about losing your customizations in the event of a media server database loss, you can simply reapply them! It is also easy to move your customizations between servers if you need to.

## Getting Started

To get started with Kometa, follow these simple steps:

1. Install Kometa on your device. You can find the installation instructions for a variety of platforms [here](https://kometa.wiki/en/latest/kometa/install/overview/).

2. Once you have installed Kometa, create your [Configuration File](https://kometa.wiki/en/latest/config/overview/). This file contains important information such as URLs and credentials needed to connect to services like Plex and TMDb

3. After creating the Configuration File, you can start updating Metadata and building automatic Collections by creating a [Collection File](https://kometa.wiki/en/latest/files/collections/) for each Library you want to work with. If you'd rather use some of our pre-made Collection Files, take a look at the [Kometa Defaults](https://kometa.wiki/en/latest/defaults/guide/)

4. Finally, check out the [Wiki](https://kometa.wiki/), you'll find new and exciting ways to truly unlock the potential of your libraries.

## Step-by-Step Guides

If you're a beginner to the concepts of Python, Git and/or Kometa and find the above steps challenging, don't worry. We've got some step-by-step guides that can help you get started. These guides will take you through the process of installing Kometa, creating your Configuration File and getting some basic Collections up and running.

For those who need full installation walkthroughs, please refer to the following walkthrough guides:

  * [Local Walkthrough](https://kometa.wiki/en/latest/kometa/install/local/) - follow this if you are running the script directly on Windows, OS X, or Linux
  * [Docker Walkthrough](https://kometa.wiki/en/latest/kometa/install/docker/) - this discusses using Docker at the command line

If you are using unRAID, Kubernetes, QNAP, or Synology refer to the following basic guide to Docker container setup for each system:

**this doesn't cover the Kometa setup specifics found in the guides above with regard to creating the config file and collection file, so you may want to go through the [Docker Walkthrough](https://kometa.wiki/en/latest/kometa/install/docker/) first on your computer to gain that understanding.**

  * [unRAID Walkthrough](https://kometa.wiki/en/latest/kometa/install/unraid/)
  * [Kubernetes Walkthrough](https://kometa.wiki/en/latest/kometa/install/kubernetes/)
  * [QNAP Walkthrough](https://kometa.wiki/en/latest/kometa/install/qnap/)
  * [Synology Walkthrough](https://kometa.wiki/en/latest/kometa/install/synology/)

## Example Usage

Kometa puts you in control of your media library by letting you create custom Collections that make discovering and organizing your content a breeze. With powerful search and filtering options, you can build Collections based on popular builders like TMDb, IMDb, Trakt, and many more.

Imagine having Collections like these at your fingertips:

  * Trending and Popular (based on TMDb, IMDb, Trakt, etc.)
  * Streaming Services (like Netflix, Disney+, and more)
  * Networks
  * Studios
  * Genres
  * Actors
  * Decades

Kometa gives you endless possibilities to curate and organize your media library any way you want. Create custom Collections and Overlays that fit your unique preferences and make discovering your content effortless.

But if you don't want to spend time manually creating Collections and Overlays, we've got you covered. Check out the [Kometa Defaults](https://kometa.wiki/en/latest/defaults/guide/) - a handcrafted selection of tried-and-tested Collections and Overlays made by the Kometa team.

## Develop & Nightly Branches

The Develop and Nightly branches are "beta" versions of Kometa that are updated more frequently than the stable version (Master branch). These branches are where bug fixes, new features, and other changes are added before being released to the Master branch.

However, these branches (especially Nightly) are recommended for more technical users who don't mind updating frequently to get the latest changes. Keep in mind that these beta branches may have bugs or other issues that could cause problems with Kometa or your media server. So, if you're not comfortable with technical issues, it's best to stick with the Master branch.

  Develop Branch (click to expand)

[![Develop GitHub commits since latest stable release (by SemVer)](https://img.shields.io/github/commits-since/Kometa-Team/Kometa/latest/develop?label=Commits%20in%20Develop&style=plastic)](https://github.com/Kometa-Team/Kometa/tree/develop)

The [develop](https://github.com/Kometa-Team/Kometa/tree/develop) branch has the most updated **documented** fixes and enhancements to Kometa.  This version is tested and documented to some degree, but it is still an active Develop branch, so there may be rough edges.

Switching to `develop`:
````{tab} Running in Docker
Add ":develop" to the image name in your run command or configuration:
```
kometateam/kometa:develop
```
````
````{tab} Running on the Host
In the directory where you cloned Kometa:
```bash
git checkout develop
```
To switch back:
```bash
git checkout master
```
````

If switching to the develop branch, it is recommended to also use the [develop branch of the wiki](https://kometa.wiki/en/develop/), which documents any changes made from the Master branch.

  Nightly Branch  (click to expand)

[![Nightly GitHub commits since latest stable release (by SemVer)](https://img.shields.io/github/commits-since/Kometa-Team/Kometa/latest/nightly?label=Commits%20in%20Nightly&style=plastic)](https://github.com/Kometa-Team/Kometa/tree/nightly)

**This branch will have squashed commits which can cause `git pull`/`git fetch` to error you can use `git reset origin/nightly --hard` to fix the branch.**

There is also a [nightly](https://github.com/Kometa-Team/Kometa/tree/nightly) branch which will have the absolute latest version of the script, but it could easily break, there is no guarantee that it even works, and any new features will not be documented until they have progressed enough to reach the develop branch.

Switching to `nightly`:

````{tab} Running in Docker
Add ":nightly" to the image name in your run command or configuration:
```
kometateam/kometa:nightly
```
````
````{tab} Running on the Host
In the directory where you cloned Kometa:
```bash
git checkout nightly
```
To switch back:
```bash
git checkout master
```
````

As this branch is subject to extreme change, there is no promise of the feature being documented in the [nightly](https://kometa.wiki/en/nightly/) branch of the wiki and all discussions relating to changes made in the nightly branch will be held within the [Kometa Discord Server](https://kometa.wiki/en/latest/discord/).

## Discord Support Server

If you're looking for support for any questions or issues you might have, or if you just want to be a part of our growing community, Join the [Kometa Discord Server](https://kometa.wiki/en/latest/discord/).

## Feature Requests

At Kometa, we value our community's input and actively seek feedback to drive the evolution of our product. We want to hear your ideas on how to enhance Kometa, and we encourage you to visit our [Feature Request](https://features.kometa.wiki/features) page to share your thoughts or vote on what features you would like to see added next. Your voice matters and helps shape the future of Kometa, so please don't hesitate to join in the conversation and be a part of our community-driven development process.

## Errors and Configuration Questions

If you're having trouble, we recommend first joining the [Kometa Discord Server](https://kometa.wiki/en/latest/discord/) and seeking support there. If that isn't possible for you, here's what you can do:

* If you get an error, update to the latest version and check if the issue persists. If it does, report the bug by filling out the [Bug Report](https://github.com/Kometa-Team/Kometa/issues/new?assignees=meisnate12&labels=status%3Anot-yet-viewed%2C+bug&template=bug_report.md&title=Bug%3A+) template.
* If you spot a mistake or have an idea to improve the [Kometa Wiki](https://kometa.wiki/), submit a request using the [Wiki Request](https://github.com/Kometa-Team/Kometa/issues/new?assignees=meisnate12&labels=status%3Anot-yet-viewed%2C+documentation&template=3.docs_request.yml&title=%5BDocs%5D%3A+) template.
* If you have a question about metadata configuration, start a discussion on the [Discussions](https://github.com/Kometa-Team/Kometa/discussions). Remember, the community helps shape the future of Kometa, so your input is valuable!

For support on any of the above, visit the [Discord server](https://kometa.wiki/en/latest/discord/).

## Contributing
* Pull Requests are greatly encouraged, please submit all Pull Requests to the nightly branch.


[![Contributors][contributors-shield]][contributors-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![Apache License][license-shield]][license-url]

# Pace

Pace is an implementation of the FV3GFS / SHiELD atmospheric model developed by NOAA/GFDL using the [NDSL](https://github.com/NOAA-GFDL/NDSL) middleware in Python, itself based on [GT4Py](https://github.com/GridTools/gt4py) and [DaCe](https://github.com/spcl/dace). The model can be run on a laptop using Python-based backend or on thousands of heterogeneous compute nodes of a large supercomputer.

🚧 **WARNING** This repo is under active development - supported features and procedures can change rapidly and without notice. 🚧

The repository model code is split between [pyFV3](https://github.com/NOAA-GFDL/pyFV3) for the dynamical core and [pySHiELD](https://github.com/NOAA-GFDL/pySHiELD) for the physics parametrization. A full depencies looks like the following:

```mermaid
flowchart TD
GT4Py.cartesian --> |Stencil DSL|NDSL
DaCe  --> |Full program opt|NDSL
NDSL --> pyFV3
NDSL --> pySHiELD
pyFV3 --> |Dynamics|Pace
pySHiELD --> |Physics|Pace

```

## Quickstart - bare metal

### Build

Pace requires:

- GCC > 9.2
- MPI
- Python 3.8.

For GPU backends CUDA and/or ROCm is required depending on the targeted hardware.

For GT stencils backends, you will also need the headers of the boost libraries in your `$PATH`. This could be down like this.

```shell
cd BOOST/ROOT
wget https://boostorg.jfrog.io/artifactory/main/release/1.79.0/source/boost_1_79_0.tar.gz
tar -xzf boost_1_79_0.tar.gz
mkdir -p boost_1_79_0/include
mv boost_1_79_0/boost boost_1_79_0/include/
export BOOST_ROOT=BOOST/ROOT/boost_1_79_0
```

When cloning Pace you will need to update the repository's submodules as well:

```shell
git clone --recursive https://github.com/NOAA-GFDL/pace.git
```

or if you have already cloned the repository:

```
git submodule update --init --recursive
```

We recommend creating a python `venv` or `conda` environment specifically for Pace.

```shell
python3 -m venv venv_name
source venv_name/bin/activate
```

Inside of your pace `venv` or conda environment pip install the Python requirements, GT4Py, and Pace:

```shell
pip3 install -r requirements_dev.txt -c constraints.txt
```

Shell scripts to install Pace on specific machines such as Gaea can be found in `examples/build_scripts/`.

### Run

With the environment activated, you can run an example baroclinic test case with the following command:

```shell
mpirun -n 6 python3 -m pace.run examples/configs/baroclinic_c12.yaml

# or with oversubscribe if you do not have at least 6 cores
mpirun -n 6 --oversubscribe python3 -m pace.run examples/configs/baroclinic_c12.yaml
```

After the run completes, you will see an output direcotry `output.zarr`. An example to visualize the output is provided in `examples/plot_output.py`. See the [driver example](examples/README.md) section for more details.

### Environment variable configuration

- `PACE_CONSTANTS`: Pace is bundled with various constants.
  - `GFDL` NOAA's FV3 dynamical core constants (original port)
  - `GFS` Constant as defined in NOAA GFS
  - `GEOS`  Constant as defined in GEOS v13
- `PACE_FLOAT_PRECISION`: default precision of the field & scalars in the numerics. Default to 64.
- `PACE_LOGLEVEL`: logging level to display (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default to INFO.

## Quickstart - Docker

### Build

While it is possible to install and build pace bare-metal, we can ensure all system libraries are installed with the correct versions by using a Docker container to test and develop pace.

First, you will need to update the git submodules so that any dependencies are cloned and at the correct version:

```shell
git submodule update --init --recursive
```

Then build the `pace` docker image at the top level.

```shell
make build
```

### Run

```shell
make dev
mpirun --mca btl_vader_single_copy_mechanism none -n 6 python3 -m pace.run /examples/configs/baroclinic_c12.yaml
```

## History

This repository was first developed at [AI2](https://github.com/ai2cm/pace) and the institute conserves an archived copy with the latest state before the NOAA took over.

[contributors-shield]: https://img.shields.io/github/contributors/NOAA-GFDL/pace.svg
[contributors-url]: https://github.com/NOAA-GFDL/pace/graphs/contributors
[stars-shield]: https://img.shields.io/github/stars/NOAA-GFDL/pace.svg
[stars-url]: https://github.com/NOAA-GFDL/pace/stargazers
[issues-shield]: https://img.shields.io/github/issues/NOAA-GFDL/pace.svg
[issues-url]: https://github.com/NOAA-GFDL/pace/issues
[license-shield]: https://img.shields.io/github/license/NOAA-GFDL/pace.svg
[license-url]: https://github.com/NOAA-GFDL/pace/blob/main/LICENSE.md

## Running pace in containers
Docker images exist in the Github Container Registry associated with the NOAA-GFDL organization.
These images are publicly accessible and can be used to run a Docker container to work with pace.
The following are directions on how to setup the pace conda environment interactively in a container.

The latest images can be pulled with the Docker as shown below or
with any other container management tools:

```shell
docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8
```
for MPICH installation of MPI; and
```shell
docker pull ghcr.io/noaa-gfdl/pace_openmpi:3.8
```
for OpenMPI installation of MPI.

If permission issues arise during the pull, a Github personal token
may be required.  The steps to create a personal token is found
[here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)

Once the token has been generated, the image can be pulled for example with with:
```shell
docker login --username GITHUB_USERNAME --password TOKEN
docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8
```

Any container management tools compatible with Docker images can be used
to run the container interactively from the pulled image.
With Docker, the following command runs the container interactively.
```shell
docker run -it pace_mpich:3.8
```

In the container, the default `base` conda environment is already activated.
The `pace` conda environment can be created by following the steps below:

```shell
git clone --recursive -b develop https://github.com/NOAA-GFDL/pace.git pace
cd pace
cp /home/scripts/setup_env.sh . && chmod +x setup_env.sh
source ./setup_env.sh
```