## Open-Sora: Democratizing Efficient Video Production for All

We design and implement **Open-Sora**, an initiative dedicated to **efficiently** producing high-quality video. We hope to make the model,
tools and all details accessible to all. By embracing **open-source** principles,
Open-Sora not only democratizes access to advanced video generation techniques, but also offers a
streamlined and user-friendly platform that simplifies the complexities of video generation.
With Open-Sora, our goal is to foster innovation, creativity, and inclusivity within the field of content creation.

[[中文文档](/docs/zh_CN/README.md)] [[潞晨云](https://cloud.luchentech.com/)|[OpenSora镜像](https://cloud.luchentech.com/doc/docs/image/open-sora/)|[视频教程](https://www.bilibili.com/video/BV1ow4m1e7PX/?vd_source=c6b752764cd36ff0e535a768e35d98d2)]

## 📰 News

- **[2024.06.17]** 🔥 We released **Open-Sora 1.2**, which includes **3D-VAE**, **rectified flow**, and **score condition**. The video quality is greatly improved. [[checkpoints]](#open-sora-10-model-weights) [[report]](/docs/report_03.md)   [[blog]](https://hpc-ai.com/blog/open-sora-from-hpc-ai-tech-team-continues-open-source-generate-any-16-second-720p-hd-video-with-one-click-model-weights-ready-to-use)
- **[2024.04.25]** 🤗 We released the [Gradio demo for Open-Sora](https://huggingface.co/spaces/hpcai-tech/open-sora) on Hugging Face Spaces.
- **[2024.04.25]** We released **Open-Sora 1.1**, which supports **2s~15s, 144p to 720p, any aspect ratio** text-to-image, **text-to-video, image-to-video, video-to-video, infinite time** generation. In addition, a full video processing pipeline is released. [[checkpoints]]() [[report]](/docs/report_02.md)
- **[2024.03.18]** We released **Open-Sora 1.0**, a fully open-source project for video generation.
  Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with

  acceleration,
  inference, and more. Our model can produce 2s 512x512 videos with only 3 days training. [[checkpoints]](#open-sora-10-model-weights)
  [[blog]](https://hpc-ai.com/blog/open-sora-v1.0) [[report]](/docs/report_01.md)
- **[2024.03.04]** Open-Sora provides training with 46% cost reduction.
  [[blog]](https://hpc-ai.com/blog/open-sora)

## 🎥 Latest Demo

🔥 You can experience Open-Sora on our [🤗 Gradio application on Hugging Face](https://huggingface.co/spaces/hpcai-tech/open-sora). More samples and corresponding prompts are available in our [Gallery](https://hpcaitech.github.io/Open-Sora/).

| **4s 720×1280**                                                                                | **4s 720×1280**                                                                                | **4s 720×1280**                                                                                |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/7895aab6-ed23-488c-8486-091480c26327) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/20f07c7b-182b-4562-bbee-f1df74c86c9a) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/3d897e0d-dc21-453a-b911-b3bda838acc2) |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/644bf938-96ce-44aa-b797-b3c0b513d64c) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/272d88ac-4b4a-484d-a665-8d07431671d0) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/ebbac621-c34e-4bb4-9543-1c34f8989764) |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/a1e3a1a3-4abd-45f5-8df2-6cced69da4ca) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/d6ce9c13-28e1-4dff-9644-cc01f5f11926) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/561978f8-f1b0-4f4d-ae7b-45bec9001b4a) |

OpenSora 1.1 Demo

| **2s 240×426**                                                                                | **2s 240×426**                                                                                |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) | [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c31ebc52-de39-4a4e-9b1e-9211d45e05b2) |
| [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/f7ce4aaa-528f-40a8-be7a-72e61eaacbbd)  | [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/5d58d71e-1fda-4d90-9ad3-5f2f7b75c6a9) |

| **2s 426×240**                                                                                | **4s 480×854**                                                                                |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/34ecb4a0-4eef-4286-ad4c-8e3a87e5a9fd) | [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/c1619333-25d7-42ba-a91c-18dbc1870b18) |

| **16s 320×320**                                                                                | **16s 224×448**                                                                                | **2s 426×240**                                                                                |
| ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/3cab536e-9b43-4b33-8da8-a0f9cf842ff2) | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/9fb0b9e0-c6f4-4935-b29e-4cac10b373c4) | [](https://github.com/hpcaitech/Open-Sora-dev/assets/99191637/3e892ad2-9543-4049-b005-643a4c1bf3bf) |

OpenSora 1.0 Demo

| **2s 512×512**                                                                                | **2s 512×512**                                                                                | **2s 512×512**                                                                                |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/de1963d3-b43b-4e68-a670-bb821ebb6f80)                                 | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/13f8338f-3d42-4b71-8142-d234fbd746cc)                              | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/fa6a65a6-e32a-4d64-9a9e-eabb0ebb8c16)    |
| A serene night scene in a forested area. [...] The video is a time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop. | A soaring drone footage captures the majestic beauty of a coastal cliff, [...] The water gently laps at the rock base and the greenery that clings to the top of the cliff. | The majestic beauty of a waterfall cascading down a cliff into a serene lake. [...] The camera angle provides a bird's eye view of the waterfall. |
| [](https://github.com/hpcaitech/Open-Sora/assets/99191637/64232f84-1b36-4750-a6c0-3e610fa9aa94)                                 | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/983a1965-a374-41a7-a76b-c07941a6c1e9)                              | [](https://github.com/hpcaitech/Open-Sora/assets/99191637/ec10c879-9767-4c31-865f-2e8d6cf11e65)    |
| A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. [...]                                                           | The vibrant beauty of a sunflower field. The sunflowers are arranged in neat rows, creating a sense of order and symmetry. [...]                                            | A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell [...]                   |

Videos are downsampled to `.gif` for display. Click for original videos. Prompts are trimmed for display,
see [here](/assets/texts/t2v_samples.txt) for full prompts.

## 🔆 New Features/Updates

- 📍 **Open-Sora 1.2** released. Model weights are available [here](#model-weights). See our **[report 1.2](/docs/report_03.md)** for more details.
- ✅ Support rectified flow scheduling.
- ✅ Support more conditioning including fps, aesthetic score, motion strength and camera motion.
- ✅ Trained our 3D-VAE for temporal dimension compression.
- 📍 **Open-Sora 1.1** released. Model weights are available [here](#model-weights). It is trained on **0s~15s, 144p to 720p, various aspect ratios** videos. See our **[report 1.1](/docs/report_02.md)** for more discussions.
- 🔧 **Data processing pipeline v1.1** is released. An automatic [processing pipeline](#data-processing) from raw videos to (text, video clip) pairs is provided, including scene cutting $\rightarrow$ filtering(aesthetic, optical flow, OCR, etc.) $\rightarrow$ captioning $\rightarrow$ managing. With this tool, you can easily build your video dataset.

View more

- ✅ Improved ST-DiT architecture includes rope positional encoding, qk norm, longer text length, etc.
- ✅ Support training with any resolution, aspect ratio, and duration (including images).
- ✅ Support image and video conditioning and video editing, and thus support animating images, connecting videos, etc.
- 📍 **Open-Sora 1.0** released. Model weights are available [here](#model-weights). With only 400K video clips and 200 H800
  days (compared with 152M samples in Stable Video Diffusion), we are able to generate 2s 512×512 videos. See our **[report 1.0](docs/report_01.md)** for more discussions.
- ✅ Three-stage training from an image diffusion model to a video diffusion model. We provide the weights for each
  stage.
- ✅ Support training acceleration including accelerated transformer, faster T5 and VAE, and sequence parallelism.
  Open-Sora improves **55%** training speed when training on 64x512x512 videos. Details locates
  at [acceleration.md](docs/acceleration.md).
- 🔧 **Data preprocessing pipeline v1.0**,
  including [downloading](tools/datasets/README.md), [video cutting](tools/scene_cut/README.md),
  and [captioning](tools/caption/README.md) tools. Our data collection plan can be found
  at [datasets.md](docs/datasets.md).
- ✅ We find VQ-VAE from [VideoGPT](https://wilson1yan.github.io/videogpt/index.html) has a low quality and thus adopt a
  better VAE from [Stability-AI](https://huggingface.co/stabilityai/sd-vae-ft-mse-original). We also find patching in
  the time dimension deteriorates the quality. See our **[report](docs/report_01.md)** for more discussions.
- ✅ We investigate different architectures including DiT, Latte, and our proposed STDiT. Our **STDiT** achieves a better
  trade-off between quality and speed. See our **[report](docs/report_01.md)** for more discussions.
- ✅ Support clip and T5 text conditioning.
- ✅ By viewing images as one-frame videos, our project supports training DiT on both images and videos (e.g., ImageNet &
  UCF101). See [commands.md](docs/commands.md) for more instructions.
- ✅ Support inference with official weights
  from [DiT](https://github.com/facebookresearch/DiT), [Latte](https://github.com/Vchitect/Latte),
  and [PixArt](https://pixart-alpha.github.io/).
- ✅ Refactor the codebase. See [structure.md](docs/structure.md) to learn the project structure and how to use the
  config files.

### TODO list sorted by priority

View more

- [x] Training Video-VAE and adapt our model to new VAE.
- [x] Scaling model parameters and dataset size.
- [x] Incoporate a better scheduler (rectified flow).
- [x] Evaluation pipeline.
- [x] Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, etc.). See [the dataset](/docs/datasets.md) for more information
- [x] Support image and video conditioning.
- [x] Support variable aspect ratios, resolutions, durations.

## Contents

- [Installation](#installation)
- [Model Weights](#model-weights)
- [Gradio Demo](#gradio-demo)
- [Inference](#inference)
- [Data Processing](#data-processing)
- [Training](#training)
- [Evaluation](#evaluation)
- [VAE Training & Evaluation](#vae-training--evaluation)
- [Contribution](#contribution)
- [Citation](#citation)
- [Acknowledgement](#acknowledgement)

Other useful documents and links are listed below.

- Report: each version is trained from a image base seperately (not continuously trained), while a newer version will incorporate the techniques from the previous version.
  - [report 1.2](docs/report_03.md): rectified flow, 3d-VAE, score condition, evaluation, etc.
  - [report 1.1](docs/report_02.md): multi-resolution/length/aspect-ratio, image/video conditioning/editing, data preprocessing, etc.
  - [report 1.0](docs/report_01.md): architecture, captioning, etc.
  - [acceleration.md](docs/acceleration.md)
- Repo structure: [structure.md](docs/structure.md)
- Config file explanation: [config.md](docs/config.md)
- Useful commands: [commands.md](docs/commands.md)
- Data processing pipeline and dataset: [datasets.md](docs/datasets.md)
- Each data processing tool's README: [dataset conventions and management](/tools/datasets/README.md), [scene cutting](/tools/scene_cut/README.md), [scoring](/tools/scoring/README.md), [caption](/tools/caption/README.md)
- Evaluation: [eval/README.md](/eval/README.md)
- Gallery: [gallery](https://hpcaitech.github.io/Open-Sora/)

## Installation

### Install from Source

For CUDA 12.1, you can install the dependencies with the following commands. Otherwise, please refer to [Installation Documentation](docs/installation.md) for more instructions on different cuda version, and additional dependency for data preprocessing, VAE, and model evaluation.

```bash
# create a virtual env and activate (conda as an example)
conda create -n opensora python=3.9
conda activate opensora

# download the repo
git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora

# install torch, torchvision and xformers
pip install -r requirements/requirements-cu121.txt

# the default installation is for inference only
pip install -v . # for development mode, `pip install -v -e .`
```

(Optional, recommended for fast speed, especially for training) To enable `layernorm_kernel` and `flash_attn`, you need to install `apex` and `flash-attn` with the following commands.

```bash
# install flash attention
# set enable_flash_attn=False in config to disable flash attention
pip install packaging ninja
pip install flash-attn --no-build-isolation

# install apex
# set enable_layernorm_kernel=False in config to disable apex
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
```

### Use Docker

Run the following command to build a docker image from Dockerfile provided.

```bash
docker build -t opensora .
```

Run the following command to start the docker container in interactive mode.

```bash
docker run -ti --gpus all -v .:/workspace/Open-Sora opensora
```

## Model Weights

### Open-Sora 1.2 Model Weights

| Model     | Model Size | Data | #iterations | Batch Size | URL                                                           |
| --------- | ---------- | ---- | ----------- | ---------- | ------------------------------------------------------------- |
| Diffusion | 1.1B       | 30M  | 70k         | Dynamic    | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v3) |
| VAE       | 384M       | 3M   | 1M          | 8          | [:link:](https://huggingface.co/hpcai-tech/OpenSora-VAE-v1.2) |

See our **[report 1.2](docs/report_03.md)** for more infomation. Weight will be automatically downloaded when you run the inference script.

> For users from mainland China, try `export HF_ENDPOINT=https://hf-mirror.com` to successfully download the weights.

### Open-Sora 1.1 Model Weights

View more

| Resolution         | Model Size | Data                       | #iterations | Batch Size                                        | URL                                                                  |
| ------------------ | ---------- | -------------------------- | ----------- | ------------------------------------------------- | -------------------------------------------------------------------- |
| mainly 144p & 240p | 700M       | 10M videos + 2M images     | 100k        | [dynamic](/configs/opensora-v1-1/train/stage2.py) | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage2) |
| 144p to 720p       | 700M       | 500K HQ videos + 1M images | 4k          | [dynamic](/configs/opensora-v1-1/train/stage3.py) | [:link:](https://huggingface.co/hpcai-tech/OpenSora-STDiT-v2-stage3) |

See our **[report 1.1](docs/report_02.md)** for more infomation.

:warning: **LIMITATION**: This version contains known issues which we are going to fix in the next version (as we save computation resource for the next release). In addition, the video generation may fail for long duration, and high resolution will have noisy results due to this problem.

### Open-Sora 1.0 Model Weights

View more

| Resolution | Model Size | Data   | #iterations | Batch Size | GPU days (H800) | URL                                                                                |
| ---------- | ---------- | ------ | ----------- | ---------- | --------------- | --------------------------------------------------------------------------------------------- |
| 16×512×512 | 700M       | 20K HQ | 20k         | 2×64       | 35              | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x512x512.pth) |
| 16×256×256 | 700M       | 20K HQ | 24k         | 8×64       | 45              | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-HQ-16x256x256.pth) |
| 16×256×256 | 700M       | 366K   | 80k         | 8×64       | 117             | [:link:](https://huggingface.co/hpcai-tech/Open-Sora/blob/main/OpenSora-v1-16x256x256.pth)    |

Training orders: 16x256x256 $\rightarrow$ 16x256x256 HQ $\rightarrow$ 16x512x512 HQ.

Our model's weight is partially initialized from [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha). The number of
parameters is 724M. More information about training can be found in our **[report](/docs/report_01.md)**. More about
the dataset can be found in [datasets.md](/docs/datasets.md). HQ means high quality.

:warning: **LIMITATION**: Our model is trained on a limited budget. The quality and text alignment is relatively poor.
The model performs badly, especially on generating human beings and cannot follow detailed instructions. We are working
on improving the quality and text alignment.

## Gradio Demo

🔥 You can experience Open-Sora on our [🤗 Gradio application](https://huggingface.co/spaces/hpcai-tech/open-sora) on Hugging Face online.

### Local Deployment

If you want to deploy gradio locally, we have also provided a [Gradio application](./gradio) in this repository, you can use the following the command to start an interactive web application to experience video generation with Open-Sora.

```bash
pip install gradio spaces
python gradio/app.py
```

This will launch a Gradio application on your localhost. If you want to know more about the Gradio applicaiton, you can refer to the [Gradio README](./gradio/README.md).

To enable prompt enhancement and other language input (e.g., 中文输入), you need to set the `OPENAI_API_KEY` in the environment. Check [OpenAI's documentation](https://platform.openai.com/docs/quickstart) to get your API key.

```bash
export OPENAI_API_KEY=YOUR_API_KEY
```

### Getting Started

In the Gradio application, the basic options are as follows:

![Gradio Demo](assets/readme/gradio_basic.png)

The easiest way to generate a video is to input a text prompt and click the "**Generate video**" button (scroll down if you cannot find). The generated video will be displayed in the right panel. Checking the "**Enhance prompt with GPT4o**" will use GPT-4o to refine the prompt, while "**Random Prompt**" button will generate a random prompt by GPT-4o for you. Due to the OpenAI's API limit, the prompt refinement result has some randomness.

Then, you can choose the **resolution**, **duration**, and **aspect ratio** of the generated video. Different resolution and video length will affect the video generation speed. On a 80G H100 GPU, the generation speed (with `num_sampling_step=30`) and peak memory usage is:

|      | Image   | 2s       | 4s        | 8s        | 16s       |
| ---- | ------- | -------- | --------- | --------- | --------- |
| 360p | 3s, 24G | 18s, 27G | 31s, 27G  | 62s, 28G  | 121s, 33G |
| 480p | 2s, 24G | 29s, 31G | 55s, 30G  | 108s, 32G | 219s, 36G |
| 720p | 6s, 27G | 68s, 41G | 130s, 39G | 260s, 45G | 547s, 67G |

Note that besides text to video, you can also use **image to video generation**. You can upload an image and then click the "**Generate video**" button to generate a video with the image as the first frame. Or you can fill in the text prompt and click the "**Generate image**" button to generate an image with the text prompt, and then click the "**Generate video**" button to generate a video with the image generated with the same model.

![Gradio Demo](assets/readme/gradio_option.png)

Then you can specify more options, including "**Motion Strength**", "**Aesthetic**" and "**Camera Motion**". If "Enable" not checked or the choice is "none", the information is not passed to the model. Otherwise, the model will generate videos with the specified motion strength, aesthetic score, and camera motion.

For the **aesthetic score**, we recommend using values higher than 6. For **motion strength**, a smaller value will lead to a smoother but less dynamic video, while a larger value will lead to a more dynamic but likely more blurry video. Thus, you can try without it and then adjust it according to the generated video. For the **camera motion**, sometimes the model cannot follow the instruction well, and we are working on improving it.

You can also adjust the "**Sampling steps**", this is directly related to the generation speed as it is the number of denoising. A number smaller than 30 usually leads to a poor generation results, while a number larger than 100 usually has no significant improvement. The "**Seed**" is used for reproducibility, you can set it to a fixed number to generate the same video. The "**CFG Scale**" controls how much the model follows the text prompt, a smaller value will lead to a more random video, while a larger value will lead to a more text-following video (7 is recommended).

For more advanced usage, you can refer to [Gradio README](./gradio/README.md#advanced-usage).

## Inference

### Open-Sora 1.2 Command Line Inference

The basic command line inference is as follows:

```bash
# text to video
python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
  --num-frames 4s --resolution 720p --aspect-ratio 9:16 \
  --prompt "a beautiful waterfall"
```

You can add more options to the command line to customize the generation.

```bash
python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
  --num-frames 4s --resolution 720p --aspect-ratio 9:16 \
  --num-sampling-steps 30 --flow 5 --aes 6.5 \
  --prompt "a beautiful waterfall"
```

For image to video generation and other functionalities, the API is compatible with Open-Sora 1.1. See [here](docs/commands.md) for more instructions.

If your installation do not contain `apex` and `flash-attn`, you need to disable them in the config file, or via the folowing command.

```bash
python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
  --num-frames 4s --resolution 720p \
  --layernorm-kernel False --flash-attn False \
  --prompt "a beautiful waterfall"
```

### Sequence Parallelism Inference

To enable sequence parallelism, you need to use `torchrun` to run the inference script. The following command will run the inference with 2 GPUs.

```bash
# text to video
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 scripts/inference.py configs/opensora-v1-2/inference/sample.py \
  --num-frames 4s --resolution 720p --aspect-ratio 9:16 \
  --prompt "a beautiful waterfall"
```

:warning: **LIMITATION**: The sequence parallelism is not supported for gradio deployment. For now, the sequence parallelism is only supported when the dimension can be divided by the number of GPUs. Thus, it may fail for some cases. We tested 4 GPUs for 720p and 2 GPUs for 480p.

### GPT-4o Prompt Refinement

We find that GPT-4o can refine the prompt and improve the quality of the generated video. With this feature, you can also use other language (e.g., Chinese) as the prompt. To enable this feature, you need prepare your openai api key in the environment:

```bash
export OPENAI_API_KEY=YOUR_API_KEY
```

Then you can inference with `--llm-refine True` to enable the GPT-4o prompt refinement, or leave prompt empty to get a random prompt generated by GPT-4o.

```bash
python scripts/inference.py configs/opensora-v1-2/inference/sample.py \
  --num-frames 4s --resolution 720p --llm-refine True
```

### Open-Sora 1.1 Command Line Inference

View more

Since Open-Sora 1.1 supports inference with dynamic input size, you can pass the input size as an argument.

```bash
# text to video
python scripts/inference.py configs/opensora-v1-1/inference/sample.py --prompt "A beautiful sunset over the city" --num-frames 32 --image-size 480 854
```

If your installation do not contain `apex` and `flash-attn`, you need to disable them in the config file, or via the folowing command.

```bash
python scripts/inference.py configs/opensora-v1-1/inference/sample.py --prompt "A beautiful sunset over the city" --num-frames 32 --image-size 480 854 --layernorm-kernel False --flash-attn False
```

See [here](docs/commands.md#inference-with-open-sora-11) for more instructions including text-to-image, image-to-video, video-to-video, and infinite time generation.

### Open-Sora 1.0 Command Line Inference

View more

We have also provided an offline inference script. Run the following commands to generate samples, the required model weights will be automatically downloaded. To change sampling prompts, modify the txt file passed to `--prompt-path`. See [here](docs/structure.md#inference-config-demos) to customize the configuration.

```bash
# Sample 16x512x512 (20s/sample, 100 time steps, 24 GB memory)
torchrun --standalone --nproc_per_node 1 scripts/inference.py configs/opensora/inference/16x512x512.py --ckpt-path OpenSora-v1-HQ-16x512x512.pth --prompt-path ./assets/texts/t2v_samples.txt

# Sample 16x256x256 (5s/sample, 100 time steps, 22 GB memory)
torchrun --standalone --nproc_per_node 1 scripts/inference.py configs/opensora/inference/16x256x256.py --ckpt-path OpenSora-v1-HQ-16x256x256.pth --prompt-path ./assets/texts/t2v_samples.txt

# Sample 64x512x512 (40s/sample, 100 time steps)
torchrun --standalone --nproc_per_node 1 scripts/inference.py configs/opensora/inference/64x512x512.py --ckpt-path ./path/to/your/ckpt.pth --prompt-path ./assets/texts/t2v_samples.txt

# Sample 64x512x512 with sequence parallelism (30s/sample, 100 time steps)
# sequence parallelism is enabled automatically when nproc_per_node is larger than 1
torchrun --standalone --nproc_per_node 2 scripts/inference.py configs/opensora/inference/64x512x512.py --ckpt-path ./path/to/your/ckpt.pth --prompt-path ./assets/texts/t2v_samples.txt
```

The speed is tested on H800 GPUs. For inference with other models, see [here](docs/commands.md) for more instructions.
To lower the memory usage, set a smaller `vae.micro_batch_size` in the config (slightly lower sampling speed).

## Data Processing

High-quality data is crucial for training good generation models.
To this end, we establish a complete pipeline for data processing, which could seamlessly convert raw videos to high-quality video-text pairs.
The pipeline is shown below. For detailed information, please refer to [data processing](docs/data_processing.md).
Also check out the [datasets](docs/datasets.md) we use.

![Data Processing Pipeline](assets/readme/report_data_pipeline.png)

## Training

### Open-Sora 1.2 Training

The training process is same as Open-Sora 1.1.

```bash
# one node
torchrun --standalone --nproc_per_node 8 scripts/train.py \
    configs/opensora-v1-2/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
# multiple nodes
colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \
    configs/opensora-v1-2/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
```

### Open-Sora 1.1 Training

View more

Once you prepare the data in a `csv` file, run the following commands to launch training on a single node.

```bash
# one node
torchrun --standalone --nproc_per_node 8 scripts/train.py \
    configs/opensora-v1-1/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
# multiple nodes
colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py \
    configs/opensora-v1-1/train/stage1.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
```

### Open-Sora 1.0 Training

View more

Once you prepare the data in a `csv` file, run the following commands to launch training on a single node.

```bash
# 1 GPU, 16x256x256
torchrun --nnodes=1 --nproc_per_node=1 scripts/train.py configs/opensora/train/16x256x256.py --data-path YOUR_CSV_PATH
# 8 GPUs, 64x512x512
torchrun --nnodes=1 --nproc_per_node=8 scripts/train.py configs/opensora/train/64x512x512.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
```

To launch training on multiple nodes, prepare a hostfile according
to [ColossalAI](https://colossalai.org/docs/basics/launch_colossalai/#launch-with-colossal-ai-cli), and run the
following commands.

```bash
colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train.py configs/opensora/train/64x512x512.py --data-path YOUR_CSV_PATH --ckpt-path YOUR_PRETRAINED_CKPT
```

For training other models and advanced usage, see [here](docs/commands.md) for more instructions.

## Evaluation

We support evaluation based on:

- Validation loss
- [VBench](https://github.com/Vchitect/VBench/tree/master) score
- VBench-i2v score
- Batch generation for human evaluation

All the evaluation code is released in `eval` folder. Check the [README](/eval/README.md) for more details. Our [report](/docs/report_03.md#evaluation) also provides more information about the evaluation during training. The following table shows Open-Sora 1.2 greatly improves Open-Sora 1.0.

| Model          | Total Score | Quality Score | Semantic Score |
| -------------- | ----------- | ------------- | -------------- |
| Open-Sora V1.0 | 75.91%      | 78.81%        | 64.28%         |
| Open-Sora V1.2 | 79.23%      | 80.71%        | 73.30%         |

## VAE Training & Evaluation

We train a VAE pipeline that consists of a spatial VAE followed by a temporal VAE.
For more details, refer to [VAE Documentation](docs/vae.md).
Before you run the following commands, follow our [Installation Documentation](docs/installation.md) to install the required dependencies for VAE and Evaluation.

If you want to train your own VAE, we need to prepare data in the csv following the [data processing](#data-processing) pipeline, then run the following commands.
Note that you need to adjust the number of trained epochs (`epochs`) in the config file accordingly with respect to your own csv data size.

```bash
# stage 1 training, 380k steps, 8 GPUs
torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage1.py --data-path YOUR_CSV_PATH
# stage 2 training, 260k steps, 8 GPUs
torchrun --nnodes=1 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage2.py --data-path YOUR_CSV_PATH
# stage 3 training, 540k steps, 24 GPUs
torchrun --nnodes=3 --nproc_per_node=8 scripts/train_vae.py configs/vae/train/stage3.py --data-path YOUR_CSV_PATH
```

To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:

```bash
# video generation
torchrun --standalone --nnodes=1 --nproc_per_node=1 scripts/inference_vae.py configs/vae/inference/video.py --ckpt-path YOUR_VAE_CKPT_PATH --data-path YOUR_CSV_PATH --save-dir YOUR_VIDEO_DIR
# the original videos will be saved to `YOUR_VIDEO_DIR_ori`
# the reconstructed videos through the pipeline will be saved to `YOUR_VIDEO_DIR_rec`
# the reconstructed videos through the spatial VAE only will be saved to `YOUR_VIDEO_DIR_spatial`

# score calculation
python eval/vae/eval_common_metric.py --batch_size 2 --real_video_dir YOUR_VIDEO_DIR_ori --generated_video_dir YOUR_VIDEO_DIR_rec --device cuda --sample_fps 24 --crop_size 256 --resolution 256 --num_frames 17 --sample_rate 1 --metric ssim psnr lpips flolpips
```

## Contribution

Thanks goes to these wonderful contributors:

If you wish to contribute to this project, please refer to the [Contribution Guideline](./CONTRIBUTING.md).

## Acknowledgement

Here we only list a few of the projects. For other works and datasets, please refer to our report.

- [ColossalAI](https://github.com/hpcaitech/ColossalAI): A powerful large model parallel acceleration and optimization
  system.
- [DiT](https://github.com/facebookresearch/DiT): Scalable Diffusion Models with Transformers.
- [OpenDiT](https://github.com/NUS-HPC-AI-Lab/OpenDiT): An acceleration for DiT training. We adopt valuable acceleration
  strategies for training progress from OpenDiT.
- [PixArt](https://github.com/PixArt-alpha/PixArt-alpha): An open-source DiT-based text-to-image model.
- [Latte](https://github.com/Vchitect/Latte): An attempt to efficiently train DiT for video.
- [StabilityAI VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse-original): A powerful image VAE model.
- [CLIP](https://github.com/openai/CLIP): A powerful text-image embedding model.
- [T5](https://github.com/google-research/text-to-text-transfer-transformer): A powerful text encoder.
- [LLaVA](https://github.com/haotian-liu/LLaVA): A powerful image captioning model based on [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) and [Yi-34B](https://huggingface.co/01-ai/Yi-34B).
- [PLLaVA](https://github.com/magic-research/PLLaVA): A powerful video captioning model.
- [MiraData](https://github.com/mira-space/MiraData): A large-scale video dataset with long durations and structured caption.

We are grateful for their exceptional work and generous contribution to open source. Special thanks go to the authors of [MiraData](https://github.com/mira-space/MiraData) and [Rectified Flow](https://github.com/gnobitab/RectifiedFlow) for their valuable advice and help. We wish to express gratitude towards AK for sharing this project on social media and Hugging Face for providing free GPU resources for our online Gradio demo.

## Citation

```bibtex
@software{opensora,
  author = {Zangwei Zheng and Xiangyu Peng and Tianji Yang and Chenhui Shen and Shenggui Li and Hongxin Liu and Yukun Zhou and Tianyi Li and Yang You},
  title = {Open-Sora: Democratizing Efficient Video Production for All},
  month = {March},
  year = {2024},
  url = {https://github.com/hpcaitech/Open-Sora}
}
```

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=hpcaitech/Open-Sora&type=Date)](https://star-history.com/#hpcaitech/Open-Sora&Date)


Combinatorial Specification Searcher
====================================
.. image:: https://travis-ci.org/PermutaTriangle/comb_spec_searcher.svg?branch=master
    :alt: Travis
    :target: https://travis-ci.org/PermutaTriangle/comb_spec_searcher
.. image:: https://img.shields.io/coveralls/github/PermutaTriangle/comb_spec_searcher.svg
    :alt: Coveralls
    :target: https://coveralls.io/github/PermutaTriangle/comb_spec_searcher
.. image:: https://img.shields.io/pypi/v/comb_spec_searcher.svg
    :alt: PyPI
    :target: https://pypi.python.org/pypi/comb_spec_searcher
.. image:: https://img.shields.io/pypi/l/comb_spec_searcher.svg
    :target: https://pypi.python.org/pypi/comb_spec_searcher
.. image:: https://img.shields.io/pypi/pyversions/comb_spec_searcher.svg
    :target: https://pypi.python.org/pypi/comb_spec_searcher
.. image:: http://img.shields.io/badge/readme-tested-brightgreen.svg
    :alt: Travis
    :target: https://travis-ci.org/PermutaTriangle/comb_spec_searcher
.. image:: https://requires.io/github/PermutaTriangle/comb_spec_searcher/requirements.svg?branch=master
     :target: https://requires.io/github/PermutaTriangle/comb_spec_searcher/requirements/?branch=master
     :alt: Requirements Status
.. image:: https://zenodo.org/badge/121520109.svg
   :target: https://zenodo.org/badge/latestdoi/121520109

The ``comb_spec_searcher`` package contains code for combinatorial
exploration.

If this code is useful to you in your work, please consider citing it. To generate a
BibTeX entry (or another format), click the "DOI" badge above and locate the "Cite As"
section.

Installing
----------

To install ``comb_spec_searcher`` on your system, run:

.. code:: bash

       pip install comb_spec_searcher

It is also possible to install comb_spec_searcher in development mode to
work on the source code, in which case you run the following after
cloning the repository:

.. code:: bash

       ./setup.py develop

Combinatorial exploration
-------------------------

A (combinatorial) class is a set of objects with a notion of size such
that there are finitely many objects of each size. One of the primary
goals of enumerative combinatorics is to count how many objects of each
size there are in a class. One method for doing this is to find a
(combinatorial) specification, which is a collection of (combinatorial)
rules that describe how to build a class from other classes using
well-defined constructors. Such a specification can then be used to
count the number of objects of each size.

*Combinatorial exploration* is a systematic application of strategies to
create rules about a class of interest, until a specification can be
found. This package can be used to perform this process automatically.
See the `Combinatorial Exploration
project <https://permutatriangle.github.io/papers/2019-02-27-combex.html>`__
and `Christian Bean’s PhD
thesis <https://opinvisindi.is/handle/20.500.11815/1184>`__ for more details.

The remainder of this README will be an example of how to use this
package for performing combinatorial exploration on a specific class,
namely words avoiding consecutive patterns.

Avoiding consecutive patterns in words
--------------------------------------

A word ``w`` over an alphabet ``Σ`` is a string consisting of letters
from ``Σ``. We say that ``w`` contains the word ``p`` if there is a
consecutive sequence of letters in ``w`` equal to ``p``. We say ``w``
avoids ``p`` if it does not contain ``p``. In this context, we call
``p`` a pattern. In ``python``, this containment check can be checked
using ``in``.

.. code:: python

   >>> from comb_spec_searcher import CombinatorialObject

   >>> class Word(str, CombinatorialObject):
   ...     def size(self):
   ...         return str.__len__(self)

   >>> w = Word("acbabcabbb")
   >>> p = Word("abcab")
   >>> p in w
   True

For an alphabet ``Σ`` and a set of patterns ``P`` we define ``Σ*(P)`` to
be the set of words over ``Σ`` that avoid every pattern in ``P``. These
are the classes that we will count. Of course, these all form regular
languages, but it will serve as a good example of how to use the
``comb_spec_searcher`` package.

The first step is to create the classes that will be used for
discovering the underlying structure of the class of interest. In this
case, considering the prefix of the words is what we need. We then
create a new python ``class`` representing this that inherits from
``CombinatorialClass`` which can be imported from
``comb_spec_searcher``.

.. code:: python

   >>> from comb_spec_searcher import CombinatorialClass

   >>> class AvoidingWithPrefix(CombinatorialClass[Word]):
   ...     def __init__(self, prefix, patterns, alphabet, just_prefix=False):
   ...         self.alphabet = frozenset(alphabet)
   ...         self.prefix = Word(prefix)
   ...         self.patterns = frozenset(map(Word, patterns))
   ...         self.just_prefix = just_prefix # this will be needed later

Inheriting from ``CombinatorialClass`` requires you to implement a few
functions for combinatorial exploration: ``is_empty``, ``to_jsonable``,
``__eq__``, ``__hash__``, ``__repr__``, and ``__str__``.

We will start by implementing the dunder methods (the ones with double
underscores) required. The ``__eq__`` method is particularly important
as the ``CombinatorialSpecificationSearcher`` will use it to recognise
if the same class appears multiple times.

.. code:: python

   ...     # The dunder methods required to perform combinatorial exploration
   ...
   ...     def __eq__(self, other):
   ...         return (self.alphabet == other.alphabet and
   ...                 self.prefix == other.prefix and
   ...                 self.patterns == other.patterns and
   ...                 self.just_prefix == other.just_prefix)
   ...
   ...     def __hash__(self):
   ...         return hash(hash(self.prefix) + hash(self.patterns) +
   ...                     hash(self.alphabet) + hash(self.just_prefix))
   ...
   ...     def __str__(self):
   ...         prefix = self.prefix if self.prefix else '""'
   ...         if self.just_prefix:
   ...             return "The word {}".format(prefix)
   ...         return ("Words over {{{}}} avoiding {{{}}} with prefix {}"
   ...                 "".format(", ".join(l for l in self.alphabet),
   ...                           ", ".join(p for p in self.patterns),
   ...                           prefix))
   ...
   ...     def __repr__(self):
   ...         return "AvoidingWithPrefix({}, {}, {}".format(repr(self.prefix),
   ...                                                       repr(self.patterns),
   ...                                                       repr(self.alphabet))

Perhaps the most important function to be implemented is the
``is_empty`` function. This should return ``True`` if there are no
objects of any size in the class, otherwise ``False``. If it is not
correctly implemented it may lead to tautological specifications. For
example, in our case the class is empty if and only if the prefix
contains a pattern to be avoided.

.. code:: python

   ...     def is_empty(self):
   ...         return any(p in self.prefix for p in self.patterns)

The final function required is ``to_jsonable``. This is primarily for
the output, and only necessary for saving the output. It should be in a
format that can be interpretated by ``json``. What is important is that
the ``from_dict`` function is written in such a way that for any class
``c`` we have ``CombinatorialClass.from_dict(c.to_jsonable()) == c``.

.. code:: python

   ...     def to_jsonable(self):
   ...         return {"prefix": self.prefix,
   ...                 "patterns": tuple(sorted(self.patterns)),
   ...                 "alphabet": tuple(sorted(self.alphabet)),
   ...                 "just_prefix": int(self.just_prefix)}
   ...
   ...     @classmethod
   ...     def from_dict(cls, data):
   ...         return cls(data['prefix'],
   ...                    data['patterns'],
   ...                    data['alphabet'],
   ...                    bool(int(data['just_prefix'])))

We also add some methods that we will need to get the enumerations of the
objects later.

.. code:: python

   ...     def is_atom(self):
   ...        """Return True if the class contains a single word."""
   ...        return self.just_prefix
   ...
   ...     def minimum_size_of_object(self):
   ...        """Return the size of the smallest object in the class."""
   ...        return len(self.prefix)

Our ``CombinatorialClass`` is now ready. What is left to do is create
the strategies that the ``CombinatorialSpecificationSearcher`` will use
for performing combinatorial exploration. This is given in the form of a
``StrategyPack`` which can be imported from ``comb_spec_searcher`` that
we will populate in the remainder of this example.

.. code:: python

   >>> from comb_spec_searcher import AtomStrategy, StrategyPack
   >>> pack = StrategyPack(initial_strats=[],
   ...                     inferral_strats=[],
   ...                     expansion_strats=[],
   ...                     ver_strats=[AtomStrategy()],
   ...                     name=("Finding specification for words avoiding "
   ...                           "consecutive patterns."))

Strategies are functions that take as input a class ``C`` and produce
rules about ``C``. The types of strategies are as follows: -
``initial_strats``: yields rules for classes - ``inferral_strats``:
returns a single equivalence rule - ``expansion_strats``: yields rules
for classes - ``ver_strats``: returns a rule when the count of a class
is known.

In our pack we have already added the AtomStrategy. This will verify any
combinatorial class that is an atom, in particular this is determined by the
``is_atom`` method we implemented on ``CombinatorialClass``. To get the
enumeration at the end, the strategy also uses the method
``minimum_size_of_object`` on ``CombinatorialClass``. As we've implemented
these two methods already, we are free to use the ``AtomStrategy``.

Now we will create our first strategy. Every word over the alphabet ``Σ``
starting with prefix ``p`` is either just ``p`` or has prefix ``pa`` for some
``a`` in ``Σ``. This rule is splitting the original into disjoint subsets. We
call a rule using disjoint union a ``DisjointUnionStrategy``. Although in this
case thereis a unique rule created by the strategy, strategies are assumed to
create multiple rules, and as such should be implemented as generators.

.. code:: python

   >>> from comb_spec_searcher import DisjointUnionStrategy

   >>> class ExpansionStrategy(DisjointUnionStrategy[AvoidingWithPrefix, Word]):
   ...     def decomposition_function(self, avoiding_with_prefix):
   ...        if not avoiding_with_prefix.just_prefix:
   ...           alphabet, prefix, patterns = (
   ...                 avoiding_with_prefix.alphabet,
   ...                 avoiding_with_prefix.prefix,
   ...                 avoiding_with_prefix.patterns,
   ...           )
   ...           children = [AvoidingWithPrefix(prefix, patterns, alphabet, True)]
   ...           for a in alphabet:
   ...                 ends_with_a = AvoidingWithPrefix(prefix + a, patterns, alphabet)
   ...                 children.append(ends_with_a)
   ...           return tuple(children)
   ...
   ...     def formal_step(self):
   ...        return "Either just the prefix, or append a letter from the alphabet"
   ...
   ...     def forward_map(self, avoiding_with_prefix, word, children=None):
   ...        """
   ...        The backward direction of the underlying bijection used for object
   ...        generation and sampling.
   ...        """
   ...        assert isinstance(word, Word)
   ...        if children is None:
   ...           children = self.decomposition_function(avoiding_with_prefix)
   ...           assert children is not None
   ...        if len(word) == len(avoiding_with_prefix.prefix):
   ...           return (word,) + tuple(None for i in range(len(children) - 1))
   ...        for idx, child in enumerate(children[1:]):
   ...           if word[: len(child.prefix)] == child.prefix:
   ...                 return (
   ...                    tuple(None for _ in range(idx + 1))
   ...                    + (word,)
   ...                    + tuple(None for _ in range(len(children) - idx - 1))
   ...                 )
   ...
   ...     def __str__(self):
   ...        return self.formal_step()
   ...
   ...     def __repr__(self):
   ...        return self.__class__.__name__ + "()"
   ...
   ...     @classmethod
   ...     def from_dict(cls, d):
   ...        return cls()

The final strategy we will need is one that peels off much as possible
from the front of the prefix ``p`` such that the avoidance conditions
are unaffected. This should then give a rule that is a cartesian product
of the part that is peeled off together with the words whose prefix is
that of the remainder of the original prefix. We call rules whose
constructor is cartesian product a ``DecompositionRule``.

.. code:: python

   >>> from comb_spec_searcher import CartesianProductStrategy

   >>> class RemoveFrontOfPrefix(CartesianProductStrategy[AvoidingWithPrefix, Word]):
   ...     def decomposition_function(self, avoiding_with_prefix):
   ...        """If the k is the maximum size of a pattern to be avoided, then any
   ...        occurrence using indices further to the right of the prefix can use at
   ...        most the last k - 1 letters in the prefix."""
   ...        if not avoiding_with_prefix.just_prefix:
   ...           safe = self.index_safe_to_remove_up_to(avoiding_with_prefix)
   ...           if safe > 0:
   ...                 prefix, patterns, alphabet = (
   ...                    avoiding_with_prefix.prefix,
   ...                    avoiding_with_prefix.patterns,
   ...                    avoiding_with_prefix.alphabet,
   ...                 )
   ...                 start_prefix = prefix[:safe]
   ...                 end_prefix = prefix[safe:]
   ...                 start = AvoidingWithPrefix(start_prefix, patterns, alphabet, True)
   ...                 end = AvoidingWithPrefix(end_prefix, patterns, alphabet)
   ...                 return (start, end)
   ...
   ...     def index_safe_to_remove_up_to(self, avoiding_with_prefix):
   ...        prefix, patterns = (
   ...           avoiding_with_prefix.prefix,
   ...           avoiding_with_prefix.patterns,
   ...        )
   ...        # safe will be the index of the prefix in which we can remove upto without
   ...        # affecting the avoidance conditions
   ...        m = max(len(p) for p in patterns) if patterns else 1
   ...        safe = max(0, len(prefix) - m + 1)
   ...        for i in range(safe, len(prefix)):
   ...           end = prefix[i:]
   ...           if any(end == patt[: len(end)] for patt in patterns):
   ...                 break
   ...           safe = i + 1
   ...        return safe
   ...
   ...     def formal_step(self):
   ...        return "removing redundant prefix"
   ...
   ...     def backward_map(self, avoiding_with_prefix, words, children=None):
   ...        """
   ...        The forward direction of the underlying bijection used for object
   ...        generation and sampling.
   ...        """
   ...        assert len(words) == 2
   ...        assert isinstance(words[0], Word)
   ...        assert isinstance(words[1], Word)
   ...        if children is None:
   ...           children = self.decomposition_function(avoiding_with_prefix)
   ...           assert children is not None
   ...        yield Word(words[0] + words[1])
   ...
   ...     def forward_map(self, comb_class, word, children=None):
   ...        """
   ...        The backward direction of the underlying bijection used for object
   ...        generation and sampling.
   ...        """
   ...        assert isinstance(word, Word)
   ...        if children is None:
   ...           children = self.decomposition_function(comb_class)
   ...           assert children is not None
   ...        return Word(children[0].prefix), Word(word[len(children[0].prefix) :])
   ...
   ...     @classmethod
   ...     def from_dict(cls, d):
   ...        return cls()
   ...
   ...     def __str__(self):
   ...        return self.formal_step()
   ...
   ...     def __repr__(self):
   ...        return self.__class__.__name__ + "()"

With these three strategies we are now ready to perform combinatorial
exploration using the following pack.

.. code:: python

   >>> pack = StrategyPack(initial_strats=[RemoveFrontOfPrefix()],
   ...                     inferral_strats=[],
   ...                     expansion_strats=[[ExpansionStrategy()]],
   ...                     ver_strats=[AtomStrategy()],
   ...                     name=("Finding specification for words avoiding "
   ...                           "consecutive patterns."))

First we need to create the combinatorial class we want to count. For
example, consider the words over the alphabet ``{a, b}`` that avoid
``ababa`` and ``babb``. This class can be created using our initialise
function.

.. code:: python

   >>> prefix = ''
   >>> patterns = ['ababa', 'babb']
   >>> alphabet = ['a', 'b']
   >>> start_class = AvoidingWithPrefix(prefix, patterns, alphabet)

We can then initialise our ``CombinatorialSpecificationSearcher``, and
use the ``auto_search`` function which will return a
``CombinatorialSpecification`` assuming one is found (which in this case always
will).

.. code:: python

   >>> from comb_spec_searcher import CombinatorialSpecificationSearcher

   >>> searcher = CombinatorialSpecificationSearcher(start_class, pack)
   >>> spec = searcher.auto_search()
   >>> # spec.show() will display the specification in your browser

Now that we have a ``CombinatorialSpecification``, the obvious
thing we want to do is find the generating function for the class that
counts the number of objects of each size. This can be done by using the
``get_genf`` methods on ``CombinatorialSpecification``.

Finally, in order to get initial terms, you will also need to implement
the ``objects_of_size`` function which should yield all of the objects
of a given size in the class.

.. code:: python

   >>> from itertools import product

   >>> def objects_of_size(self, size):
   ...     """Yield the words of given size that start with prefix and avoid the
   ...     patterns. If just_prefix, then only yield that word."""
   ...     def possible_words():
   ...         """Yield all words of given size over the alphabet with prefix"""
   ...         if len(self.prefix) > size:
   ...            return
   ...         for letters in product(self.alphabet,
   ...                                 repeat=size - len(self.prefix)):
   ...             yield Word(self.prefix + "".join(a for a in letters))
   ...
   ...     if self.just_prefix:
   ...         if size == len(self.prefix) and not self.is_empty():
   ...             yield Word(self.prefix)
   ...         return
   ...     for word in possible_words():
   ...         if all(patt not in word for patt in self.patterns):
   ...             yield word
   >>> AvoidingWithPrefix.objects_of_size = objects_of_size

With these in place if we then call the ``get_genf`` function

.. code:: python

   >>> spec.get_genf()
   -(x + 1)*(x**2 - x + 1)**2*(x**2 + x + 1)/(x**6 + x**3 - x**2 + 2*x - 1)

we see that the the generating function is
``F = -(x**7 + x**5 + x**4 + x**3 + x**2 + 1)/(x**6 + x**3 - x**2 + 2*x - 1)``.

Moreover, we can get directly the number of objects by size with the method
`count_objects_of_size`.

.. code:: python

   >>> [spec.count_objects_of_size(i) for i in range(11)]
   [1, 2, 4, 8, 15, 27, 48, 87, 157, 283, 511]

You can now try this yourself using the file ``example.py``, which can
count any set of words avoiding consecutive patterns.

Now we will demonstrate how a bijection can be found between classes.
We will first need a couple of imports.

.. code:: python

   >>> from comb_spec_searcher.bijection import ParallelSpecFinder
   >>> from comb_spec_searcher.isomorphism import Bijection

We start by defining our two classes that we wish to find a bijection between.

.. code:: python

   >>> prefix1 = ''
   >>> patterns1 = ["00"]
   >>> alphabet1 = ['0', '1']
   >>> class1 = AvoidingWithPrefix(prefix1, patterns1, alphabet1)
   >>> prefix2 = ''
   >>> patterns2 = ["bb"]
   >>> alphabet2 = ['a', 'b']
   >>> class2 = AvoidingWithPrefix(prefix2, patterns2, alphabet2)

To find a bijection we expand the universe given a pack for both classes
and try to construct specifications that are parallel.

.. code:: python

   >>> searcher1 = CombinatorialSpecificationSearcher(class1, pack)
   >>> searcher2 = CombinatorialSpecificationSearcher(class2, pack)

We get two parallel specs if successful, ``None`` otherwise.

.. code:: python

   >>> specs = ParallelSpecFinder(searcher1, searcher2).find()

We then construct the bijection from the parallel specifications.

.. code:: python

   >>> bijection = Bijection.construct(*specs)

We can use the `Bijection` object to map (either way) sampled objects
from the sepcifications.

.. code:: python

   >>> for i in range(10):
   ...     for w in bijection.domain.generate_objects_of_size(i):
   ...         assert w == bijection.inverse_map(bijection.map(w))
   ...     for w in bijection.codomain.generate_objects_of_size(i):
   ...         assert w == bijection.map(bijection.inverse_map(w))
   ...

Whether we find a bijection or not (when one exists) is highly
dependent on the packs chosen.

Citing
######

If you found this library helpful with your research and would like to cite us,
you can use the following `BibTeX`_ or go to `Zenodo`_ for alternative formats.

.. _BibTex: https://zenodo.org/record/4944021/export/hx#.YMcpIC2l30o

.. _Zenodo: https://doi.org/10.5281/zenodo.4944020


# Taipy Doc

## License
Copyright 2021-2024 Avaiga Private Limited

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

## Usage
- [License](#license)
- [Usage](#usage)
- [Taipy Doc](#what-is-taipy-doc)
- [Installation](#installation)
- [Contributing](#contributing)
- [Code of conduct](#code-of-conduct)
- [Directory Structure](#directory-structure)

## What is Taipy Doc

Taipy is a Python library for creating Business Applications. More information on our
[website](https://www.taipy.io). Taipy is split into multiple repositories to let users
install the minimum they need.

[Taipy Doc](https://github.com/Avaiga/taipy-doc) has the responsibility to hold and build the full Taipy
documentation set. The `taipy-doc` repository uses [MkDocs](https://www.mkdocs.org/) to generate its entire
content.

## Installation

Want to install _Taipy Doc_? Check out our [`INSTALLATION.md`](INSTALLATION.md) file.

## Contributing

Want to help build _Taipy Doc_? Check out our [`CONTRIBUTING.md`](CONTRIBUTING.md) file.

## Code of conduct

Want to be part of the _Taipy Doc_ community? Check out our [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) file.

## Directory Structure

- `docs`: Documentation content (mainly as Markdown files) used by mkdocs to generate the Taipy documentation set.
- `tools`: Mkdocs and other tools used to build the Taipy documentation set.
- `CODE_OF_CONDUCT.md`: Code of conduct for members and contributors of _taipy-doc_.
- `CONTRIBUTING.md`: Instructions to contribute to _taipy-doc_.
- `INSTALLATION.md`: Instructions to install _taipy-doc_.
- `LICENSE`: The Apache 2.0 License.
- `Pipfile`: File used by the Pipenv virtual environment to manage project dependencies.
- `README.md`: Current file.
- `mkdocs.yml_template`: The template used by mkdocs and other tools to build the Taipy documentation set.


![PyTest](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/build_and_test.yml/badge.svg)
![Bandit](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/bandit.yml/badge.svg)
![CodeQL](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/codeql.yml/badge.svg)
![PyLint](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/pylint.yml/badge.svg)

![License Scanner](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/license_scanner.yml/badge.svg)
![Docker](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/build_docker_image.yml/badge.svg)
![Docker Compose](https://github.com/CounterpartyXCP/counterparty-core/actions/workflows/test_compose.yml/badge.svg)

**Counterparty Core** is the reference implementation of the [Counterparty Protocol](https://counterparty.io), an extension to the Bitcoin protocol which implements a number of features that Bitcoin itself does not offer. These include token issuance, a fully decentralized and trustless asset exchange, contracts for difference, native oracles and trustless gaming. Counterparty works by ‘writing in the margins’ of Bitcoin transactions, and all Counterparty transactions are Bitcoin transactions with additional data that the Counterparty software can read and interpret.

See the **[official project documentation](http://docs.counterparty.io)** for more information, and for instructions on installing and running the Counterparty software.


<img width="100%" alt="SEACrowd Logo" src="https://github.com/SEACrowd/.github/blob/main/profile/assets/seacrowd-email-banner-without-logo.png?raw=true">

# Welcome to SEACrowd!

Southeast Asia is home to more than 1,000 native languages. Nevertheless, Southeast Asian NLP, vision-language, and speech processing is underrepresented in the research community, and one of the reasons is the lack of access to public datasets ([Aji et al., 2022](https://aclanthology.org/2022.acl-long.500/)). To address this issue, we initiate **SEACrowd**, a joint collaboration to collect NLP datasets for Southeast Asian languages. Help us collect and centralize Southeast Asian datasets, and be a co-author of our upcoming paper.

## How to Use

### Library Installation

Find seacrowd library (v0.1.3) at https://pypi.org/project/seacrowd/. (See our release notes [here](https://github.com/SEACrowd/seacrowd-datahub/releases/tag/0.1.3).)

To install SEACrowd, install the `seacrowd` package in your python environment via `pip`.

```
pip install seacrowd
```

### Using `seacrowd` library

To use the `seacrowd` package, simply import it in your code:
```
import seacrowd as sc
````

### List & Load Dataset
SEACrowd provides functions for listing and loading all datasets that are implemented in NusaCrowd
```
# List all datasets
dset_names = sc.list_datasets()

# List all datasets with their config names
dset_configs_dict = sc.list_datasets(with_config=True)

# Load a single dataset based on the dataset name
khpos_dset = sc.load_dataset("khpos", schema="seacrowd")

# Load multiple datasets based on the dataset names
dsets = sc.load_datasets(["thai_sum", "vsolscsum"], schema="seacrowd_t2t")
```

### List & Load Benchmark
In addition to dataset-related functions, SEACrowd provides additional functions for listing and loading some SEA benchmarks.
```
# List all benchmarks
benchmark_names = sc.list_benchmarks()

# Load all datasets in a benchmark
seacrowd_vl_dsets = sc.load_benchmark("SEACrowd-VL")
```

### Load Metadata
Aside from loading datasets and benchmarks, `seacrowd` also supports loading the metadata (e.g., license, description, citation,  etc.) of the dataloaders.
```
# Load metadata of a dataloader
khpos_meta = sc.for_dataset("khpos")

# Load metadata of multiple dataloaders
meta_dsets = sc.for_datasets(["thai_sum", "vsolscsum"])

# Load metadata of a config name
nusaparagraph_meta = sc.for_config_name("nusaparagraph_emot_jav_seacrowd_text")

# Load metadata of multiple config names
meta_dsets = sc.for_config_names(["sentiment_nathasa_review_seacrowd_text", "indonli_seacrowd_pairs"])
```

We can also load the dataloader from the metadata if we want.

```
# Load dataset from metadata
khpos_dset = khpos_meta.load_dataset()
```

> For the functions' sample outputs, check our [release notes](https://github.com/SEACrowd/seacrowd-datahub/releases/tag/0.1.3).

## How to Contribute

Check out our [CONTRIBUTING.md](https://github.com/SEACrowd/seacrowd-datahub/blob/master/CONTRIBUTING.md) for a gentle introduction to contributing in SEACrowd. Jump straight ahead to [DATALOADER.md](https://github.com/SEACrowd/seacrowd-datahub/blob/master/DATALOADER.md) if you have decided to contribute by implementing dataloaders for our Data Hub!

## Citation

If you are using any resources from SEACrowd, including datasheets, dataloaders, code, etc., please cite [the following publication](https://arxiv.org/pdf/2406.10118):

```
@article{lovenia2024seacrowd,
      title={SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages},
      author={Holy Lovenia and Rahmad Mahendra and Salsabil Maulana Akbar and Lester James V. Miranda and Jennifer Santoso and Elyanah Aco and Akhdan Fadhilah and Jonibek Mansurov and Joseph Marvin Imperial and Onno P. Kampman and Joel Ruben Antony Moniz and Muhammad Ravi Shulthan Habibi and Frederikus Hudi and Railey Montalan and Ryan Ignatius and Joanito Agili Lopo and William Nixon and Börje F. Karlsson and James Jaya and Ryandito Diandaru and Yuze Gao and Patrick Amadeus and Bin Wang and Jan Christian Blaise Cruz and Chenxi Whitehouse and Ivan Halim Parmonangan and Maria Khelli and Wenyu Zhang and Lucky Susanto and Reynard Adha Ryanda and Sonny Lazuardi Hermawan and Dan John Velasco and Muhammad Dehan Al Kautsar and Willy Fitra Hendria and Yasmin Moslem and Noah Flynn and Muhammad Farid Adilazuarda and Haochen Li and Johanes Lee and R. Damanhuri and Shuo Sun and Muhammad Reza Qorib and Amirbek Djanibekov and Wei Qi Leong and Quyet V. Do and Niklas Muennighoff and Tanrada Pansuwan and Ilham Firdausi Putra and Yan Xu and Ngee Chia Tai and Ayu Purwarianti and Sebastian Ruder and William Tjhi and Peerat Limkonchotiwat and Alham Fikri Aji and Sedrick Keh and Genta Indra Winata and Ruochen Zhang and Fajri Koto and Zheng-Xin Yong and Samuel Cahyawijaya},
      year={2024},
      eprint={2406.10118},
      journal={arXiv preprint arXiv: 2406.10118}
}
```

## Acknowledgements

Our initiative is heavily inspired by [NusaCrowd](https://github.com/IndoNLP/nusa-crowd/tree/master/nusacrowd) which provides open access data to 100+ Indonesian NLP corpora. You can check NusaCrowd paper (published in ACL Findings 2023) on the following [link](https://aclanthology.org/2023.findings-acl.868/).


# ⚡ LitGPT

**Pretrain, finetune, evaluate, and deploy 20+ LLMs on your own data**

Uses the latest state-of-the-art techniques:

✅ flash attention    ✅ fp4/8/16/32        ✅ LoRA, QLoRA, Adapter
✅ FSDP               ✅ 1-1000+ GPUs/TPUs  ✅ 20+ LLMs

---

![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pytorch-lightning)
![cpu-tests](https://github.com/lightning-AI/lit-stablelm/actions/workflows/cpu-tests.yml/badge.svg) [![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Lightning-AI/lit-stablelm/blob/master/LICENSE) [![Discord](https://img.shields.io/discord/1077906959069626439)](https://discord.gg/VptPCZkGNa)

  Lightning AI •
  Models •
  Quick start •
  Inference •
  Finetune •
  Pretrain •
    Deploy •
  Features •
  Training recipes (YAML)

# Finetune, pretrain and deploy LLMs Lightning fast ⚡⚡
LitGPT is a command-line tool designed to easily [finetune](#finetune-an-llm), [pretrain](#pretrain-an-llm), [evaluate](#use-an-llm), and [deploy](#deploy-an-llm) [20+ LLMs](#choose-from-20-llms) **on your own data**. It features highly-optimized [training recipes](#training-recipes) for the world's most powerful open-source large language models (LLMs).

We reimplemented all model architectures and training recipes from scratch for 4 reasons:

1. Remove all abstraction layers and have single file implementations.
2. Guarantee Apache 2.0 compliance to enable enterprise use without limits.
3. Optimized each model's architectural detail to maximize performance, reduce costs, and speed up training.
4. Highly-optimized [recipe configs](#training-recipes) we have tested at enterprise scale.

---

# Choose from 20+ LLMs
LitGPT has 🤯 **custom, from-scratch implementations** of [20+ LLMs](tutorials/download_model_weights.md) without layers of abstraction:

| Model | Model size | Author | Reference |
|----|----|----|----|
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/)                                                                      |
| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/)                                                                         |
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
| ... | ... | ... | ...   |

  See full list of 20+ LLMs

#### All models

| Model | Model size | Author | Reference |
|----|----|----|----|
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma)                                                                 |
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950)                                                                   |
| Danube2 | 1.8B | H2O.ai | [H2O.ai](https://h2o.ai/platform/danube-1-8b/)                                                                                |
| Dolly | 3B, 7B, 12B | Databricks | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm)      |
| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae)                                                                                |
| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models)                 |
| Function Calling Llama 2 | 7B | Trelis | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2)                                  |
| Gemma | 2B, 7B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf)                                       |
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288)                                                                           |
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3)                                                                                |
| LongChat | 7B, 13B | LMSYS | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/)                                                                       |
| MicroLlama | 300M | Ken Wang | [MicroLlama repo](https://github.com/keeeeenw/MicroLlama)
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/)                                                                     |
| Mistral | 7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/)                                                                        |
| Nous-Hermes | 7B, 13B, 70B | NousResearch | [Org page](https://huggingface.co/NousResearch)                                                                          |
| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama)                                                         |
| Phi | 1.3B, 2.7B | Microsoft Research  | [Li et al. 2023](https://arxiv.org/abs/2309.05463)                                                                          |
| Platypus | 7B, 13B, 70B |  Lee et al. | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317)                                                               |
| Pythia | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | EleutherAI | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373)                                            |
| RedPajama-INCITE | 3B, 7B | Together | [Together 2023](https://together.ai/blog/redpajama-models-v1)                                                                 |
| StableCode | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding)                                                  |
| StableLM  | 3B, 7B | Stability AI | [Stability AI 2023](https://github.com/Stability-AI/StableLM)                                                                    |
| StableLM Zephyr | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding)                                             |
| TinyLlama | 1.1B | Zhang et al. | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama)                                                                         |
| Vicuna | 7B, 13B, 33B | LMSYS | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/)                                                                          |

**Tip**: You can list all available models by running the `litgpt download list` command.

## Install LitGPT

Install LitGPT with all dependencies (including CLI, quantization, tokenizers for all models, etc.):

```bash
pip install 'litgpt[all]'
```

  Advanced install options

Install from source:

```bash
git clone https://github.com/Lightning-AI/litgpt
cd litgpt
pip install -e '.[all]'
```

---

# Quick start
After installing LitGPT, select the model and action you want to take on that model (finetune, pretrain, evaluate, deploy, etc...):

```bash
# ligpt [action] [model]
litgpt  download  meta-llama/Meta-Llama-3-8B-Instruct
litgpt  chat      meta-llama/Meta-Llama-3-8B-Instruct
litgpt  finetune  meta-llama/Meta-Llama-3-8B-Instruct
litgpt  pretrain  meta-llama/Meta-Llama-3-8B-Instruct
litgpt  serve     meta-llama/Meta-Llama-3-8B-Instruct
```

###  Use an LLM for inference
Use LLMs for inference to test its chatting capabilities, run evaluations, or extract embeddings, etc.
Here's an example showing how to use the Phi-2 LLM.

```bash
# 1) List all available models in litgpt
litgpt download list

# 2) Download a pretrained model
litgpt download microsoft/phi-2

# 3) Chat with the model
litgpt chat microsoft/phi-2

>> Prompt: What do Llamas eat?
```

The download of certain models requires an additional access token. You can read more about this in the [download](tutorials/download_model_weights.md#specific-models-and-access-tokens) documentation.
For more information on the different inference options, refer to the [inference](tutorials/inference.md) tutorial.

### Finetune an LLM
[Finetune](tutorials/finetune.md) a model to specialize it on your own custom dataset:

```bash
# 1) Download a pretrained model
litgpt download microsoft/phi-2

# 2) Finetune the model
curl -L https://huggingface.co/datasets/ksaw008/finance_alpaca/resolve/main/finance_alpaca.json -o my_custom_dataset.json

litgpt finetune microsoft/phi-2 \
  --data JSON \
  --data.json_path my_custom_dataset.json \
  --data.val_split_fraction 0.1 \
  --out_dir out/custom-model

# 3) Chat with the model
litgpt chat out/custom-model/final
```

### Pretrain an LLM
Train an LLM from scratch on your own data via pretraining:

```bash
mkdir -p custom_texts
curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt

# 1) Download a tokenizer
litgpt download EleutherAI/pythia-160m \
  --tokenizer_only True

# 2) Pretrain the model
litgpt pretrain EleutherAI/pythia-160m \
  --tokenizer_dir EleutherAI/pythia-160m \
  --data TextFiles \
  --data.train_data_path "custom_texts/" \
  --train.max_tokens 10_000_000 \
  --out_dir out/custom-model

# 3) Chat with the model
litgpt chat out/custom-model/final
```

### Continue pretraining an LLM
This is another way of finetuning that specializes an already pretrained model by training on custom data:

```bash
mkdir -p custom_texts
curl https://www.gutenberg.org/cache/epub/24440/pg24440.txt --output custom_texts/book1.txt
curl https://www.gutenberg.org/cache/epub/26393/pg26393.txt --output custom_texts/book2.txt

# 1) Download a pretrained model
litgpt download EleutherAI/pythia-160m

# 2) Continue pretraining the model
litgpt pretrain EleutherAI/pythia-160m \
  --tokenizer_dir EleutherAI/pythia-160m \
  --initial_checkpoint_dir EleutherAI/pythia-160m \
  --data TextFiles \
  --data.train_data_path "custom_texts/" \
  --train.max_tokens 10_000_000 \
  --out_dir out/custom-model

# 3) Chat with the model
litgpt chat out/custom-model/final
```

### Deploy an LLM
Once you're ready to deploy a finetuned LLM, run this command:

```bash
# locate the checkpoint to your finetuned or pretrained model and call the `serve` command:
litgpt serve microsoft/phi-2

# Alternative: if you haven't finetuned, download any checkpoint to deploy it:
litgpt download microsoft/phi-2
litgpt serve microsoft/phi-2
```

Test the server in a separate terminal and integrate the model API into your AI product:
```python
# 3) Use the server (in a separate Python session)
import requests, json
response = requests.post(
    "http://127.0.0.1:8000/predict",
    json={"prompt": "Fix typos in the following sentence: Exampel input"}
)
print(response.json()["output"])
```

> [!NOTE]
> **[Read the full docs](tutorials/0_to_litgpt.md)**.

----

# State-of-the-art features
✅  State-of-the-art optimizations: Flash Attention v2, multi-GPU support via fully-sharded data parallelism, [optional CPU offloading](tutorials/oom.md#do-sharding-across-multiple-gpus), and [TPU and XLA support](extensions/xla).

✅  [Pretrain](tutorials/pretrain.md), [finetune](tutorials/finetune.md), and [deploy](tutorials/inference.md)

✅  Reduce compute requirements with low-precision settings: FP16, BF16, and FP16/FP32 mixed.

✅  Lower memory requirements with [quantization](tutorials/quantize.md): 4-bit floats, 8-bit integers, and double quantization.

✅  [Configuration files](config_hub) for great out-of-the-box performance.

✅  Parameter-efficient finetuning: [LoRA](tutorials/finetune_lora.md), [QLoRA](tutorials/finetune_lora.md), [Adapter](tutorials/finetune_adapter.md), and [Adapter v2](tutorials/finetune_adapter.md).

✅  [Exporting](tutorials/convert_lit_models.md) to other popular model weight formats.

✅  Many popular datasets for [pretraining](tutorials/pretrain.md) and [finetuning](tutorials/prepare_dataset.md), and [support for custom datasets](tutorials/prepare_dataset.md#preparing-custom-datasets-for-instruction-finetuning).

✅  Readable and easy-to-modify code to experiment with the latest research ideas.

---

# Training recipes

LitGPT comes with validated recipes (YAML configs) to train models under different conditions.  We've generated these recipes based on the parameters we found to perform the best for different training conditions.

Browse all training recipes [here](config_hub).

### Example

```bash
litgpt finetune \
  --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml
```

### What is a config
Configs let you customize training for all granular parameters like:

```yaml
# The path to the base model's checkpoint directory to load for finetuning. (type: <class 'Path'>, default: checkpoints/stabilityai/stablelm-base-alpha-3b)
checkpoint_dir: checkpoints/meta-llama/Llama-2-7b-hf

# Directory in which to save checkpoints and logs. (type: <class 'Path'>, default: out/lora)
out_dir: out/finetune/qlora-llama2-7b

# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null)
precision: bf16-true

...
```

  Example: LoRA finetuning config

```yaml
# The path to the base model's checkpoint directory to load for finetuning. (type: <class 'Path'>, default: checkpoints/stabilityai/stablelm-base-alpha-3b)
checkpoint_dir: checkpoints/meta-llama/Llama-2-7b-hf

# Directory in which to save checkpoints and logs. (type: <class 'Path'>, default: out/lora)
out_dir: out/finetune/qlora-llama2-7b

# The precision to use for finetuning. Possible choices: "bf16-true", "bf16-mixed", "32-true". (type: Optional[str], default: null)
precision: bf16-true

# If set, quantize the model with this algorithm. See ``tutorials/quantize.md`` for more information. (type: Optional[Literal['nf4', 'nf4-dq', 'fp4', 'fp4-dq', 'int8-training']], default: null)
quantize: bnb.nf4

# How many devices/GPUs to use. (type: Union[int, str], default: 1)
devices: 1

# The LoRA rank. (type: int, default: 8)
lora_r: 32

# The LoRA alpha. (type: int, default: 16)
lora_alpha: 16

# The LoRA dropout value. (type: float, default: 0.05)
lora_dropout: 0.05

# Whether to apply LoRA to the query weights in attention. (type: bool, default: True)
lora_query: true

# Whether to apply LoRA to the key weights in attention. (type: bool, default: False)
lora_key: false

# Whether to apply LoRA to the value weights in attention. (type: bool, default: True)
lora_value: true

# Whether to apply LoRA to the output projection in the attention block. (type: bool, default: False)
lora_projection: false

# Whether to apply LoRA to the weights of the MLP in the attention block. (type: bool, default: False)
lora_mlp: false

# Whether to apply LoRA to output head in GPT. (type: bool, default: False)
lora_head: false

# Data-related arguments. If not provided, the default is ``litgpt.data.Alpaca``.
data:
  class_path: litgpt.data.Alpaca2k
  init_args:
    mask_prompt: false
    val_split_fraction: 0.05
    prompt_style: alpaca
    ignore_index: -100
    seed: 42
    num_workers: 4
    download_dir: data/alpaca2k

# Training-related arguments. See ``litgpt.args.TrainArgs`` for details
train:

  # Number of optimizer steps between saving checkpoints (type: Optional[int], default: 1000)
  save_interval: 200

  # Number of iterations between logging calls (type: int, default: 1)
  log_interval: 1

  # Number of samples between optimizer steps across data-parallel ranks (type: int, default: 128)
  global_batch_size: 8

  # Number of samples per data-parallel rank (type: int, default: 4)
  micro_batch_size: 2

  # Number of iterations with learning rate warmup active (type: int, default: 100)
  lr_warmup_steps: 10

  # Number of epochs to train on (type: Optional[int], default: 5)
  epochs: 4

  # Total number of tokens to train on (type: Optional[int], default: null)
  max_tokens:

  # Limits the number of optimizer steps to run (type: Optional[int], default: null)
  max_steps:

  # Limits the length of samples (type: Optional[int], default: null)
  max_seq_length: 512

  # Whether to tie the embedding weights with the language modeling head weights (type: Optional[bool], default: null)
  tie_embeddings:

  #   (type: float, default: 0.0003)
  learning_rate: 0.0002

  #   (type: float, default: 0.02)
  weight_decay: 0.0

  #   (type: float, default: 0.9)
  beta1: 0.9

  #   (type: float, default: 0.95)
  beta2: 0.95

  #   (type: Optional[float], default: null)
  max_norm:

  #   (type: float, default: 6e-05)
  min_lr: 6.0e-05

# Evaluation-related arguments. See ``litgpt.args.EvalArgs`` for details
eval:

  # Number of optimizer steps between evaluation calls (type: int, default: 100)
  interval: 100

  # Number of tokens to generate (type: Optional[int], default: 100)
  max_new_tokens: 100

  # Number of iterations (type: int, default: 100)
  max_iters: 100

# The name of the logger to send metrics to. (type: Literal['wandb', 'tensorboard', 'csv'], default: csv)
logger_name: csv

# The random seed to use for reproducibility. (type: int, default: 1337)
seed: 1337
```

### Override config params via CLI
Override any parameter in the CLI:

```bash
litgpt finetune \
  --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/llama-2-7b/lora.yaml \
  --lora_r 4
```

# Community

## Get involved!

We appreciate your feedback and contributions. If you have feature requests, questions, or want to contribute code or config files, please don't hesitate to use the [GitHub Issue](https://github.com/Lightning-AI/litgpt/issues) tracker.

We welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.

> [!TIP]
> Unsure about contributing? Check out our [How to Contribute to LitGPT](https://lightning.ai/pages/community/tutorial/how-to-contribute-to-litgpt/) guide.

If you have general questions about building with LitGPT, please [join our Discord](https://discord.gg/VptPCZkGNa).

## Tutorials, how-to guides, and docs

> [!NOTE]
> We recommend starting with the **[Zero to LitGPT: Getting Started with Pretraining, Finetuning, and Using LLMs](tutorials/0_to_litgpt.md)** if you are looking to get started with using LitGPT.

Tutorials and in-depth feature documentation can be found below:

-  Finetuning, incl. LoRA, QLoRA, and Adapters ([tutorials/finetune.md](tutorials/finetune.md))
-  Pretraining ([tutorials/pretrain.md](tutorials/pretrain.md))
-  Model evaluation ([tutorials/evaluation.md](tutorials/evaluation.md))
-  Supported and custom datasets ([tutorials/prepare_dataset.md](tutorials/prepare_dataset.md))
-  Quantization ([tutorials/quantize.md](tutorials/quantize.md))
-  Tips for dealing with out-of-memory (OOM) errors ([tutorials/oom.md](tutorials/oom.md))

## XLA

Lightning AI has partnered with Google to add first-class support for [Cloud TPUs](https://cloud.google.com/tpu) in [Lightning's frameworks](https://github.com/Lightning-AI/lightning) and LitGPT,
helping democratize AI for millions of developers and researchers worldwide.

Using TPUs with Lightning is as straightforward as changing one line of code.

We provide scripts fully optimized for TPUs in the [XLA directory](extensions/xla).

## Acknowledgements

This implementation extends on [Lit-LLaMA](https://github.com/lightning-AI/lit-llama) and [nanoGPT](https://github.com/karpathy/nanoGPT), and it's **powered by [Lightning Fabric](https://lightning.ai/docs/fabric/stable/) ⚡**.

- [@karpathy](https://github.com/karpathy) for [nanoGPT](https://github.com/karpathy/nanoGPT)
- [@EleutherAI](https://github.com/EleutherAI) for [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) and the [Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
- [@TimDettmers](https://github.com/TimDettmers) for [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
- [@Microsoft](https://github.com/microsoft) for [LoRA](https://github.com/microsoft/LoRA)
- [@tridao](https://github.com/tridao) for [Flash Attention 2](https://github.com/Dao-AILab/flash-attention)

## Community showcase

Check out the projects below that use and build on LitGPT. If you have a project you'd like to add to this section, please don't hesitate to open a pull request.

📊 **SAMBA: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling**

The [Samba](https://github.com/microsoft/Samba) project by researchers at Microsoft is built on top of the LitGPT code base and combines state space models with sliding window attention, which outperforms pure state space models.

**🏆 NeurIPS 2023 Large Language Model Efficiency Challenge: 1 LLM + 1 GPU + 1 Day**

The LitGPT repository was the official starter kit for the [NeurIPS 2023 LLM Efficiency Challenge](https://llm-efficiency-challenge.github.io), which is a competition focused on finetuning an existing non-instruction tuned LLM for 24 hours on a single GPU.

**🦙 TinyLlama: An Open-Source Small Language Model**

LitGPT powered the [TinyLlama project](https://github.com/jzhang38/TinyLlama) and [TinyLlama: An Open-Source Small Language Model](https://arxiv.org/abs/2401.02385) research paper.

**🍪 MicroLlama: MicroLlama-300M**

[MicroLlama](https://github.com/keeeeenw/MicroLlama) is a 300M Llama model pretrained on 50B tokens powered by TinyLlama and LitGPT.

**🔬 Pre-training Small Base LMs with Fewer Tokens**

The research paper ["Pre-training Small Base LMs with Fewer Tokens"](https://arxiv.org/abs/2404.08634), which utilizes LitGPT, develops smaller base language models by inheriting a few transformer blocks from larger models and training on a tiny fraction of the data used by the larger models. It demonstrates that these smaller models can perform comparably to larger models despite using significantly less training data and resources.

## Citation

If you use LitGPT in your research, please cite the following work:

```bibtex
@misc{litgpt-2023,
  author       = {Lightning AI},
  title        = {LitGPT},
  howpublished = {\url{https://github.com/Lightning-AI/litgpt}},
  year         = {2023},
}
```

## License

LitGPT is released under the [Apache 2.0](https://github.com/Lightning-AI/litgpt/blob/main/LICENSE) license.


# Isso – a commenting server similar to Disqus

Isso – *Ich schrei sonst* – is a lightweight commenting server written in
Python and JavaScript. It aims to be a drop-in replacement for
[Disqus](http://disqus.com).

## Features

- **Comments written in Markdown**
  Users can edit or delete own comments (within 15 minutes by default).
  Comments in moderation queue are not publicly visible before activation.
- **SQLite backend**
  *Because comments are not Big Data.*
- **Disqus & WordPress Import**
  You can migrate your Disqus/WordPress comments without any hassle.
- **Configurable JS client**
  Embed a single JS file, 65kB (20kB gzipped) and you are done.

See **[isso-comments.de](https://isso-comments.de/)** for a **live demo**, more
details and [documentation](https://isso-comments.de/docs/).

## Screenshot

![Isso in Action](https://user-images.githubusercontent.com/10212877/167268553-3f30b448-25ff-4850-afef-df2f2e599c93.png)

## Getting started

### Requirements

- Python 3.8+ (+ devel headers)
- SQLite 3.3.8 or later
- a working C compiler

Install Isso from [PyPi](https://pypi.python.org/pypi/isso/):

```console
pip install isso
```

Then, follow the [Quickstart](https://isso-comments.de/docs/guides/quickstart/) guide.

If you're stuck, follow the [Install guide](https://isso-comments.de/docs/reference/installation/),
see [Troubleshooting](https://isso-comments.de/docs/guides/troubleshooting/) and browse
the [the full documentation](https://isso-comments.de/docs/).

## Docker

> [!NOTE]
> The Docker image tagging scheme for stable releases was changed from `:latest`
> to `:release` as of March 2024
> ([#970](https://github.com/isso-comments/isso/pull/970), [#1012](https://github.com/isso-comments/isso/issues/1012))

A [Docker image](https://github.com/isso-comments/isso/pkgs/container/isso) with
the latest stable release is provided at `ghcr.io/isso-comments/isso:release`,
while `isso:latest` is rebuilt on every push to the `master` branch. See
[Using Docker](https://isso-comments.de/docs/reference/installation/#using-docker).

The maintainers recommend pinning the image to a
[release tag](https://github.com/isso-comments/isso/pkgs/container/isso), e.g.
`isso:0.13.0`.

## Contributing

- Pull requests are very much welcome! These might be
  [good first issues](https://github.com/isso-comments/isso/labels/good-first-issue)
- See [Ways to Contribute](https://isso-comments.de/docs/contributing/)
- [Translate](https://isso-comments.de/docs/contributing/#translations)

### Development

<!-- TODO also mention "Development & Testing" section once new docs uploaded -->
Refer to the docs for
[Installing from Source](https://isso-comments.de/docs/reference/installation/#install-from-source).

### Help

- Join `#isso` via IRC on [Libera.Chat](https://libera.chat/)
- Ask a question on [GitHub Discussions](https://github.com/isso-comments/isso/discussions).

## License

MIT, see [LICENSE](LICENSE).


<img src="https://raw.githubusercontent.com/janelia-cellmap/dacapo/main/docs/source/_static/CellMapLogo.png" alt="CellMap logo" width="85%">

# DaCapo ![DaCapo](https://raw.githubusercontent.com/janelia-cellmap/dacapo/main/docs/source/_static/icon_dacapo.png) ![GitHub Org's stars](https://img.shields.io/github/stars/Janelia-cellmap/dacapo)

![PyPI - Downloads](https://img.shields.io/pypi/dm/dacapo-ml)
[![Documentation Status](https://readthedocs.org/projects/dacapo/badge/?version=stable)](https://dacapo.readthedocs.io/en/stable/?badge=stable)
![Github Created At](https://img.shields.io/github/created-at/funkelab/dacapo)
![GitHub License](https://img.shields.io/github/license/janelia-cellmap/dacapo)
![Python Version from PEP 621 TOML](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fjanelia-cellmap%2Fdacapo%2Fmain%2Fpyproject.toml)

[![tests](https://github.com/janelia-cellmap/dacapo/actions/workflows/tests.yaml/badge.svg)](https://github.com/janelia-cellmap/dacapo/actions/workflows/tests.yaml)
[![black](https://github.com/janelia-cellmap/dacapo/actions/workflows/black.yaml/badge.svg)](https://github.com/janelia-cellmap/dacapo/actions/workflows/black.yaml)
[![mypy](https://github.com/janelia-cellmap/dacapo/actions/workflows/mypy.yaml/badge.svg)](https://github.com/janelia-cellmap/dacapo/actions/workflows/mypy.yaml)
[![docs](https://github.com/janelia-cellmap/dacapo/actions/workflows/docs.yaml/badge.svg)](https://janelia-cellmap.github.io/dacapo/)
[![codecov](https://codecov.io/gh/janelia-cellmap/dacapo/branch/main/graph/badge.svg)](https://codecov.io/gh/janelia-cellmap/dacapo)

A framework for easy application of established machine learning techniques on large, multi-dimensional images.

`dacapo` allows you to configure machine learning jobs as combinations of
[DataSplits](http://docs/api.html#datasplits),
[Architectures](http://docs/api.html#architectures),
[Tasks](http://docs/api.html#tasks),
[Trainers](http://docs/api.html#trainers),
on arbitrarily large volumes of
multi-dimensional images. `dacapo` is not tied to a particular learning
framework, but currently only supports [`torch`](https://pytorch.org/) with
plans to support [`tensorflow`](https://www.tensorflow.org/).

## Installation and Setup
Currently, python>=3.10 is supported. We recommend creating a new conda environment for dacapo with python 3.10.
```
conda create -n dacapo python=3.10
conda activate dacapo
```

Then install DaCapo using pip with the following command:
```
pip install dacapo-ml
```
This will install the minimum required dependencies.

You may additionally utilize a MongoDB server for storing outputs. To install and run MongoDB locally, refer to the MongoDB documentation [here](https://www.mongodb.com/docs/manual/installation/).

The use of MongoDB, as well as specifying the compute context (on cluster or not) should be specified in the ```dacapo.yaml``` in the main directory.

## Functionality Overview

Tasks we support and approaches for those tasks:
 - Instance Segmentation
    - Affinities
    - Local Shape Descriptors
 - Semantic segmentation
    - Signed distances
    - One-hot encoding of different types of objects

## Helpful Resources & Tools
 - Chunked data, zarr, and n5
    - OME-Zarr: a cloud-optimized bioimaging file format with international community support (doi: [10.1101/2023.02.17.528834](https://pubmed.ncbi.nlm.nih.gov/36865282/))
    - Videos about N5 and Fiji can be found in [this playlist](https://www.youtube.com/playlist?list=PLmZHHIZ9Gz-IJA7HtW8quZcuLViz9Em6e). For other questions, join the discussion on the [Image.sc forum](https://forum.image.sc/tag/n5).
    - Read about chunked storage plugins in Fiji in this blog: [N5 plugins for Fiji](https://openorganelle.janelia.org/news/2023-02-06-n5-plugins-for-fiji)
    - Script for converting tif to zarr can be found [here](https://github.com/yuriyzubov/tif-to-zarr)
 - Segmentations
    - A description of local shape descriptors used for affinities task. Read the blog [here](https://localshapedescriptors.github.io/). Example image from the blog showing the difference between segmentations:
    - ![](https://localshapedescriptors.github.io/assets/img/detection_vs_segmentation_neurons.jpeg)
 - CellMap Models
    - [GitHub Repo](https://github.com/janelia-cellmap/cellmap-models) of published models
    - For example, the COSEM trained pytorch networks are located [here](https://github.com/janelia-cellmap/cellmap-models/tree/main/src/cellmap_models/pytorch/cosem).
 - [OpenOrganelle.org](https://openorganelle.janelia.org)
    - ![](https://raw.githubusercontent.com/janelia-cellmap/dacapo/main/docs/source/_static/mito_pred-seg.gif)
    - Example of [unprocessed distance predictions](https://tinyurl.com/3kw2tuab)
    - Example of [refined segmentations](https://tinyurl.com/k59pba98) that have undergone post-processing (e.g., thresholding, masking, smoothing)
    - Example of [groundtruth data](https://tinyurl.com/pu8mespz)
 - Visualization
    - [Neuroglancer GitHub Repo](https://github.com/google/neuroglancer)