Instructions to use aipracticecafe/dual-stream-dit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use aipracticecafe/dual-stream-dit with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("aipracticecafe/dual-stream-dit", dtype=torch.bfloat16, device_map="cuda") prompt = "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, parted lips, official alternate costume, pilot suit, red bodysuit, black eye patch, animal hat, cabbie hat, cat hat, black hat, gloves, hat, hat horns, jacket, mecha pilot suit, bodysuit under clothes, red gloves, red jacket, track jacket, turtleneck, turtleneck jacket, interface headset, multicolored clothes, interface headset \\(evangelion\\), upper body, simple background" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,215 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: openrail++
|
| 3 |
+
library_name: diffusers
|
| 4 |
+
tags:
|
| 5 |
+
- text-to-image
|
| 6 |
+
- flow-matching
|
| 7 |
+
- rectified-flow
|
| 8 |
+
- dit
|
| 9 |
+
- dual-stream-dit
|
| 10 |
+
- sprint
|
| 11 |
+
- anime
|
| 12 |
+
- pytorch
|
| 13 |
+
widget:
|
| 14 |
+
- output:
|
| 15 |
+
url: images/sample_asuka_zoom.png
|
| 16 |
+
text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, parted lips, official alternate costume, pilot suit, red bodysuit, black eye patch, animal hat, cabbie hat, cat hat, black hat, gloves, hat, hat horns, jacket, mecha pilot suit, bodysuit under clothes, red gloves, red jacket, track jacket, turtleneck, turtleneck jacket, interface headset, multicolored clothes, interface headset \\(evangelion\\), upper body, simple background"
|
| 17 |
+
- output:
|
| 18 |
+
url: images/sample_hoshino_ai.png
|
| 19 |
+
text: 1girl, hoshino ai, oshi no ko, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, dress, gloves, hair ornament, idol, idol clothes, long hair, looking at viewer, microphone, one eye closed, open mouth, pink dress, pink gloves, pointing, purple eyes, purple hair, solo, star-shaped pupils, star, star in eye, symbol-shaped pupils, symbol in eye, anime coloring, black shirt, brooch, frilled gloves, frilled shirt, frilled sleeves, frills, hair between eyes, heart, heart brooch, jewelry, messy hair, one side up, pink shirt, rabbit hair ornament, shiny clothes, shiny skin, shirt, short sleeves, solo, star \(symbol\), turtleneck, wing brooch, wings, yellow shirt, upper body, simple background
|
| 20 |
+
- output:
|
| 21 |
+
url: images/sample_asuka_bodysuit.png
|
| 22 |
+
text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, official alternate costume, bodysuit, breasts, brown hair, closed mouth, clothing cutout, dark background, expressionless, eyepatch, falling petals, floating hair, hair ornament, holding, lance of longinus \\(evangelion\\), long hair, looking at viewer, looking down, mecha pilot suit, medium breasts, petals, plugsuit \\(evangelion\\), polearm, red bodysuit, solo, spear, standing, taut clothes, weapon, holding weapon, holding polearm, plugsuit, interface headset, light particle,upper body"
|
| 23 |
+
- output:
|
| 24 |
+
url: images/sample_yuri_claire.png
|
| 25 |
+
text: 2girls, claire francois, rae taylor, watashi no oshi wa akuyaku reijou, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, :d, ^ ^, blonde hair, blue eyes, blush, white ascot, bow, brown hair, closed eyes, collared shirt, cropped jacket, drill hair, drill sidelocks, floating hair, frills, gem, hair bow, half updo, holding another's arm, jacket, long hair, long sleeves, multiple girls, red bow, red jacket, school uniform, shirt, short hair, smile, yuri, brown eyes, multiple girls, white shirt, ascot, brooch, frilled jacket, jewelry, medium breasts, very long hair, upper body
|
| 26 |
+
datasets:
|
| 27 |
+
- aipracticecafe-mirror/curated-danbooru-2026-256px-flux2-vae
|
| 28 |
+
- aipracticecafe/curated-danbooru-2026
|
| 29 |
+
---
|
| 30 |
+
|
| 31 |
+
# Asuka-FM: 200M Dual-Stream SPRINT DiT (Flow Matching)
|
| 32 |
+
|
| 33 |
+
An ultra-efficient, 200M-parameter **Dual-Stream Diffusion Transformer
|
| 34 |
+
(DiT)** trained from scratch on consumer hardware (1x NVIDIA RTX 4090 in ~50
|
| 35 |
+
hours) using **Rectified Flow / Flow Matching**, **SPRINT** token-dropping,
|
| 36 |
+
and **Continuous Aspect-Normalized RoPE**.
|
| 37 |
+
|
| 38 |
+
Despite its compact size, the model generates high-fidelity anime art with
|
| 39 |
+
accurate character reproduction, prompt alignment, and zero-shot camera
|
| 40 |
+
control (panning and zoom) at inference time.
|
| 41 |
+
|
| 42 |
+
- **Developed by**: [aipracticecafe](https://huggingface.co/aipracticecafe)
|
| 43 |
+
- **Repository**: [deeplearningcafe/asuka-fm](https://github.com/deeplearningcafe/asuka-fm)
|
| 44 |
+
- **Dataset**: [curated-danbooru-2026](https://huggingface.co/datasets/aipracticecafe/curated-danbooru-2026) (~337K samples)
|
| 45 |
+
- **Architecture**: Dual-Stream DiT (SPRINT-aligned, SwiGLU, Sandwich RMSNorm)
|
| 46 |
+
- **Parameters**: ~200M (Backbone) + 230M (Text Encoder)
|
| 47 |
+
- **Objective**: Conditional Flow Matching (Logit-Normal timestep sampling)
|
| 48 |
+
- **License**: OpenRAIL++
|
| 49 |
+
|
| 50 |
+
---
|
| 51 |
+
|
| 52 |
+
## Key Architecture & Features
|
| 53 |
+
|
| 54 |
+
### 1. SPRINT Dual-Stream Transformer Backbone
|
| 55 |
+
The base architecture is based on the results from (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models)[https://arxiv.org/abs/2606.11289]
|
| 56 |
+
- **Minimalist Conditioning**: Uses prepended time tokens rather than heavy
|
| 57 |
+
per-block AdaLN modulation, reducing parameter overhead.
|
| 58 |
+
- **Sandwich Normalization & SwiGLU**: RMSNorm placed before and after
|
| 59 |
+
attention and FFN operations ensures training stability in bfloat16.
|
| 60 |
+
- **SPRINT Acceleration**: Employs structured group-wise $2 \times 2$ patch
|
| 61 |
+
dropping ($75\%$ drop ratio during early pretraining), cutting compute
|
| 62 |
+
demands by half while retaining global semantic context.
|
| 63 |
+
|
| 64 |
+
### 2. HDM Continuous Aspect-Normalized RoPE (Camera Control)
|
| 65 |
+
Based on the [Patch Diffusion: Faster and More Data-Efficient](https://arxiv.org/abs/2304.12526) implementation proposed in the [HDM](https://arxiv.org/abs/2509.06068) paper. Early experiments with discrete integer position IDs caused severe multi-head
|
| 66 |
+
replication when deviating from trained aspect ratios. We resolve this by
|
| 67 |
+
adopting continuous aspect-normalized coordinate mapping which was the original idea proposed in HDM:
|
| 68 |
+
|
| 69 |
+
$$r_H = \sqrt{\frac{H}{W}}, \quad r_W = \sqrt{\frac{W}{H}}$$
|
| 70 |
+
|
| 71 |
+
$$y_{\text{pos}} = \left(\frac{y_{\text{center}}}{H}\right) \cdot (2 r_H) - r_H, \quad
|
| 72 |
+
x_{\text{pos}} = \left(\frac{x_{\text{center}}}{W}\right) \cdot (2 r_W) - r_W$$
|
| 73 |
+
|
| 74 |
+
Manipulating $(x_{\text{pos}}, y_{\text{pos}})$ directly controls the visual
|
| 75 |
+
framing at inference time:
|
| 76 |
+
- **X-Shift / Y-Shift**: Translates the virtual camera viewport horizontally
|
| 77 |
+
or vertically.
|
| 78 |
+
- **Zoom**: Scales spatial coordinates around the focal center.
|
| 79 |
+
|
| 80 |
+
### 3. Components
|
| 81 |
+
- **Text Encoder**: `LiquidAI/LFM2.5-230M` (unidirectional causal LM hidden
|
| 82 |
+
states passed via a 2-layer Transformer Text Adapter).
|
| 83 |
+
- **Autoencoder**: `kaiyuyue/FLUX.2-dev-vae` ($C_{\text{lat}}=32$, $f=8$,
|
| 84 |
+
patch size $p=2$, effective image token size $16 \times 16$).
|
| 85 |
+
|
| 86 |
+
---
|
| 87 |
+
|
| 88 |
+
## Training Recipe (1x RTX 4090, ~50h)
|
| 89 |
+
|
| 90 |
+
The model was pretrained on **337,000 curated Danbooru samples** at 256px base
|
| 91 |
+
resolution with shifted square cropping and cached in-RAM latents.
|
| 92 |
+
|
| 93 |
+
| Phase | Epochs | Token Drop | RoPE Format | Description |
|
| 94 |
+
|---|---|---|---|---|
|
| 95 |
+
| **Phase 1** | 100 ep | 0.75 (SPRINT) | Discrete | Fast semantic convergence |
|
| 96 |
+
| **Phase 2** | 20 ep | 0.75 (SPRINT) | Continuous | RoPE coordinates alignment |
|
| 97 |
+
| **Phase 3** | 25 ep | 0.00 (Dense) | Continuous | High-frequency detail recovery |
|
| 98 |
+
| **Phase 4** | 25 ep | 0.00 (Dense) | Continuous | Cosine LR decay to minimum LR |
|
| 99 |
+
|
| 100 |
+
- **Total Epochs**: 175
|
| 101 |
+
- **Optimizer**: 8-bit AdamW (`bitsandbytes`), $\beta=(0.9, 0.95)$, WD=0.01
|
| 102 |
+
- **Learning Rate**: $5 \times 10^{-4}$ with warmup and cosine annealing only for last stage
|
| 103 |
+
- **Effective Batch Size**: 256 (Batch size 64 $\times$ 4 gradient accumulation)
|
| 104 |
+
- **Precision**: Mixed Precision (`torch.bfloat16` with `torch.compile`)
|
| 105 |
+
|
| 106 |
+
---
|
| 107 |
+
|
| 108 |
+
## Inference and WebUI
|
| 109 |
+
The training repository has a simple Gradio UI to perform sampling with camera controls.
|
| 110 |
+
[ui_app.py](https://github.com/deeplearningcafe/asuka-fm/blob/main/scripts/ui_app.py)
|
| 111 |
+
|
| 112 |
+
### Recommended Settings
|
| 113 |
+
- **Sampling Algorithm**: Forward Euler ODE Solver (Rectified Flow)
|
| 114 |
+
- **Inference Steps**: 25 – 35
|
| 115 |
+
- **CFG Scale**: 4.5 – 6.0
|
| 116 |
+
- **Time Shift**: 1.0 – 3.0 (higher values enhance contrast/saturation)
|
| 117 |
+
|
| 118 |
+
### Tag Formatting Template
|
| 119 |
+
```text
|
| 120 |
+
1girl/1boy, character name, series name, artists, rating, quality tags, actions/clothing
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
Negative prompt:
|
| 124 |
+
```text
|
| 125 |
+
sketch, monochrome, english text, realistic, traditional media, very displeasing, displeasing, bad score, worse score, bad quality,worst quality, worst detail, watermark, signature, blurry, bad hands, bad anatomy, bad fingers, extra fingers, extra limbs, deformed limbs, comic, speech bubble, lowres
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
### Python Sampling Script
|
| 129 |
+
|
| 130 |
+
```python
|
| 131 |
+
import torch
|
| 132 |
+
from omegaconf import OmegaConf
|
| 133 |
+
from src.models.factory import load_trainable_model
|
| 134 |
+
from src.diffusion.schedules import LinearSchedule
|
| 135 |
+
from src.diffusion.sampling import generate_samples
|
| 136 |
+
|
| 137 |
+
cfg = OmegaConf.load("config.yaml")
|
| 138 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 139 |
+
dtype = torch.bfloat16
|
| 140 |
+
|
| 141 |
+
unet, text_encoder, vae, tokenizer, _ = load_trainable_model(
|
| 142 |
+
models_path=cfg.paths.models,
|
| 143 |
+
device=device,
|
| 144 |
+
dtype=dtype,
|
| 145 |
+
resume_from_checkpoint="path_to_checkpoint",
|
| 146 |
+
model_type="sprint_dual",
|
| 147 |
+
model_cfg=cfg.models,
|
| 148 |
+
autocast_dtype=dtype,
|
| 149 |
+
)
|
| 150 |
+
|
| 151 |
+
schedule = LinearSchedule(device=device)
|
| 152 |
+
|
| 153 |
+
sample_configs = [
|
| 154 |
+
{
|
| 155 |
+
"prompt": "1girl, souryuu asuka langley, neon genesis evangelion, "
|
| 156 |
+
"masterpiece, absurdres",
|
| 157 |
+
"negative_prompt": "very displeasing, displeasing, bad score",
|
| 158 |
+
"height": 256,
|
| 159 |
+
"width": 384,
|
| 160 |
+
"sample_steps": 30,
|
| 161 |
+
"cfg_scale": 6.0,
|
| 162 |
+
"shift": 1.0,
|
| 163 |
+
"zoom": 1.0,
|
| 164 |
+
"x_shift": 0.0,
|
| 165 |
+
"y_shift": 0.0,
|
| 166 |
+
"seed": 42,
|
| 167 |
+
}
|
| 168 |
+
]
|
| 169 |
+
|
| 170 |
+
images = generate_samples(
|
| 171 |
+
unet=unet,
|
| 172 |
+
text_encoder=text_encoder,
|
| 173 |
+
tokenizer=tokenizer,
|
| 174 |
+
vae=vae,
|
| 175 |
+
schedule=schedule,
|
| 176 |
+
sample_configs=sample_configs,
|
| 177 |
+
global_batch_size=1,
|
| 178 |
+
diffusion_type="flow_matching",
|
| 179 |
+
device=device,
|
| 180 |
+
dtype=dtype,
|
| 181 |
+
autocast_dtype=dtype,
|
| 182 |
+
use_unet_mult=False,
|
| 183 |
+
vae_mean=cfg.models.vae_mean,
|
| 184 |
+
vae_std=cfg.models.vae_std,
|
| 185 |
+
in_channels=cfg.models.in_channels,
|
| 186 |
+
coord_system="aspect_norm",
|
| 187 |
+
)
|
| 188 |
+
|
| 189 |
+
images[0].save("output.png")
|
| 190 |
+
```
|
| 191 |
+
|
| 192 |
+
---
|
| 193 |
+
|
| 194 |
+
## Content Advisory & Limitations
|
| 195 |
+
|
| 196 |
+
- **Resolution**: Trained on 256px shifted crops. While continuous RoPE allows
|
| 197 |
+
sampling at 384px and 512px, high resolutions may exhibit slight softness
|
| 198 |
+
until a dedicated 512px/1024px fine-tuning pass is completed.
|
| 199 |
+
- **Complex Hands**: Small 200M parameter count prioritizes overall composition
|
| 200 |
+
and character concepts over fine micro-structures like fingernails.
|
| 201 |
+
- **NSFW & Sensitive Content:** The model was trained on a dataset that contains uncensored anime-style art, including sensitive, questionable, and explicit material (`g`, `s`, `q`, `e` ratings). Use appropriate tag filtering during training if a SFW-only model is required.
|
| 202 |
+
- **Legal Disclaimer:** Images are sourced from Danbooru for research, educational, and generative modeling purposes under fair use. Copyright of all underlying illustrations belongs to their respective creators.
|
| 203 |
+
---
|
| 204 |
+
|
| 205 |
+
## References
|
| 206 |
+
* **i1**: [i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models](https://arxiv.org/abs/2606.11289)
|
| 207 |
+
* **Patch Diffusion**: [Patch Diffusion: Faster and More Data-Efficient](https://arxiv.org/abs/2304.12526)
|
| 208 |
+
* **Home-made Diffusion Model from Scratch to Hatch**: [HDM](https://arxiv.org/abs/2509.06068)
|
| 209 |
+
* **Danbooru Dataset Toolkit**: Dataset processed and formatted using the [Danbooru Dataset Toolkit](https://github.com/deeplearningcafe/danbooru-dataset-toolkit).
|
| 210 |
+
|
| 211 |
+
## License
|
| 212 |
+
|
| 213 |
+
This model is licensed under the **CreativeML Open RAIL++-M License**. You are
|
| 214 |
+
free to use, modify, and distribute the model weights provided you include the
|
| 215 |
+
license text and respect the behavioral use restrictions.
|