Text-to-Image
Diffusers
PyTorch
flow-matching
rectified-flow
dit
dual-stream-dit
sprint
anime
File size: 13,969 Bytes
f3fd58e
ad580cf
 
 
 
 
 
 
 
 
 
 
 
 
 
f3fd58e
ad580cf
 
f3fd58e
 
 
 
 
ad580cf
f3fd58e
 
ad580cf
c711288
f3fd58e
ad580cf
 
f3fd58e
ad580cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7b2cee9
 
ad580cf
 
 
 
 
 
 
 
 
 
7b2cee9
 
ad580cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7b2cee9
 
ad580cf
 
 
f3fd58e
ad580cf
f3fd58e
 
ad580cf
 
7b2cee9
 
ad580cf
 
f3fd58e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ad580cf
 
f3fd58e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ad580cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f3fd58e
 
 
 
 
 
ad580cf
 
 
 
 
 
 
 
 
 
 
 
 
 
f3fd58e
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253

---
license: openrail++
library_name: diffusers
tags:
- text-to-image
- flow-matching
- rectified-flow
- dit
- dual-stream-dit
- sprint
- anime
- pytorch
widget:
- output:
    url: images/sample_512_asuka.png
  text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, parted lips, official alternate costume, pilot suit, red bodysuit, black eye patch, animal hat, cabbie hat, cat hat, black hat, gloves, hat, hat horns, jacket, mecha pilot suit, bodysuit under clothes, red gloves, red jacket, track jacket, turtleneck, turtleneck jacket, interface headset, multicolored clothes, interface headset \\(evangelion\\), upper body, simple background"
- output:
    url: images/sample_512_hoshino_ai.png
  text: "1girl, hoshino ai, oshi no ko, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, dress, gloves, hair ornament, idol, idol clothes, long hair, looking at viewer, microphone, one eye closed, open mouth, pink dress, pink gloves, pointing, purple eyes, purple hair, solo, star-shaped pupils, star, star in eye, symbol-shaped pupils, symbol in eye, anime coloring, black shirt, brooch, frilled gloves, frilled shirt, frilled sleeves, frills, hair between eyes, heart, heart brooch, jewelry, messy hair, one side up, pink shirt, rabbit hair ornament, shiny clothes, shiny skin, shirt, short sleeves, solo, star \\(symbol\\), turtleneck, wing brooch, wings, yellow shirt, upper body, simple background"
- output:
    url: images/sample_512_asuka_bodysuit.png
  text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, official alternate costume, bodysuit, breasts, brown hair, closed mouth, clothing cutout, dark background, expressionless, eyepatch, falling petals, floating hair, hair ornament, holding, lance of longinus \\(evangelion\\), long hair, looking at viewer, looking down, mecha pilot suit, medium breasts, petals, plugsuit \\(evangelion\\), polearm, red bodysuit, solo, spear, standing, taut clothes, weapon, holding weapon, holding polearm, plugsuit, interface headset, light particle, upper body"
- output:
    url: images/sample_512_yuri_claire.png
  text: "2girls, claire francois, rae taylor, watashi no oshi wa akuyaku reijou, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, :d, ^ ^, blonde hair, blue eyes, blush, white ascot, bow, brown hair, closed eyes, collared shirt, cropped jacket, drill hair, drill sidelocks, floating hair, frills, gem, hair bow, half updo, holding another's arm, jacket, long hair, long sleeves, multiple girls, red bow, red jacket, school uniform, shirt, short hair, smile, yuri, brown eyes, multiple girls, white shirt, ascot, brooch, frilled jacket, jewelry, medium breasts, very long hair, upper body"
- output:
    url: images/sample_512_akane.png
  text: "1girl, kurokawa akane, oshi no ko, momoko \\(momopoco\\), toosaka asagi, ebifurya, hanagata, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, blue eyes, blue hair, cherry blossoms, closed mouth, dutch angle, falling petals, floating hair, floral print, floral print kimono, hair ribbon, hakama, half updo, holding, holding umbrella, japanese clothes, kimono, light blush, long hair, looking at viewer, multicolored eyes, official alternate hairstyle, oil-paper umbrella, parted bangs, petals, pink hakama, pink kimono, pink petals, pink ribbon, print hakama, print kimono, ribbon, simple background, smile, solo, straight hair, swept bangs, two-tone eyes, umbrella, white background, wide sleeves, light particle,upper body"
datasets:
- aipracticecafe-mirror/curated-danbooru-2026-256px-flux2-vae
- aipracticecafe/curated-danbooru-2026-512px-flux2-vae
- aipracticecafe/curated-danbooru-2026
---
# Asuka-FM: 200M Dual-Stream SPRINT DiT (Flow Matching)

An ultra-efficient, 200M-parameter **Dual-Stream Diffusion Transformer
(DiT)** trained from scratch on consumer hardware (1x NVIDIA RTX 4090 in ~50
hours) using **Rectified Flow / Flow Matching**, **SPRINT** token-dropping,
and **Continuous Aspect-Normalized RoPE**.

Despite its compact size, the model generates high-fidelity anime art with
accurate character reproduction, prompt alignment, and zero-shot camera
control (panning and zoom) at inference time.

- **Developed by**: [aipracticecafe](https://huggingface.co/aipracticecafe)
- **Repository**: [deeplearningcafe/asuka-fm](https://github.com/deeplearningcafe/asuka-fm)
- **Dataset**: [curated-danbooru-2026](https://huggingface.co/datasets/aipracticecafe/curated-danbooru-2026) (~337K samples)
- **Architecture**: Dual-Stream DiT (SPRINT-aligned, SwiGLU, Sandwich RMSNorm)
- **Parameters**: ~200M (Backbone) + 230M (Text Encoder)
- **Objective**: Conditional Flow Matching (Logit-Normal timestep sampling)
- **License**: OpenRAIL++

<Gallery /> 

---

## Key Architecture & Features

### 1. SPRINT Dual-Stream Transformer Backbone
The base architecture is based on the results from (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models)[https://arxiv.org/abs/2606.11289]
- **Minimalist Conditioning**: Uses prepended time tokens rather than heavy
  per-block AdaLN modulation, reducing parameter overhead.
- **Sandwich Normalization & SwiGLU**: RMSNorm placed before and after
  attention and FFN operations ensures training stability in bfloat16.
- **SPRINT Acceleration**: Employs structured group-wise 2x2 patch
  dropping (75% drop ratio during early pretraining), cutting compute
  demands by half while retaining global semantic context.

### 2. HDM Continuous Aspect-Normalized RoPE (Camera Control)
Based on the [Patch Diffusion: Faster and More Data-Efficient](https://arxiv.org/abs/2304.12526) implementation proposed in the [HDM](https://arxiv.org/abs/2509.06068) paper. Early experiments with discrete integer position IDs caused severe multi-head
replication when deviating from trained aspect ratios. We resolve this by
adopting continuous aspect-normalized coordinate mapping which was the original idea proposed in HDM:

$$r_H = \sqrt{\frac{H}{W}}, \quad r_W = \sqrt{\frac{W}{H}}$$

$$y_{\text{pos}} = \left(\frac{y_{\text{center}}}{H}\right) \cdot (2 r_H) - r_H, \quad
x_{\text{pos}} = \left(\frac{x_{\text{center}}}{W}\right) \cdot (2 r_W) - r_W$$

Manipulating $(x_{\text{pos}}, y_{\text{pos}})$ directly controls the visual
framing at inference time:
- **X-Shift / Y-Shift**: Translates the virtual camera viewport horizontally
  or vertically.
- **Zoom**: Scales spatial coordinates around the focal center.

### 3. Components
- **Text Encoder**: `LiquidAI/LFM2.5-230M` (unidirectional causal LM hidden
  states passed via a 2-layer Transformer Text Adapter).
- **Autoencoder**: `kaiyuyue/FLUX.2-dev-vae` (C=32, f=8,
  patch size p=2, effective image token size 16x16).

---

## Training Recipe & Multi-Stage Progression (1x RTX 4090, ~100h)

The model uses progressive multi-resolution pretraining on **337,000 curated
Danbooru samples** with shifted square cropping and in-RAM latent caching.

- **Optimizer**: 8-bit AdamW (`bitsandbytes`), $\beta=(0.9, 0.95)$, WD=0.01
- **Learning Rate**: 5e-4 with warmup and cosine annealing only for last stage
- **Effective Batch Size**: 256 (Batch size 64x4 gradient accumulation)
- **Precision**: Mixed Precision (`torch.bfloat16` with `torch.compile`)

### Stage 1: 256px Base Pretraining (Epochs 0 – 170)
* **Goal**: Establish character concept learning, global structure, and RoPE
  spatial alignment.

| Phase | Epochs | Token Drop | RoPE Mode | Description |
|---|---|---|---|---|
| **1A** | 0 – 100 | 0.75 (SPRINT) | Discrete | Rapid macro semantic convergence |
| **1B** | 100 – 120 | 0.75 (SPRINT) | Continuous | RoPE coordinate adaptation |
| **1C** | 120 – 145 | 0.00 (Dense) | Continuous | Full-sequence detail refinement |
| **1D** | 145 – 170 | 0.00 (Dense) | Continuous | WSD cosine learning rate decay |

### Stage 2: 512px Resolution Adaptation (Epochs 170 – 209)
* **Goal**: Adapt patch receptive fields to higher frequency textures and
  sharper lineart.

| Phase | Epochs | Token Drop | RoPE Mode | Description |
|---|---|---|---|---|
| **2A** | 170 – 194 (25 ep) | 0.75 (SPRINT) | Continuous | High-res structural adaptation |
| **2B** | 194 – 204 (10 ep) | 0.00 (Dense) | Continuous | Fine-tuning with constant LR |
| **2C** | 204 – 209 (5 ep) | 0.00 (Dense) | Continuous | Final cosine annealing decay |

### Stage 3: 1MP Native Training (Active Run)
* Scaling from 512px to 1024px aspect ratio buckets.

---

## Checkpoint Guide

| Checkpoint Directory | Stage | Epoch / Step | Recommended Usage |
|---|---|---|---|
| `epoch_169_step_229206` | Stage 1 Final | Ep 169 (Step 229k) | Final 256px decayed checkpoint |
| `epoch_194_step_265389` | Stage 2A Final | Ep 194 (Step 265k) | Fast 512px SPRINT-sparse checkpoint |
| `epoch_204_step_281281` | Stage 2B Final | Ep 204 (Step 281k) | Dense 512px pre-decay checkpoint |
| `epoch_209_step_290199` | Stage 2 Final | Ep 209 (Step 290k) | **Recommended for 512px–768px inference** |

---

## 512px Generation Outputs

![512px Samples](images/sample_512_grid_comparison.png)
*Figure: 512px generations demonstrating improved micro-texture, eyes,
and line definition achieved in Stage 2.*

---


## Inference and WebUI
The training repository has a simple Gradio UI to perform sampling with camera controls.
[ui_app.py](https://github.com/deeplearningcafe/asuka-fm/blob/main/scripts/ui_app.py)

### Recommended Settings
- **Sampling Algorithm**: Forward Euler ODE Solver (Rectified Flow)
- **Inference Steps**: 25 – 35
- **CFG Scale**: 4.5 – 6.0
- **Time Shift**: 1.0 – 3.0 (higher values enhance contrast/saturation)

### Tag Formatting Template
```text
1girl/1boy, character name, series name, artists, rating, quality tags, actions/clothing
```

Negative prompt:
```text
sketch, monochrome, english text, realistic, traditional media, very displeasing, displeasing, bad score, worse score, bad quality,worst quality, worst detail, watermark, signature, blurry, bad hands, bad anatomy, bad fingers, extra fingers, extra limbs, deformed limbs, comic, speech bubble, lowres
```

### Python Sampling Script

```python
import torch
from omegaconf import OmegaConf
from src.models.factory import load_trainable_model
from src.diffusion.schedules import LinearSchedule
from src.diffusion.sampling import generate_samples
cfg = OmegaConf.load("config.yaml")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16
unet, text_encoder, vae, tokenizer, _ = load_trainable_model(
    models_path=cfg.paths.models,
    device=device,
    dtype=dtype,
    resume_from_checkpoint="path_to_checkpoint",
    model_type="sprint_dual",
    model_cfg=cfg.models,
    autocast_dtype=dtype,
)
schedule = LinearSchedule(device=device)
sample_configs = [
    {
        "prompt": "1girl, souryuu asuka langley, neon genesis evangelion, "
                  "masterpiece, absurdres",
        "negative_prompt": "very displeasing, displeasing, bad score",
        "height": 256,
        "width": 384,
        "sample_steps": 30,
        "cfg_scale": 6.0,
        "shift": 1.0,
        "zoom": 1.0,
        "x_shift": 0.0,
        "y_shift": 0.0,
        "seed": 42,
    }
]
images = generate_samples(
    unet=unet,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    vae=vae,
    schedule=schedule,
    sample_configs=sample_configs,
    global_batch_size=1,
    diffusion_type="flow_matching",
    device=device,
    dtype=dtype,
    autocast_dtype=dtype,
    use_unet_mult=False,
    vae_mean=cfg.models.vae_mean,
    vae_std=cfg.models.vae_std,
    in_channels=cfg.models.in_channels,
    coord_system="aspect_norm",
)
images[0].save("output.png")
```

---

## Content Advisory & Limitations

- **Resolution Scope**: Checkpoints up to `epoch_209_*` are natively optimized
  for 512px generation and reliably scale up to 768px/1024px with aspect-normalized
  RoPE. Native 1024px fidelity will be unlocked upon completion of Stage 3.
- **Complex Hands**: The compact 200M parameter count prioritizes prompt
  faithfulness and character identity; complex hand interactions may still
  require multiple seed attempts.
- **NSFW & Sensitive Content:** The model was trained on a dataset that contains uncensored anime-style art, including sensitive, questionable, and explicit material (`g`, `s`, `q`, `e` ratings). Use appropriate tag filtering during training if a SFW-only model is required.
- **Legal Disclaimer:** Images are sourced from Danbooru for research, educational, and generative modeling purposes under fair use. Copyright of all underlying illustrations belongs to their respective creators.
---

## References
*   **i1**: [i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models](https://arxiv.org/abs/2606.11289)
*   **Patch Diffusion**:  [Patch Diffusion: Faster and More Data-Efficient](https://arxiv.org/abs/2304.12526)
*   **Home-made Diffusion Model from Scratch to Hatch**: [HDM](https://arxiv.org/abs/2509.06068) 
*   **Danbooru Dataset Toolkit**: Dataset processed and formatted using the [Danbooru Dataset Toolkit](https://github.com/deeplearningcafe/danbooru-dataset-toolkit).

## License

This model is licensed under the **CreativeML Open RAIL++-M License**. You are
free to use, modify, and distribute the model weights provided you include the
license text and respect the behavioral use restrictions.