Text-to-Image
Diffusers
PyTorch
flow-matching
rectified-flow
dit
dual-stream-dit
sprint
anime
aipracticecafe commited on
Commit
f3fd58e
·
verified ·
1 Parent(s): 4f94f6a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +66 -31
README.md CHANGED
@@ -1,3 +1,4 @@
 
1
  ---
2
  license: openrail++
3
  library_name: diffusers
@@ -12,22 +13,25 @@ tags:
12
  - pytorch
13
  widget:
14
  - output:
15
- url: images/sample_asuka_zoom.png
16
  text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, parted lips, official alternate costume, pilot suit, red bodysuit, black eye patch, animal hat, cabbie hat, cat hat, black hat, gloves, hat, hat horns, jacket, mecha pilot suit, bodysuit under clothes, red gloves, red jacket, track jacket, turtleneck, turtleneck jacket, interface headset, multicolored clothes, interface headset \\(evangelion\\), upper body, simple background"
17
  - output:
18
- url: images/sample_hoshino_ai.png
19
- text: 1girl, hoshino ai, oshi no ko, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, dress, gloves, hair ornament, idol, idol clothes, long hair, looking at viewer, microphone, one eye closed, open mouth, pink dress, pink gloves, pointing, purple eyes, purple hair, solo, star-shaped pupils, star, star in eye, symbol-shaped pupils, symbol in eye, anime coloring, black shirt, brooch, frilled gloves, frilled shirt, frilled sleeves, frills, hair between eyes, heart, heart brooch, jewelry, messy hair, one side up, pink shirt, rabbit hair ornament, shiny clothes, shiny skin, shirt, short sleeves, solo, star \(symbol\), turtleneck, wing brooch, wings, yellow shirt, upper body, simple background
 
 
 
20
  - output:
21
- url: images/sample_asuka_bodysuit.png
22
- text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, official alternate costume, bodysuit, breasts, brown hair, closed mouth, clothing cutout, dark background, expressionless, eyepatch, falling petals, floating hair, hair ornament, holding, lance of longinus \\(evangelion\\), long hair, looking at viewer, looking down, mecha pilot suit, medium breasts, petals, plugsuit \\(evangelion\\), polearm, red bodysuit, solo, spear, standing, taut clothes, weapon, holding weapon, holding polearm, plugsuit, interface headset, light particle,upper body"
23
  - output:
24
- url: images/sample_yuri_claire.png
25
- text: 2girls, claire francois, rae taylor, watashi no oshi wa akuyaku reijou, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, :d, ^ ^, blonde hair, blue eyes, blush, white ascot, bow, brown hair, closed eyes, collared shirt, cropped jacket, drill hair, drill sidelocks, floating hair, frills, gem, hair bow, half updo, holding another's arm, jacket, long hair, long sleeves, multiple girls, red bow, red jacket, school uniform, shirt, short hair, smile, yuri, brown eyes, multiple girls, white shirt, ascot, brooch, frilled jacket, jewelry, medium breasts, very long hair, upper body
26
  datasets:
27
  - aipracticecafe-mirror/curated-danbooru-2026-256px-flux2-vae
 
28
  - aipracticecafe/curated-danbooru-2026
29
  ---
30
-
31
  # Asuka-FM: 200M Dual-Stream SPRINT DiT (Flow Matching)
32
 
33
  An ultra-efficient, 200M-parameter **Dual-Stream Diffusion Transformer
@@ -87,26 +91,62 @@ framing at inference time:
87
 
88
  ---
89
 
90
- ## Training Recipe (1x RTX 4090, ~50h)
91
 
92
- The model was pretrained on **337,000 curated Danbooru samples** at 256px base
93
- resolution with shifted square cropping and cached in-RAM latents.
94
 
95
- | Phase | Epochs | Token Drop | RoPE Format | Description |
96
- |---|---|---|---|---|
97
- | **Phase 1** | 100 ep | 0.75 (SPRINT) | Discrete | Fast semantic convergence |
98
- | **Phase 2** | 20 ep | 0.75 (SPRINT) | Continuous | RoPE coordinates alignment |
99
- | **Phase 3** | 25 ep | 0.00 (Dense) | Continuous | High-frequency detail recovery |
100
- | **Phase 4** | 25 ep | 0.00 (Dense) | Continuous | Cosine LR decay to minimum LR |
101
-
102
- - **Total Epochs**: 175
103
  - **Optimizer**: 8-bit AdamW (`bitsandbytes`), $\beta=(0.9, 0.95)$, WD=0.01
104
  - **Learning Rate**: 5e-4 with warmup and cosine annealing only for last stage
105
  - **Effective Batch Size**: 256 (Batch size 64x4 gradient accumulation)
106
  - **Precision**: Mixed Precision (`torch.bfloat16` with `torch.compile`)
107
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
  ---
109
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
  ## Inference and WebUI
111
  The training repository has a simple Gradio UI to perform sampling with camera controls.
112
  [ui_app.py](https://github.com/deeplearningcafe/asuka-fm/blob/main/scripts/ui_app.py)
@@ -135,11 +175,9 @@ from omegaconf import OmegaConf
135
  from src.models.factory import load_trainable_model
136
  from src.diffusion.schedules import LinearSchedule
137
  from src.diffusion.sampling import generate_samples
138
-
139
  cfg = OmegaConf.load("config.yaml")
140
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
141
  dtype = torch.bfloat16
142
-
143
  unet, text_encoder, vae, tokenizer, _ = load_trainable_model(
144
  models_path=cfg.paths.models,
145
  device=device,
@@ -149,9 +187,7 @@ unet, text_encoder, vae, tokenizer, _ = load_trainable_model(
149
  model_cfg=cfg.models,
150
  autocast_dtype=dtype,
151
  )
152
-
153
  schedule = LinearSchedule(device=device)
154
-
155
  sample_configs = [
156
  {
157
  "prompt": "1girl, souryuu asuka langley, neon genesis evangelion, "
@@ -168,7 +204,6 @@ sample_configs = [
168
  "seed": 42,
169
  }
170
  ]
171
-
172
  images = generate_samples(
173
  unet=unet,
174
  text_encoder=text_encoder,
@@ -187,7 +222,6 @@ images = generate_samples(
187
  in_channels=cfg.models.in_channels,
188
  coord_system="aspect_norm",
189
  )
190
-
191
  images[0].save("output.png")
192
  ```
193
 
@@ -195,11 +229,12 @@ images[0].save("output.png")
195
 
196
  ## Content Advisory & Limitations
197
 
198
- - **Resolution**: Trained on 256px shifted crops. While continuous RoPE allows
199
- sampling at 384px and 512px, high resolutions may exhibit slight softness
200
- until a dedicated 512px/1024px fine-tuning pass is completed.
201
- - **Complex Hands**: Small 200M parameter count prioritizes overall composition
202
- and character concepts over fine micro-structures like fingernails.
 
203
  - **NSFW & Sensitive Content:** The model was trained on a dataset that contains uncensored anime-style art, including sensitive, questionable, and explicit material (`g`, `s`, `q`, `e` ratings). Use appropriate tag filtering during training if a SFW-only model is required.
204
  - **Legal Disclaimer:** Images are sourced from Danbooru for research, educational, and generative modeling purposes under fair use. Copyright of all underlying illustrations belongs to their respective creators.
205
  ---
@@ -214,4 +249,4 @@ images[0].save("output.png")
214
 
215
  This model is licensed under the **CreativeML Open RAIL++-M License**. You are
216
  free to use, modify, and distribute the model weights provided you include the
217
- license text and respect the behavioral use restrictions.
 
1
+
2
  ---
3
  license: openrail++
4
  library_name: diffusers
 
13
  - pytorch
14
  widget:
15
  - output:
16
+ url: images/sample_512_asuka.png
17
  text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, parted lips, official alternate costume, pilot suit, red bodysuit, black eye patch, animal hat, cabbie hat, cat hat, black hat, gloves, hat, hat horns, jacket, mecha pilot suit, bodysuit under clothes, red gloves, red jacket, track jacket, turtleneck, turtleneck jacket, interface headset, multicolored clothes, interface headset \\(evangelion\\), upper body, simple background"
18
  - output:
19
+ url: images/sample_512_hoshino_ai.png
20
+ text: "1girl, hoshino ai, oshi no ko, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, dress, gloves, hair ornament, idol, idol clothes, long hair, looking at viewer, microphone, one eye closed, open mouth, pink dress, pink gloves, pointing, purple eyes, purple hair, solo, star-shaped pupils, star, star in eye, symbol-shaped pupils, symbol in eye, anime coloring, black shirt, brooch, frilled gloves, frilled shirt, frilled sleeves, frills, hair between eyes, heart, heart brooch, jewelry, messy hair, one side up, pink shirt, rabbit hair ornament, shiny clothes, shiny skin, shirt, short sleeves, solo, star \\(symbol\\), turtleneck, wing brooch, wings, yellow shirt, upper body, simple background"
21
+ - output:
22
+ url: images/sample_512_asuka_bodysuit.png
23
+ text: "1girl, souryuu asuka langley, neon genesis evangelion, rebuild of evangelion, evangelion: 3.0 you can \\(not\\) redo, momoko \\(momopoco\\), toosaka asagi, ebifurya, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, long hair, blue eyes, orange hair, breasts, official alternate costume, bodysuit, breasts, brown hair, closed mouth, clothing cutout, dark background, expressionless, eyepatch, falling petals, floating hair, hair ornament, holding, lance of longinus \\(evangelion\\), long hair, looking at viewer, looking down, mecha pilot suit, medium breasts, petals, plugsuit \\(evangelion\\), polearm, red bodysuit, solo, spear, standing, taut clothes, weapon, holding weapon, holding polearm, plugsuit, interface headset, light particle, upper body"
24
  - output:
25
+ url: images/sample_512_yuri_claire.png
26
+ text: "2girls, claire francois, rae taylor, watashi no oshi wa akuyaku reijou, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, :d, ^ ^, blonde hair, blue eyes, blush, white ascot, bow, brown hair, closed eyes, collared shirt, cropped jacket, drill hair, drill sidelocks, floating hair, frills, gem, hair bow, half updo, holding another's arm, jacket, long hair, long sleeves, multiple girls, red bow, red jacket, school uniform, shirt, short hair, smile, yuri, brown eyes, multiple girls, white shirt, ascot, brooch, frilled jacket, jewelry, medium breasts, very long hair, upper body"
27
  - output:
28
+ url: images/sample_512_yuri_claire.png
29
+ text: "1girl, kurokawa akane, oshi no ko, momoko \\(momopoco\\), toosaka asagi, ebifurya, hanagata, sensitive, masterpiece, good score, absurdres, highres, best quality, amazing quality, great quality, solo, blue eyes, blue hair, cherry blossoms, closed mouth, dutch angle, falling petals, floating hair, floral print, floral print kimono, hair ribbon, hakama, half updo, holding, holding umbrella, japanese clothes, kimono, light blush, long hair, looking at viewer, multicolored eyes, official alternate hairstyle, oil-paper umbrella, parted bangs, petals, pink hakama, pink kimono, pink petals, pink ribbon, print hakama, print kimono, ribbon, simple background, smile, solo, straight hair, swept bangs, two-tone eyes, umbrella, white background, wide sleeves, light particle,upper body"
30
  datasets:
31
  - aipracticecafe-mirror/curated-danbooru-2026-256px-flux2-vae
32
+ - aipracticecafe/curated-danbooru-2026-512px-flux2-vae
33
  - aipracticecafe/curated-danbooru-2026
34
  ---
 
35
  # Asuka-FM: 200M Dual-Stream SPRINT DiT (Flow Matching)
36
 
37
  An ultra-efficient, 200M-parameter **Dual-Stream Diffusion Transformer
 
91
 
92
  ---
93
 
94
+ ## Training Recipe & Multi-Stage Progression (1x RTX 4090, ~100h)
95
 
96
+ The model uses progressive multi-resolution pretraining on **337,000 curated
97
+ Danbooru samples** with shifted square cropping and in-RAM latent caching.
98
 
 
 
 
 
 
 
 
 
99
  - **Optimizer**: 8-bit AdamW (`bitsandbytes`), $\beta=(0.9, 0.95)$, WD=0.01
100
  - **Learning Rate**: 5e-4 with warmup and cosine annealing only for last stage
101
  - **Effective Batch Size**: 256 (Batch size 64x4 gradient accumulation)
102
  - **Precision**: Mixed Precision (`torch.bfloat16` with `torch.compile`)
103
 
104
+ ### Stage 1: 256px Base Pretraining (Epochs 0 – 170)
105
+ * **Goal**: Establish character concept learning, global structure, and RoPE
106
+ spatial alignment.
107
+
108
+ | Phase | Epochs | Token Drop | RoPE Mode | Description |
109
+ |---|---|---|---|---|
110
+ | **1A** | 0 – 100 | 0.75 (SPRINT) | Discrete | Rapid macro semantic convergence |
111
+ | **1B** | 100 – 120 | 0.75 (SPRINT) | Continuous | RoPE coordinate adaptation |
112
+ | **1C** | 120 – 145 | 0.00 (Dense) | Continuous | Full-sequence detail refinement |
113
+ | **1D** | 145 – 170 | 0.00 (Dense) | Continuous | WSD cosine learning rate decay |
114
+
115
+ ### Stage 2: 512px Resolution Adaptation (Epochs 170 – 209)
116
+ * **Goal**: Adapt patch receptive fields to higher frequency textures and
117
+ sharper lineart.
118
+
119
+ | Phase | Epochs | Token Drop | RoPE Mode | Description |
120
+ |---|---|---|---|---|
121
+ | **2A** | 170 – 194 (25 ep) | 0.75 (SPRINT) | Continuous | High-res structural adaptation |
122
+ | **2B** | 194 – 204 (10 ep) | 0.00 (Dense) | Continuous | Fine-tuning with constant LR |
123
+ | **2C** | 204 – 209 (5 ep) | 0.00 (Dense) | Continuous | Final cosine annealing decay |
124
+
125
+ ### Stage 3: 1MP Native Training (Active Run)
126
+ * Scaling from 512px to 1024px aspect ratio buckets.
127
+
128
  ---
129
 
130
+ ## Checkpoint Guide
131
+
132
+ | Checkpoint Directory | Stage | Epoch / Step | Recommended Usage |
133
+ |---|---|---|---|
134
+ | `epoch_169_step_229206` | Stage 1 Final | Ep 169 (Step 229k) | Final 256px decayed checkpoint |
135
+ | `epoch_194_step_265389` | Stage 2A Final | Ep 194 (Step 265k) | Fast 512px SPRINT-sparse checkpoint |
136
+ | `epoch_204_step_281281` | Stage 2B Final | Ep 204 (Step 281k) | Dense 512px pre-decay checkpoint |
137
+ | `epoch_209_step_290199` | Stage 2 Final | Ep 209 (Step 290k) | **Recommended for 512px–768px inference** |
138
+
139
+ ---
140
+
141
+ ## 512px Generation Outputs
142
+
143
+ ![512px Samples](images/sample_512_grid_comparison.png)
144
+ *Figure: 512px generations demonstrating improved micro-texture, eyes,
145
+ and line definition achieved in Stage 2.*
146
+
147
+ ---
148
+
149
+
150
  ## Inference and WebUI
151
  The training repository has a simple Gradio UI to perform sampling with camera controls.
152
  [ui_app.py](https://github.com/deeplearningcafe/asuka-fm/blob/main/scripts/ui_app.py)
 
175
  from src.models.factory import load_trainable_model
176
  from src.diffusion.schedules import LinearSchedule
177
  from src.diffusion.sampling import generate_samples
 
178
  cfg = OmegaConf.load("config.yaml")
179
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
180
  dtype = torch.bfloat16
 
181
  unet, text_encoder, vae, tokenizer, _ = load_trainable_model(
182
  models_path=cfg.paths.models,
183
  device=device,
 
187
  model_cfg=cfg.models,
188
  autocast_dtype=dtype,
189
  )
 
190
  schedule = LinearSchedule(device=device)
 
191
  sample_configs = [
192
  {
193
  "prompt": "1girl, souryuu asuka langley, neon genesis evangelion, "
 
204
  "seed": 42,
205
  }
206
  ]
 
207
  images = generate_samples(
208
  unet=unet,
209
  text_encoder=text_encoder,
 
222
  in_channels=cfg.models.in_channels,
223
  coord_system="aspect_norm",
224
  )
 
225
  images[0].save("output.png")
226
  ```
227
 
 
229
 
230
  ## Content Advisory & Limitations
231
 
232
+ - **Resolution Scope**: Checkpoints up to `epoch_209_*` are natively optimized
233
+ for 512px generation and reliably scale up to 768px/1024px with aspect-normalized
234
+ RoPE. Native 1024px fidelity will be unlocked upon completion of Stage 3.
235
+ - **Complex Hands**: The compact 200M parameter count prioritizes prompt
236
+ faithfulness and character identity; complex hand interactions may still
237
+ require multiple seed attempts.
238
  - **NSFW & Sensitive Content:** The model was trained on a dataset that contains uncensored anime-style art, including sensitive, questionable, and explicit material (`g`, `s`, `q`, `e` ratings). Use appropriate tag filtering during training if a SFW-only model is required.
239
  - **Legal Disclaimer:** Images are sourced from Danbooru for research, educational, and generative modeling purposes under fair use. Copyright of all underlying illustrations belongs to their respective creators.
240
  ---
 
249
 
250
  This model is licensed under the **CreativeML Open RAIL++-M License**. You are
251
  free to use, modify, and distribute the model weights provided you include the
252
+ license text and respect the behavioral use restrictions.