- Existing approaches often split into specialized systems: DiT-style generators can be strong for static synthesis but struggle with temporal coherence, while MLLM-based editors follow instructions well but may lose pixel-level fidelity.
- Omni Creator 2.0 bridges both by design.
-
-
Core Architectural Innovations
-
-
1) Spatio-Temporal 3D Shifted Window Attention — window partitioning with temporal shifting (Swin-style, adapted for diffusion) to improve long-range temporal dependency modeling and reduce flickering.
-
2) Native Multi-Image Conditioning + Adaptive Fusion — built-in visual projector supports up to 3 concurrent reference images; Adaptive Multi-Modal Gating dynamically balances text, image references, and temporal context.
-
3) HPC-Ready Optimization — production-oriented inference with FP8 support, RoPE, and RMSNorm for stable scaling to longer sequences and higher resolutions.
+
+ 8B-parameter native MM-DiT unifying T2I, pixel-level editing, and I2V generation.
+ Uses CLIP/T5-style text encoders and visual conditioners to deliver high-fidelity multi-modal results with shared transformer backbone.
+
+
+
+
+
+
RoPE + AdaLN-Zero DiT Blocks
+
Multi-head attention with timestep-conditioned modulation
+
+
+
Adaptive Multi-Modal Gating
+
Learned fusion of text, 3× images, and temporal context
+
+
+
HPC-Ready Optimization
+
FP8 + RoPE + RMSNorm for production-scale inference
+
+
+
+
+
+
📊 AdaLN-Zero + AMG (Full Stack)
+
+# 1) Modal fusion (text / up to 3 imgs / temporal)
+α = Softmax(MLP([ctxt; cimg; ctmp])); C = αtxt·ctxt + Σ αi·cimg,i + αtmp·ctmp
+# 2) Time-conditioned modulation
+h = TEmbed(t) ⊕ C; (γ, β, λ) = MLP(h)
+# 3) RoPE self-attn + gated residual
+Q = RoPE(Wq·LN(x)); K = RoPE(Wk·LN(x)); V = Wv·LN(x)
+A = Softmax((QKT/√d) + Brel) · V
+x ← x + λ ⊙ A · (1+γ) + β + CrossAttn(x, C)
+# 4) AdaLN-zero modulated SwiGLU FFN
+u = SwiGLU(W1·LN(x)); x ← x + λ ⊙ (W2·u) · (1+γ) + β
+
+
+
+
+
+
+
+ ⚡ OmniScheduler: Unified Hybrid Flow-Diffusion Sampler
+
+
+ Sampling framework that couples Flow Matching, Karras sigmas, and Heun/RK4 ODE solvers for fast 4–8 step generation.
+
+
+
+
+
+
🎯 Few-Step Flow Matching
+
Supports velocity/epsilon/sample prediction with flow-to-velocity conversion, enabling 4–8 step inference.
+
+
+
🔄 Multi-Stage Sampling
+
Coarse-to-fine pipeline: ~70% steps for coarse draft, optional refine passes for details.
+
+
+
📈 RK4 Hybrid ODE
+
Runge-Kutta 4th order solver with flow matching conditioning for optimal trajectory evolution.
+
+
+
+
+
+
🔢 Flow Matching Formulation
+
+
Given data distribution p₁(x) and noise distribution p₀(z) = N(0, I), the Rectified Flow defines:
+
+ x_t = (1 - t) · z + t · x where t ∈ [0, 1]
+
+
The velocity field v_θ(x_t, t) is trained to match the conditional velocity:
+
+ L_FM = E_{t,x,z}[ ||v_θ(x_t, t) - (x - z)||² ]
+
+
At inference, we solve the ODE: dx/dt = v_θ(x_t, t) from t=0 to t=1
+
+
+
+
+
+
π π-Flow Policy Network (Coarse Trajectory)
+
+
Instead of evaluating the model dozens of times, a lightweight policy network can predict a multi-step velocity trajectory in one forward pass.
+
+
# One-shot trajectory prediction (coarse stage)
+
v_{0:S-1} = π_φ(z₀, c, t_grid)
+ x_{k+1} = x_k + v_k · Δt (k = 0..S-1)
+
+
How it integrates with OmniScheduler:
+
+
Stage-1 (coarse): apply the predicted velocities directly (policy rollout) to rapidly move along the flow-matching path.
+
Stage-2 (refine): optionally switch to Heun/RK4 higher-order updates for detail recovery and stability.
+
Multi-modal conditioning: the policy is conditioned on aggregated text/visual context + time embedding, outputting velocity fields matching latent shape.
-
-
Mechanism Sketch
-
# Fourier timestep embedding (DiT)
-t_freq = [cos(t · ω_1..m), sin(t · ω_1..m)]
-t_emb = MLP(t_freq)
-
-# RoPE + RMSNorm stabilized attention
-Q,K,V = Wq·RMSNorm(x), Wk·RMSNorm(x), Wv·RMSNorm(x)
-Q,K = RoPE(Q), RoPE(K)
-Attn = Softmax((QK^T)/sqrt(d) + B_rel) V
-
-# AdaLN-Zero modulation (diffusion conditioning)
-shift,scale,gate = Linear(t_emb)
-x = x + gate * Attn( (1+scale) ⊙ RMSNorm(x) + shift )
-
-# Multi-modal fusion (text + up to 3 images)
-ctx = concat(E_text, P_visual(I_1..I_3))
-x = x + CrossAttn(RMSNorm(x), ctx)
-
-# High-throughput FFN
-x = x + gate_mlp * SwiGLU(RMSNorm(x))
+
+
+
+
+
🎲 Multi-Stage Sampling Pipeline
+
+
+
📷
+
Input
+
Text + Images
+
+
→
+
+
⚡
+
Stage 1
+
Coarse (≈70%)
+
Euler/Heun, few steps
+
+
→
+
+
💎
+
Stage 2
+
Refine (≈30%)
+
RK4 optional
+
+
→
+
+
🎨
+
Output
+
HD Image/Video
+
-
- """
-
- # 收集所有需要更新的UI组件
- all_ui_components = [
- header_title, news_banner,
- single_edit_tab, multi_edit_tab, t2i_tab, upscale_tab, faceswap_tab, watermark_tab,
- seo_html,
- ]
-
- def update_ui_lang(lang):
- show_banner = lang not in ["zh", "hi", "ru"]
- return {
- header_title: gr.update(value=f"""
-
-
- {t('header_title', lang)}
-
- """),
- news_banner: gr.update(visible=show_banner),
- single_edit_tab: gr.update(label=t("tab_single_image_edit", lang)),
- multi_edit_tab: gr.update(label=t("tab_multi_image_edit", lang)),
- t2i_tab: gr.update(label=t("tab_text_to_image", lang)),
- upscale_tab: gr.update(label=t("tab_image_upscale", lang)),
- faceswap_tab: gr.update(label=t("tab_face_swap", lang)),
- watermark_tab: gr.update(label="🧽 Remove Watermark"),
- seo_html: gr.update(value=get_seo_html(lang)),
- }
+
+
+
+
+ 🧠 Adaptive Multi-Modal Gating (AMG)
+
+
+ Dynamically balancing text, image references, and temporal context through learned gating mechanisms.
+
+
+
+
+# Multi-modal context encoding
+c_text = TextEncoder(prompt) # CLIP/T5 embeddings
+c_imgs = [VisualProjector(img) for img in reference_images] # Up to 3 images
+c_temp = TemporalEncoder(prev_frames) # For video generation
- def on_lang_change(lang):
- return lang, *update_ui_lang(lang).values()
-
- lang_dropdown.change(
- on_lang_change,
- inputs=[lang_dropdown],
- outputs=[lang_state] + all_ui_components
- )
+# Learned gating weights (softmax over modalities)
+α = Softmax(MLP([c_text; c_imgs; c_temp])) # α ∈ R^(2+n_imgs)
- ip_query_state = gr.State({"status": "pending", "ip": None, "lang": "en"})
-
- def on_load_immediate(request: gr.Request):
- client_ip = None
- try:
- client_ip = request.client.host
- headers = dict(request.headers) if hasattr(request, 'headers') else {}
- x_forwarded_for = headers.get('x-forwarded-for') or headers.get('X-Forwarded-For')
- if x_forwarded_for:
- client_ip = x_forwarded_for.split(',')[0].strip()
- except Exception:
- client_ip = "unknown"
-
- try:
- if client_ip in util.IP_Country_Cache:
- cached_lang = get_lang_from_ip(client_ip)
- query_state = {"ip": client_ip, "cached": True}
- return cached_lang, cached_lang, query_state, *update_ui_lang(cached_lang).values()
+# Weighted multi-modal fusion
+context = α_text · c_text + Σᵢ αᵢ · c_imgs[i] + α_temp · c_temp
+
+# Cross-attention with gated context
+h = h + CrossAttn(Q=h, KV=context)
+
+# AdaLN-Zero modulation with timestep + modal gates
+γ, β, α_gate = MLP(t_emb ⊕ modal_gate)
+h = α_gate · LayerNorm(h) · (1 + γ) + β
+
+
+
+
+
💬
+
Text Gate
+
Semantic control
+
+
+
🎨
+
Image Gate
+
Style & identity
+
+
+
🎥
+
Temporal Gate
+
Motion coherence
+
+
+
- detected_lang = get_lang_from_ip(client_ip)
- query_state = {"ip": client_ip, "cached": False}
- return detected_lang, detected_lang, query_state, *update_ui_lang(detected_lang).values()
- except Exception:
- query_state = {"ip": client_ip or "unknown", "cached": False}
- return "en", "en", query_state, *update_ui_lang("en").values()
-
- app.load(
- on_load_immediate,
- inputs=None,
- outputs=[lang_state, lang_dropdown, ip_query_state] + all_ui_components,
- )
+
+
+
+ 📈 State-of-the-Art Model Comparison (2025)
+
+
+
+
+
+ | Model |
+ Params |
+ Architecture |
+ Training |
+ Inference |
+ NFE |
+ Acceleration |
+
+
+
+
+ | FLUX.2-Dev |
+ 32B |
+ DiT + MM |
+ Flow Matching |
+ Euler/DPM |
+ 50 |
+ FP8 + FlashAttn |
+
+
+ | Qwen-Image |
+ 20B |
+ DiT + MLLM |
+ Rectified Flow |
+ FlowMatch Euler |
+ 30-50 |
+ Lightning (4-8步) |
+
+
+ | Qwen-Image-Edit |
+ 20B |
+ DiT + Dual-Branch |
+ Flow Matching |
+ Euler |
+ 28-50 |
+ Prompt Rewrite |
+
+
+ | HunyuanVideo |
+ 13B+ |
+ AsymmDiT |
+ Diffusion |
+ Multi-step |
+ 50+ |
+ FP8 + Multi-frame |
+
+
+ | Wan2.2 |
+ 5B/14B |
+ DiT + MoE |
+ Diffusion |
+ Multi-step |
+ 30-50 |
+ MoE Routing + FP8 |
+
+
+ | Z-Image-Turbo |
+ 6B |
+ Distilled DiT |
+ Progressive Distill |
+ Few-step |
+ 4-8 |
+ Teacher-Student |
+
+
+ | Mochi |
+ 10B |
+ Video DiT |
+ Diffusion |
+ Multi-step |
+ 50+ |
+ ComfyUI Parallel |
+
+
+ | ⭐ Omni Creator 2.0 |
+ 8B |
+ MM-DiT + AMG |
+ π-Flow + FM |
+ RK4 Hybrid |
+ 4-8 |
+ Policy Distill + Multi-Stage |
+
+
+
+
+
+
+
+ Abbreviations:
+ DiT = Diffusion Transformer | MM = Multi-Modal | MLLM = Multimodal LLM | MoE = Mixture of Experts |
+ FM = Flow Matching | NFE = Number of Function Evaluations | AMG = Adaptive Multi-Modal Gating
+
+
+