Edge0-120B latent sparse MoE architectureThe full model stack appears on the left, one transformer layer is expanded in the center, and parameter specifications, MTP, expert records and the legend appear on the right. active parameters per token are 8.64B.Edge0-120BLatent sparse MoE on Qwen3.8-Flash-Next · 48 layers · 512 experts · 10 routed per tokeninput tokensEmbeddingPLE (fp8)48 × transformer layerhybrid attentionfull every 4 layers + GDNsparse indexerlatent MoE← detail on the rightFinal normlm_head · 248Klogitsone layer · latent MoE detail · ℓ = 1280h∈ ℝT × 2560+QSA · GDN hybrid attentionruns at the full 2560 width+D · 2560 → ℓ · 8 groups (shared)router · top-10 / 512512 routed experts10 active + 1 shared per tokenΣU · ℓ → 2560 (shared)+→ next layerℓspecstotal 120Bactive / token 8.64B48 × 512 experts 60.4BPLE 51.2B · MTP 1.26Bcontext 256KMTP draft head 1.26Bextra head, 512 experts,trained with the model andused for speculative decodeone expertℓ → 640 → ℓ, SwiGLU1,164,800 B per recordpinned, swapped orstreamed onto the GPUlegendfull 2560 widthlatent width ℓidle expertoptional / side pathAttention, embedding and the shared D/U projections keep the full 2560 width; the 512 routed experts read and write ℓ only, which is what makes one expert a fixed-size record.{"active_parameters_b": 8.64, "baseline": "content/numeric-baseline.json", "baseline_sha256": "e5a4aafbd4e96388ad01c847a6779efa8c43ba78de4bedcb380689fb75986267", "edges": [{"arrow": false, "end": [206, 288], "from": "embedding", "id": "embedding-to-merge", "role": "flow", "start": [121, 256], "to": "merge"}, {"arrow": false, "end": [206, 288], "from": "ple", "id": "ple-to-merge", "role": "flow", "start": [291, 256], "to": "merge"}, {"arrow": true, "end": [206, 320], "from": "merge", "id": "merge-to-layer", "role": "flow", "start": [206, 291.5], "to": "layer"}, {"arrow": true, "end": [206, 610], "from": "layer", "id": "layer-to-final_norm", "role": "flow", "start": [206, 570], "to": "final_norm"}, {"arrow": true, "end": [206, 714], "from": "final_norm", "id": "final_norm-to-lm_head", "role": "flow", "start": [206, 666], "to": "lm_head"}, {"arrow": true, "end": [206, 813.5], "from": "lm_head", "id": "lm_head-to-logits", "role": "flow", "start": [206, 770], "to": "logits"}, {"arrow": true, "end": [840, 277], "from": "hidden", "id": "hidden-to-add_input", "role": "flow", "start": [840, 252], "to": "add_input"}, {"arrow": true, "end": [840, 338], "from": "add_input", "id": "add_input-to-attention", "role": "flow", "start": [840, 311], "to": "attention"}, {"arrow": true, "end": [840, 439], "from": "attention", "id": "attention-to-add_attention", "role": "flow", "start": [840, 414], "to": "add_attention"}, {"arrow": true, "end": [840, 500], "from": "add_attention", "id": "add_attention-to-D", "role": "flow", "start": [840, 473], "to": "D"}, {"arrow": true, "end": [640, 610], "from": "D", "id": "D-to-router", "role": "routing", "start": [640, 564], "to": "router"}, {"arrow": true, "end": [840, 590], "from": "D", "id": "D-to-experts", "role": "routing", "start": [840, 564], "to": "experts"}, {"arrow": true, "end": [794, 642], "from": "router", "id": "router-to-experts", "role": "routing", "start": [760, 642], "to": "experts"}, {"arrow": true, "end": [858, 766], "from": "experts", "id": "experts-to-aggregate", "role": "flow", "start": [977, 714], "to": "aggregate"}, {"arrow": true, "end": [840, 818], "from": "aggregate", "id": "aggregate-to-U", "role": "flow", "start": [840, 784], "to": "U"}, {"arrow": true, "end": [840, 911], "from": "U", "id": "U-to-add_output", "role": "flow", "start": [840, 882], "to": "add_output"}, {"arrow": true, "end": [840, 993.5], "from": "add_output", "id": "add_output-to-next_layer", "role": "flow", "start": [840, 945], "to": "next_layer"}, {"arrow": false, "end": [1220, 221], "from": "hidden", "id": "hidden-to-residual_bus", "role": "residual", "start": [1160, 221], "to": "residual_bus"}, {"arrow": false, "end": [857, 294], "from": "residual_bus", "id": "residual_bus-to-add_input", "role": "residual", "start": [1220, 294], "to": "add_input"}, {"arrow": false, "end": [857, 456], "from": "residual_bus", "id": "residual_bus-to-add_attention", "role": "residual", "start": [1220, 456], "to": "add_attention"}, {"arrow": false, "end": [857, 928], "from": "residual_bus", "id": "residual_bus-to-add_output", "role": "residual", "start": [1220, 928], "to": "add_output"}], "model": "Edge0-120B", "nodes": {"D": {"bounds": [520, 500, 640, 64], "shape": "rect"}, "U": {"bounds": [520, 818, 640, 64], "shape": "rect"}, "add_attention": {"cx": 840, "cy": 456, "r": 17, "shape": "circle"}, "add_input": {"cx": 840, "cy": 294, "r": 17, "shape": "circle"}, "add_output": {"cx": 840, "cy": 928, "r": 17, "shape": "circle"}, "aggregate": {"cx": 840, "cy": 766, "r": 18, "shape": "circle"}, "attention": {"bounds": [520, 338, 640, 76], "shape": "rect"}, "embedding": {"bounds": [48, 200, 146, 56], "shape": "rect"}, "experts": {"bounds": [794, 590, 366, 124], "shape": "rect"}, "final_norm": {"bounds": [48, 610, 316, 56], "shape": "rect"}, "hidden": {"bounds": [520, 190, 640, 62], "shape": "rect"}, "layer": {"bounds": [48, 320, 316, 250], "shape": "rect"}, "lm_head": {"bounds": [48, 714, 316, 56], "shape": "rect"}, "logits": {"cx": 206, "cy": 816, "r": 2.5, "shape": "circle"}, "merge": {"cx": 206, "cy": 288, "r": 3.5, "shape": "circle"}, "next_layer": {"cx": 840, "cy": 996, "r": 2.5, "shape": "circle"}, "ple": {"bounds": [218, 200, 146, 56], "shape": "rect"}, "residual_bus": {"bounds": [1219, 221, 2, 707], "shape": "rect"}, "router": {"bounds": [520, 610, 240, 64], "shape": "rect"}}, "source": "sources/hf-page-draft/Edge0-120B/assets/arch.html", "source_png": "sources/hf-page-draft/Edge0-120B/assets/arch.png", "source_png_sha256": "e75a200b93040d963a3e6daa530ac2c334295d25ed355304d9a4aecce1d26529", "source_sha256": "439478cd6909768684dcc3b344529c93cd3257330947d3fe67df7ecec91bdff5", "style": "assets/source/card-style.json", "style_sha256": "91e2b6c4f1141af837ddce11c23e3981af6909ef8db25b5cc078f110215ef1dd", "text_count": 52, "text_inventory": ["Edge0-120B", "Latent sparse MoE on Qwen3.8-Flash-Next · 48 layers · 512 experts · 10 routed per token", "input tokens", "Embedding", "PLE (fp8)", "48 × transformer layer", "hybrid attention", "full every 4 layers + GDN", "sparse indexer", "latent MoE", "← detail on the right", "Final norm", "lm_head · 248K", "logits", "one layer · latent MoE detail · ℓ = 1280", "h", "∈ ℝ", "T × 2560", "+", "QSA · GDN hybrid attention", "runs at the full 2560 width", "+", "D · 2560 → ℓ · 8 groups (shared)", "router · top-10 / 512", "512 routed experts", "10 active + 1 shared per token", "Σ", "U · ℓ → 2560 (shared)", "+", "→ next layer", "ℓ", "specs", "total 120B", "active / token 8.64B", "48 × 512 experts 60.4B", "PLE 51.2B · MTP 1.26B", "context 256K", "MTP draft head 1.26B", "extra head, 512 experts,", "trained with the model and", "used for speculative decode", "one expert", "ℓ → 640 → ℓ, SwiGLU", "1,164,800 B per record", "pinned, swapped or", "streamed onto the GPU", "legend", "full 2560 width", "latent width ℓ", "idle expert", "optional / side path", "Attention, embedding and the shared D/U projections keep the full 2560 width; the 512 routed experts read and write ℓ only, which is what makes one expert a fixed-size record."], "verification": "Faithful technical-text and connectivity redraw; no independent architecture or checkpoint validation."}