One dense giant. Sixty-four specialist minds. Morpho-180B takes the battle-tested open dense foundation and grows it into a routed collective: every token is served by the two domain experts that know it best, plus a shared generalist that never sleeps. Train each expert to convergence. Leave none behind.
⚡ Why Morpho
- 🧠 180B total / ~78B active per request — giant knowledge, agile inference.
- 🎯 64 domain experts, zero overlap — every training example maps to exactly one expert. Auditable by construction.
- 📜 Fully auditable data pipeline — raw → tagged → tokenized corpora, code included, in
Morpho-72B-MoE-Data. - 🔁 Resume-first engineering — every expert checkpoint (weights + loss + step) is versioned in
Morpho-72B-MoE-Checkpoints. Any box can pick up exactly where the last one died. - 🌌 128K context via RoPE YaRN for long documents, codebases, and books.
🏗 Architecture
| Component | Spec |
|---|---|
| Base | 72B dense backbone, 4-bit NF4 QLoRA (frozen) |
| Layers | 80 (40 dense even + 40 MoE odd) |
| Hidden | 8192 · 64 Q heads / 8 KV heads (GQA) · SwiGLU · RMSNorm |
| Router | Learned, top-2 of 8 per MoE layer + 1 shared expert |
| Adapters | LoRA rank-128 / alpha-256 on q,k,v,o,gate,up,down (~1B trainable params) |
| Context | 128K via RoPE YaRN |
| Total / active | ~180B / ~78B per request |
🗺 The 64 experts
Eight sectors — Finance, Law, Medicine, Tech, Science, Business, Arts, Humanities — each with 8 specialists. Full roster with per-expert data counts is logged at every training run; example counts range from ~10K to ~35K per expert, 200K+ examples total, each mapped to exactly one expert.
♻️ Data refinery — every byte auditable
🛠 Build pipeline
flowchart LR
A["1 · Base<br/>72B dense"] --> B["2 · Data<br/>815K raw / 200K+ SFT"]
B --> C["3 · MoE structure<br/>64 routed experts"]
C --> D["4 · SFT<br/>per-expert to convergence"]
D --> E["5 · DPO<br/>preference alignment"]
E --> F["6 · Merge<br/>unified MoE"]
F --> G["7 · Verify<br/>benchmarks + audit"]
G --> H["8 · GGUF<br/>quantized release"]
H --> I["9 · Distill<br/>compact students"]
style D fill:#ff2d78,stroke:#00f0ff,color:#fff
Stage 4 · SFT is COMPLETE — all 61 data-bearing experts converged (phase-decayed LR 2e-4 → 5e-5 → 1e-5, patience 2000 steps, up to 10 epochs). Remaining: rebuild data for 31/52/54, then stage 5 · DPO.
📉 Convergence gallery — every expert trains to its floor
No early exits. Each of the 61 secured experts trained until its loss stopped dropping for 2000 straight steps — 15 of them reached the loss floor below 1e-3. The chart above is drawn from the verified best_loss.txt of every secured adapter.
SFT checkpoint status (live, 2026-10-11 — Master Plan v2)
61/64 secured and converged. SFT is complete for every expert with training data. The last three (rebuild queue below) need data first — nothing is skipped silently.
Provenance: the 2026-09-21/22 storage incident wiped one batch of
best/weights; every affected expert has since been genuinely retrained to convergence and re-verified byte-exact on HF. Old pre-incident loss numbers are reference only, never claims.
| Expert | Domain | Best loss | Weights |
|---|---|---|---|
| 00 | Trading & Markets | 0.0547 | ✅ secured (fresh retrain, converged) |
| 01 | Investing & Portfolio | 0.2746 | ✅ secured (fresh retrain) |
| 02 | Banking & Lending | 0.1307 | ✅ secured (fresh retrain, converged) |
| 03 | Insurance & Risk | 0.4501 | ✅ secured (fresh retrain, converged) |
| 04 | Accounting & Auditing | 0.1500 | ✅ secured (fresh retrain, converged) |
| 05 | Taxation & Compliance | 0.0754 | ✅ secured (fresh retrain, converged) |
| 06 | Real Estate Finance | 0.4678 | ✅ secured (fresh retrain, converged) |
| 07 | Financial Engineering | 0.2545 | ✅ secured (fresh retrain, converged) |
| 08 | Constitutional Law | 0.0692 | ✅ secured (fresh retrain, converged) |
| 09 | Criminal Law | 0.2838 | ✅ secured (fresh retrain, converged) |
| 10 | Civil Law | 0.0955 | ✅ secured (fresh retrain, converged) |
| 11 | Corporate Law | 0.4710 | ✅ secured (fresh retrain) |
| 12 | Intellectual Property | 0.3093 | ✅ secured (fresh retrain, converged) |
| 13 | International Law | 0.3621 | ✅ secured (fresh retrain, converged) |
| 14 | Tax Law | 0.0086 | ✅ secured (fresh retrain) |
| 15 | Regulatory Compliance | 0.5570 | ✅ secured (fresh retrain, converged) |
| 16 | Clinical Medicine | 0.1744 | ✅ secured (fresh retrain, converged) |
| 17 | Pharmacology | 0.0966 | ✅ secured (fresh retrain, converged) |
| 18 | Diagnostics & Imaging | 0.0015 | ✅ secured (fresh retrain, converged) |
| 19 | Surgery & Procedures | 0.0035 | ✅ secured (fresh retrain, converged) |
| 20 | Neurology | 0.2621 | ✅ secured (fresh retrain, converged) |
| 21 | Cardiology | 0.3481 | ✅ secured (fresh retrain, converged) |
| 22 | Oncology | 0.5547 | ✅ secured (fresh retrain, converged) |
| 23 | Public Health | 0.1529 | ✅ secured (fresh retrain, converged) |
| 24 | Machine Learning & AI | 0.0002 | ✅ secured (fresh retrain, converged) |
| 25 | Systems & Infrastructure | 0.0016 | ✅ secured (fresh retrain, converged) |
| 26 | Cybersecurity | 0.0781 | ✅ secured (fresh retrain, converged) |
| 27 | Databases & SQL | 0.1779 | ✅ secured (fresh retrain, converged) |
| 28 | Networking & Cloud | 0.2036 | ✅ secured (fresh retrain, converged) |
| 29 | Web Development | 0.0003 | ✅ secured (fresh retrain, converged) |
| 30 | DevOps & MLOps | 0.000059 | ✅ secured (fresh retrain, converged) |
| 31 | Embedded & IoT | — | 🧱 rebuild queue (261 examples, see below) |
| 32 | Physics | 0.00246 | ✅ secured (fresh retrain, converged) |
| 33 | Chemistry | 0.000000596 | ✅ secured (fresh retrain, loss floor) |
| 34 | Biology & Life Sciences | 0.1974 | ✅ secured (fresh retrain, converged) |
| 35 | Mathematics | 0.3517 | ✅ secured (fresh retrain, converged) |
| 36 | Mechanical Engineering | 0.1394 | ✅ secured (fresh retrain, converged) |
| 37 | Electrical Engineering | 0.0123 | ✅ secured (fresh retrain, converged) |
| 38 | Civil Engineering | 0.1018 | ✅ secured (fresh retrain, converged) |
| 39 | Environmental Science | 0.6756 | ✅ secured (fresh retrain, converged) |
| 40 | Strategic Management | 0.0318 | ✅ secured (fresh retrain, converged) |
| 41 | Marketing & Sales | 0.000285 | ✅ secured (fresh retrain, converged) |
| 42 | Human Resources | 0.00213 | ✅ secured (fresh retrain, converged) |
| 43 | Operations & Logistics | 0.0000397 | ✅ secured (fresh retrain, converged) |
| 44 | Supply Chain | 0.0000859 | ✅ secured (fresh retrain, converged) |
| 45 | Project Management | 0.00004 | ✅ secured (fresh retrain, converged) |
| 46 | Entrepreneurship | 0.0931 | ✅ secured (fresh retrain, converged) |
| 47 | Consulting | 0.2452 | ✅ secured (fresh retrain, converged) |
| 48 | Creative Writing | 0.2855 | ✅ secured (box-24 run, converged) |
| 49 | Graphic Design | 0.0003436 | ✅ secured (box-24 run, loss floor) |
| 50 | Music & Audio | 0.00138 | ✅ secured (box-25 run, converged) |
| 51 | Film & Video | 0.0006447 | ✅ secured (box-25 run, loss floor) |
| 52 | Translation & Localization | — | 🧱 rebuild queue (246 examples, see below) |
| 53 | Linguistics | 0.0138 | ✅ secured (box-26 run, converged) |
| 54 | Storytelling & Narrative | — | 🧱 rebuild queue (240 examples, see below) |
| 55 | Content Creation | 0.0002578 | ✅ secured (box-27 run, loss floor) |
| 56 | Pedagogy | 0.4577 | ✅ secured (box-27 run, converged) |
| 57 | Psychology | 0.5472 | ✅ secured (box-27 run, converged) |
| 58 | Sociology | 0.00000834 | ✅ secured (box-27 run, loss floor) |
| 59 | Economics | 0.00013 | ✅ secured (box-30 run, loss floor, bf16) |
| 60 | Philosophy | 0.4239 | ✅ secured (box-39 run, converged, bf16) |
| 61 | History | 0.3632 | ✅ secured (box-39 run, converged, bf16) |
| 62 | Political Science | 0.0000000397 | ✅ secured (box-40 run, loss floor, bf16) |
| 63 | Critical Analysis & Reasoning | 0.0000229 | ✅ secured (box-40 run, loss floor, bf16) |
Precision note: adapters 00–58 are stored fp32 (6.7GB); from 59 on, best-saves are bf16 (3.4GB, max cast diff 6.1e-05) so pushes survive box egress. Merge casts everything to a common dtype — no effect on the final model.
Storage note (Oct 2026): the quota wall that stalled experts 59–61 is resolved — 8.28TB of orphaned LFS objects purged (byte-exact keep-set of the 57 live adapters verified before/after), history squashed, namespace trimmed to the 4 project repos. All 59 adapters live in this repo's checkpoint companion; nothing was lost.
🧱 Rebuild queue — who needs more data and why
Three experts came out of the v5 tagging with too little data to train a stable LoRA adapter. The pipeline enforces a 500-example minimum: below that, a rank-128 adapter memorizes instead of generalizing, and the "converged" loss would be a lie. So these three are parked — not skipped — until their data is rebuilt:
| Expert | Domain | Today | Why it's short | Rebuild plan |
|---|---|---|---|---|
| 31 | Embedded & IoT | 261 ex. | Niche hardware Q&A is thin across the 17 sources | Targeted collection (Arduino / IoT StackExchange-style Q&A) → tag → tokenize → upload, then train |
| 52 | Translation & Localization | 246 ex. | Parallel-sentence pairs under-tagged in v5 pass | Targeted collection (gated parallel corpora, e.g. Tatoeba-style) → tag → tokenize → upload, then train |
| 54 | Storytelling & Narrative | 240 ex. | Long-form stories split across sectors in v5 pass | Targeted collection (prompt + story pairs, e.g. WritingPrompts-style) → tag → tokenize → upload, then train |
Rules for the rebuild: real data only (no synthesis), same bluemorpholimited/Morpho-72B-MoE-Data repo, same audit trail (raw → tagged → tokenized + code), same one-example-one-expert mapping, same convergence bar (patience 2000). When each crosses 500 verified examples it joins the training queue like any other expert.
🔄 Collection running now (box 41): Arduino StackExchange + microcontroller multiturn → 31; OPUS-100 bitext (es/fr/de/it-en) → 52; WritingPrompts → 54. Tag → tokenize → upload → train follows automatically.
🔥 Live training dashboard
Checkpoints (weights + config + loss): Morpho-72B-MoE-Checkpoints · Data + code: Morpho-72B-MoE-Data
📊 Evaluation
End-to-end benchmarks (MMLU, GSM8K, HumanEval, MT-Bench + per-expert domain suites) run at stage 7 · Verify, after the merge. No scores are published before then — any numbers you see elsewhere are not from this project. This card will be updated with the full scoreboard on merge.
🚀 Usage (after merge)
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("bluemorpholimited/Morpho-72B-MoE", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
"bluemorpholimited/Morpho-72B-MoE",
device_map="auto", trust_remote_code=True,
)
out = model.generate(**tok("Explain covered calls like a trader:", return_tensors="pt"), max_new_tokens=256)
print(tok.decode(out[0]))
⚠️ Today this repo hosts the model card + visual assets. Weights land here at stage 6 (merge). Per-expert SFT adapters are already available in the Checkpoints repo for research use.
🧬 Reproduce it
All data, tags, tokenized batches, and training code are public in the Data repo — 815K raw examples across 17 source datasets, deterministically mapped to 64 experts. The SFT loop (train_sft_experts.py) is resume-safe: kill the box mid-run and the next one continues from the exact best checkpoint.
⚖️ Limitations & safety
- SFT-stage adapters are domain specialists, not yet aligned (DPO pending) — expect raw, unfiltered completions in research use.
- 4-bit base + LoRA trades some precision for trainability on free-tier GPUs.
- Training data is web-scale; domain tags are heuristic — audit logs are provided for verification, not as a guarantee.
📄 License & credit
- Weights/code: Apache 2.0.
- Built by Blue Morpho — trained on free public compute (Hugging Face + molab GPU boxes), proving industrial-grade MoE training doesn't need a datacenter.
- If you use Morpho artifacts, cite:
bluemorpholimited/Morpho-72B-MoE (2026).
🔷 NO EXPERT LEFT BEHIND 🔷