AxiomicLabs/GPT-S3-3M

GPT-S3-3M is Axiomic Labs' third-generation GPT-S base model, built on a hybrid GDN2/gated GQA architecture with Full Attention Residuals and trained from scratch with TrainWork. At 2.97M parameters, it achieves an Intelligence Index of 9.61 on the Open SLM Leaderboard achieving 1st in the <3m class and 2nd in <10m.

Performance

Relative Leaderboard Performance

Organization	Model	Parameters	Int Index	HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3
Axiomic Labs	GPT-S3-3M	2.97M	9.61	27.68%	34.76%	23.12%	57.02%	39.10%
Tech.us	Tokle-3M	2.91M	8.91	27.20%	34.85%	23.98%	55.01%	40.80%
Bench Labs	pulvis-v2	2.96Mx3	8.62	27.93%	31.86%	22.78%	56.86%	37.40%
Bench Labs	pulvis-v1	2.96Mx3	7.85	27.91%	31.61%	22.27%	55.93%	36.90%
Axiomic Labs	GPT-S2-5M	5.4M	7.12	27.87%	33.92%	22.87%	57.56%	27.90%
Axiomic Labs	GPT-S-5M	5.2M	6.90	27.46%	33.21%	21.16%	57.24%	30.20%
FromZero	ZeroS-Micro-v1.0	2.87M	6.49	28.46%	31.27%	22.01%	53.86%	35.60%
Sol Intelligence	Sol Nano	2.90M	6.07	28.40%	32.07%	21.16%	53.92%	33.80%

The Intelligence Index chance-normalizes HellaSwag, combined ARC (the mean of ARC-Easy and ARC-Challenge), PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.

---

Architecture

Component	Details
Token mixing	Gated DeltaNet-2 (GDN-2) on 6 layers, gated GQA on 2 (layers 4 and 8)
Residual stream	Full Attention Residuals (17 learned depth-wise softmax aggregations)
GDN-2	3 heads x 48, channel-wise erase (b) and write (w) gates, full-rank decay and output-gate projections, short causal conv (kernel 4), L2-normalized q/k
Attention	3 query heads / 1 KV head (3:1), head dim 48, QK-norm, sigmoid output gate
Position encoding	Partial RoPE on 24 of 48 attention dims (theta = 10,000); GDN-2 layers need none
Normalization	RMSNorm (float32 upcast)
Feed-forward	SwiGLU, 352 intermediate (2.44x)
Embedding	Weight tying
Context length	1,024 tokens
Training tokens	26.48B at the released checkpoint; 30B for the complete run
Parameters	2,973,666

Layer layout

embedding
  -> [GDN-2, GDN-2, GDN-2, gated GQA]   x 2
  -> RMSNorm -> tied LM head

Config

vocab_size        = 4,096     (digit-split byte-level BPE)
hidden_size       = 144
num_layers        = 8         (6 GDN-2 + 2 gated GQA)
gdn_heads         = 3 x 48
attention_heads   = 3 query / 1 KV, head_dim 48
rotary_dim        = 24
intermediate      = 352
block_size        = 1,024
rope_theta        = 10,000
total params      = 2,973,666

Parameter Breakdown

Component	Params
Token embeddings (4,096 x 144; LM head tied)	589,824
GDN2 attention blocks (6)	1,007,730
Gated GQA blocks (2)	152,256
SwiGLU MLPs (8)	1,216,512
Block RMSNorms	2,304
Full Attention Residuals (17 aggregation points)	4,896
Final RMSNorm	144
Total	2,973,666

---

Training

GPT-S3 was trained for 114,441 steps on one GPU, with a global batch of 262,144 tokens per step and standard next-token prediction throughout. The complete run processed 30B tokens; the released checkpoint was selected at step 101,000, after about 26.48B tokens, for its best internal Intelligence Index.

Tokenizer

A custom 4,096-token byte-level BPE trained on the training mix, with every digit split into its own token (as in bench-labs/pulvis-v1), plus <|endoftext|>, <|pad|>, <|im_start|>, and <|im_end|>.

Data

The five sources are mixed from step 0 with fixed weights throughout training. Each source was retokenized with GPT-S3's custom tokenizer. Validation uses a held-out ClimbMix shard.

Organization	Source	Weight
NVIDIA	ClimbMix	35%
NVIDIA	OpenMathInstruct-2	15%
IFM	TXT360-v2	35%
OpenBMB	UltraMath	10%
Axiomic Labs	FactBase-DCLM	5%

Optimization

• Optimizer: Hybrid Muon and AdamW. Muon handles 2D hidden weights (max LR 0.02, momentum 0.95, Nesterov, five Newton-Schulz steps); AdamW handles embeddings, convolution kernels, normalization parameters, biases, and other non-matrix parameters (max LR 8e-3, betas 0.9/0.95).
• Weight decay: 0.01 on Muon parameters; AdamW's parameter group uses 0 weight decay.
• Learning-rate schedule: 2,000-step linear warmup, stable until 80% of training, then linear decay to zero.
• Batch size: 262,144 tokens per optimizer step, using 64 sequences of 1,024 tokens per microbatch and four gradient accumulation steps.
• Sequence length: 1,024 tokens throughout training.
• Precision and stability: bfloat16 mixed precision, global gradient-norm clipping at 1.0.

Hardware

• 1x RTX 3080 Ti
• Training time: ~16.5 hours

---

Usage

GPT-S3-3M is a base model for text completion. Give it a passage to continue, such as the beginning of a paragraph. The exported model uses plain PyTorch and does not require the flash-linear-attention training kernels.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


model_name = "AxiomicLabs/GPT-S3-3M"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float32,
    device_map="auto",
)

prompt = "Artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=4,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

---

Citation

@misc{gpts3_2026,
  title={GPT-S3-3M},
  author={Axiomic Labs},
  year={2026},
  howpublished={\url{https://huggingface.co/AxiomicLabs/GPT-S3-3M}},
}