AIAAH-1-RL-DPO-iter1

Iterative DPO checkpoint of Akahsizrr/AIAAH-1, trained on coding and math preference pairs with speculative-decoding-accelerated rollouts.

Improvement over baseline

Eval	Baseline (AIAAH-1-RL-DPO-iter0)	This checkpoint	Δ
Hard coding (20 problems)	16/20 = 80.0%	18/20 = 90.0%	+10.0%
GSM8K math (30 problems)	28/30 = 93.3%	29/30 = 96.7%	+3.4%
Combined	86.7%	93.3%	+6.7%

This was the only iteration of 5 that improved on the combined coding+math gate; iterations 1-4 regressed and were rolled back.

Training configuration

• Algorithm: Iterative DPO (Direct Preference Optimization) with LoRA
• Iterations: 1 promoted (out of 5 attempted; 4 rolled back)
• Rollouts per iteration: 3,000 (500 prompts × 6 samples)
• Preference pairs: 1,552 (all-pairs construction with low reward-difference threshold)
• DPO steps: 291 (3 epochs, batch 2, grad accum 8 → effective batch 16)
• DPO max length: 4096 tokens
• Rollout max tokens: 1024
• Training tokens: ~6.8M
• DPO loss: 0.003-0.008 (final), rewards/accuracies = 1.0, rewards/margins ≈ 8.5-9.0
• Target model: AIAAH-1 (trainable, LoRA adapter merged into this checkpoint)
• Draft model: Akahsizrr/MiniCPM5-2B-DSpark-continued-step200 — frozen, used only for speculative decoding during rollout generation, not trained and not merged.

Speculative decoding (inference)

Serving uses SGLang with DSPARK speculative decoding:

• Target: this model
• Draft: Akahsizrr/MiniCPM5-2B-DSpark-continued-step200 (frozen, ~323M params)
• Draft block size: 7
• Attention backend: Triton
• Observed on RTX A6000: acceptance rate ~42%, accepted length ~3.97 tokens, aggregate throughput ~2,558 tokens/s at 32 concurrent requests

Dataset provenance

Training prompts were drawn from the supplied coding/reasoning distillation corpora under /root/data. Distillation teachers were strictly limited to the user-approved allowlist:

• Fable 5
• GPT 5.6
• GLM 5.2
• GLM 5.3
• Qwen 3.8 Max
• Kimi K3
• Grok 4.6

No other teachers were permitted. Row-level source labels in the loaded corpora include additional dataset names (e.g. gsm8k, metamathqa, codealpaca, sciq, etc.) used as prompt sources; the distillation teacher restriction applies to generated reasoning traces, not to the underlying public QA datasets.

Evaluation methodology

• Coding: 20 harder algorithmic problems (data structures, edge cases, dynamic programming) executed against test cases. Replaced an earlier 20-question gate that was saturated at 90%.
• Math: 30 GSM8K problems with numeric answer extraction and matching.
• Promotion gate: combined coding+math score must improve by ≥ 0.5% over the current best checkpoint, otherwise the iteration is rolled back.

Hardware

• GPU: RTX A6000 (48 GB VRAM)
• SGLang + DSPARK speculative decoding for rollout generation
• DPO training with LoRA (rank 16) on the target model only

Limitations

• The coding and math evals are small (20 and 30 problems respectively), so each problem is worth 5% and 3.3% — the signal is noisy at this scale.
• Only 1 of 5 iterations improved; the other 4 regressed on coding while math stayed flat, suggesting the coding preference signal is noisy and/or the small eval overfits.
• LiveCodeBench and TerminalBench 2.1 were not run for this checkpoint due to budget constraints.