Zhangdanyang/Kimi-K3-DSpark-OProj-MXFP4

Kimi-K3 DSpark O-Proj-only MXFP4

This is a selective mixed-precision PTQ checkpoint of Inferact/Kimi-K3-DSpark for speculative decoding of moonshotai/Kimi-K3.

It is not a full-MXFP4 model. Only the attention output projection (self_attn.o_proj) is MXFP4; the other quantized draft projections remain FP8, while embeddings, heads, normalization, and the excluded input/context projections remain BF16.

Exact quantization coverage

The DSpark draft contains five decoder layers. Exactly these five checkpoint weights are MXFP4:

• layers.0.self_attn.o_proj.weight
• layers.1.self_attn.o_proj.weight
• layers.2.self_attn.o_proj.weight
• layers.3.self_attn.o_proj.weight
• layers.4.self_attn.o_proj.weight

Each O-proj weight uses:

• weight format: MXFP4 E2M1 (F4 in safetensors)
• scale format: E8M0 (F8_E8M0 in safetensors)
• grouping: 32 consecutive input-channel values per scale
• stored weight shape per layer: (7168, 8192)
• stored scale shape per layer: (7168, 256)
• ATOM runtime path: dynamic MXFP4 activations with A4W4 dispatch

The following 25 weights remain FP8 E4M3 with FP32 per-output-channel scales and dynamic per-token FP8 activations (ATOM A8W8), five of each projection across layers 0–4:

• mlp.gate_proj.weight
• mlp.up_proj.weight
• mlp.down_proj.weight
• self_attn.q_b_proj.weight
• self_attn.kv_b_proj.weight

The following remain BF16:

• context_proj.weight
• self_attn.q_a_proj.weight and self_attn.kv_a_proj_with_mqa.weight (the runtime fused_qkv_a_proj)
• normalization weights
• markov_head.*
• embed_tokens.weight
• confidence_head.*
• the shared target lm_head

In total, this checkpoint has 5 MXFP4 weights and 25 FP8 weights. The exact tensor lists are also recorded in atom_dspark_ptq_manifest.json, and config.json maps layers.*.self_attn.o_proj to the MXFP4 runtime path.

Why O-proj only

Selective evaluation found that full MXFP4 reduced proposal acceptance and GSM8K accuracy. Restricting MXFP4 to attention O-proj preserved the BF16 GSM8K strict score while reducing checkpoint size:

Metric	BF16 draft	Phase 1 FP8	O-proj-only MXFP4
Checkpoint size	6.67 GiB	4.80 GiB	4.67 GiB
Fixed-length output throughput	626.37 tok/s	667.79 tok/s	643.51 tok/s
Mean TPOT	10.77 ms	9.95 ms	10.38 ms
Speed acceptance rate	50.94%	54.85%	50.23%
Accepted tokens/forward	4.565	4.840	4.516
GSM8K strict accuracy	95.830%	95.603%	95.830%

The fixed-length performance test used 8 GPUs, tensor parallel size 8, concurrency 8, 80 prompts, 1024 input tokens, 512 output tokens, 7 speculative tokens, and FP8 KV cache. BF16, Phase 1, and O-proj-only results are three-run medians.

Across the 14 Inferact natural-stop workloads, request-equal macro mean acceptance length was 4.0119 at temperature=0, compared with 4.0125 for BF16. Under ATOM's temperature=1.0, top_p=0.95 greedy-draft hybrid it was 3.9844, compared with 4.0045 for BF16. The latter is an ATOM-internal comparison and is not equivalent to vLLM probabilistic draft sampling with block rejection.

Runtime requirements

This mixed FP8/MXFP4 layout requires ATOM support for checkpoint-local layer_quant_config dispatch. ATOM reads the global FP8 configuration and overrides layers.*.self_attn.o_proj with MXFP4 A4W4 at load time.

The checkpoint is not intended for stock Transformers, vLLM, or SGLang without equivalent mixed-precision DSpark loader and kernels.