naveenrajk/nemotron35-w4a4-v13

Nemotron-3.5-Lightning-30B-A3B — NVFP4 W4A4, quantized LM head, quantized MTP drafter

This is the checkpoint from my worklog Let's optimize Nemotron 3.5 Lightning: NVIDIA's Nemotron-3.5-Lightning-30B-A3B quantized to W4A4 NVFP4 with the right coverage, plus a quantized LM head and a quantized MTP drafter. Built for one request on one Blackwell GPU, lowest TTFT and TPOT I could get.

The short version of why: at batch size 1 decode is a memory read, so the weights have to get smaller (W4A4 on the routed experts, the layers that were FP4-native in pretraining; FP8 on the dense state-carrying layers; 4-bit weight-only on the LM head). Then the model's built-in MTP head drafts 3 tokens per step so the GPU is no longer stuck at batch size 1, and that head's 128 experts are quantized to the same NVFP4 so drafting doesn't pay 16-bit prices.

Unofficial community checkpoint. Not affiliated with or endorsed by NVIDIA.

Numbers

One RTX PRO 6000 Blackwell Server Edition (sm120, 1.60 TB/s), single request, vLLM 0.28.0rc2. The first two rows are the random-token benchmark (1k in / 128 out); the MTP rows are real text (sonnet, ~540-token prompts), because random tokens make speculative decoding look bad. Medians over repeated sessions. The full step-by-step table is in the post.

config	TPOT	TTFT	tok/s per query	ppl
BF16 base, stock vLLM	5.58 ms	91.5 ms	179	2.502
This repo, no drafter	3.16 ms	104 ms	317	2.551
This repo + MTP k=3 (flags below)	2.21 ms	48 ms	418	lossless
This repo + MTP k=3 + VLLM_USE_V2_MODEL_RUNNER=1	2.0 ms	~45 ms	~455	lossless

"lossless" means speculative decoding: every draft token is verified against the full model, so the output distribution is unchanged.

The post's final step (1.97 ms, ~465 tok/s) records the drafter's draft passes as one CUDA graph and turns prefix caching off. That graph is a small local change to vLLM, not part of this repo. vLLM's opt-in V2 model runner (last row, one environment variable) fixes the same stall its own way and lands at the same ~2.0 ms with stock code, measured back to back in the same container on three boxes.

Drafter acceptance on real text is ~2.8 tokens per step at k=3 (per-position 0.65 / 0.60 / 0.56).

Concurrency is a different problem: this checkpoint does ~4,578 tokens/sec in aggregate at concurrency 256. Speculative decoding starts to hurt once the machine is full, so at high concurrency serve it without the --speculative-config flag.

The ppl column is a 57-token regression check I ran between experiments, not a real eval suite.

What is quantized

scope	precision	count
Routed MoE experts (mixer.experts.*.{up,down}_proj)	NVFP4 W4A4, block 16	5888
MTP draft block routed experts (mtp.*.experts.*)	NVFP4, block 16	256
Mamba in_proj / out_proj, shared experts	FP8 per-tensor W8A8	92
lm_head	NVFP4 weight-only (W4A16), block 16	1
Attention (q/k/v/o), router, conv1d, embeddings, MTP attention/norms	BF16	—
KV cache	BF16	—

Recipe: NVIDIA ModelOpt, "four-over-six" NVFP4 weight scaling with the act_headroom activation calibrator, 512 samples of nemotron-post-training-dataset-v2 at 8k context. The coverage split is ported from NVIDIA's Nemotron-3-Super-120B NVFP4 recipe. The MTP experts were quantized by hand, because the exporter never touches the MTP block, and registered in every place the checkpoint keeps its quantization metadata (vLLM only reads the copy in config.json).

Serving (vLLM ≥ 0.28, Blackwell sm100/sm120)

vllm serve <this-repo> \
  --max-model-len 16384 --max-num-seqs 256 \
  --gpu-memory-utilization 0.95 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3, "quantization": "modelopt_mixed"}' \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [4, 8, 16, 32, 40, 64, 96, 128, 192, 256, 384, 512, 768, 1024]}'

Add VLLM_USE_V2_MODEL_RUNNER=1 to the environment for the last row of the table. If you serve one stream at a time, --no-enable-prefix-caching is also worth a few percent on the default runner (it removes a per-step CPU sync this Mamba hybrid otherwise does).

Four of these settings fail silently if left out:

• "quantization": "modelopt_mixed" in the speculative config. vLLM builds the drafter unquantized by default and bit-casts the packed 4-bit tensors into BF16. The server boots and the text is correct (rejection sampling protects the output), but the drafter proposes noise and acceptance drops to ~1.0. Check the metrics log for Mean acceptance length: ~2.8.
• cudagraph_capture_sizes must be multiples of k+1 (here 4). The default sizes never match speculative decode's step shapes, vLLM falls back to piecewise CUDA graphs, and this ~630-kernel-per-step model slows ~6×.
• --max-num-seqs 256. This is a 23-layer Mamba hybrid; every decode sequence holds a Mamba state block, and vLLM's default of 1024 doesn't fit and kills CUDA graph capture.
• Leave the KV cache in BF16 (the default). --kv-cache-dtype fp8 forces a different attention backend that costs ~0.2 ms/token at low concurrency on this stack.

Confirm the FP4 expert path engaged: Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend in the boot log. The unconditional sm120 warning about "no native FP4 support" can be ignored.

License

Inherits the NVIDIA Open Model License from the base model. Quantization changes weights only; the only data used is the public calibration set named above.