meshapplied/Qwen3.8-Flash-Next-NVFP4-Latest

Qwen3.8-Flash-Next NVFP4

Routed MoE experts in NVFP4 W4A4. Attention and shared experts stay BF16; the MTP (NEXTN) drafter keeps FP8 block-scaled experts. Serves on a single 96 GB Blackwell GPU with SGLang --quantization modelopt_fp4.

Drafter

Path	Role
mtp.* in target shards	NEXTN / MTP drafter
spec/hot_self_32k.pt	Draft lm_head hot-vocab map (--speculative-token-map)

Default: NEXTN with the in-checkpoint MTP block (--speculative-num-steps 3, --speculative-num-draft-tokens 4) plus the hot-vocab map.

Serve: single GPU, concurrency 1

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path . \
  --tp-size 1 \
  --quantization modelopt_fp4 \
  --fp4-gemm-backend flashinfer_cutlass \
  --ple-offload-embedding \
  --ple-offload-backend pinned \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-token-map ./spec/hot_self_32k.pt \
  --mem-fraction-static 0.90 \
  --mamba-full-memory-ratio 0.95 \
  --context-length 16384 \
  --max-running-requests 1 \
  --chunked-prefill-size 2048 \
  --page-size 64 \
  --cuda-graph-backend-decode full \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-max-bs-decode 1 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --trust-remote-code \
  --host 127.0.0.1 --port 31003

Serve: single GPU, concurrency 4

Same command with:

  --mem-fraction-static 0.94 \
  --mamba-full-memory-ratio 0.98 \
  --max-running-requests 4 \
  --chunked-prefill-size 1024 \
  --cuda-graph-max-bs-decode 4

Performance

Hardware unless noted: single RTX PRO 6000 96 GB, TP1, SGLang modelopt_fp4 + flashinfer_cutlass, PLE pinned offload, FP8 KV, NEXTN 3/4 + spec/hot_self_32k.pt, T=0.

VRAM

Serve knobs for this snapshot: mem 0.92, mamba_full 0.98, chunk 8192, context 65536, max_running 1, --disable-radix-cache.

GPU used	89.3 GB (89 274 MiB)
GPU total	95.6 GB (97 887 MiB)
Scheduler process	86.4 GB (88 454 MiB)
max_total_num_tokens	392 448

Agentic coding throughput

Replay of 10 SWE-bench Pro / DeepSWE mini-swe-agent conversations × 6 turns each (run_trace.sh). Full chat history + tool observations fed back each turn; 256 output tokens/turn; prefix cache on (warm turns); hot32; context 16k. Headline concurrency-1 decode rate: 184 tok/s.

Concurrency	out tok/s	accept length	TTFT ms	ITL ms
1	184	3.13	192	4.9
4	334	3.13	354	9.9

Random prompts

sglang.benchmark.serving backend sglang-oai-chat, 32 prompts, in=512, out=256, radix cache on (default serve).

Concurrency	out tok/s	accept length	TTFT ms	ITL ms
1	157	2.58	107	5.7
4	328	2.61	179	10.6

Without the hot-vocab map, concurrency 1 is 130 tok/s on the same random protocol.

Cold prefill and long-context decode

Same GPU/stack with --disable-radix-cache and --flush-cache between lengths so every request is cold. sglang.benchmark.serving random, 8 prompts, out=256, concurrency 1, prompt lengths 1k / 4k / 16k / 32k. Prefill tok/s ≈ prompt_len / mean_TTFT (bench wall-clock input_tok/s mixes decode time).

prompt	decode tok/s	prefill tok/s	TTFT mean / p50 / p90 / p99 (ms)	ITL mean / p50 / p90 / p99 (ms)	accept
1024	152	~9.4k	109 / 98 / 139 / 154	5.8 / 3.7 / 14.4 / 14.9	2.51
4096	133	~19k	214 / 203 / 319 / 320	6.1 / 3.8 / 14.4 / 15.0	2.49
16384	89	~21k	769 / 763 / 1230 / 1474	5.9 / 3.8 / 14.4 / 15.0	2.53
32768	69	~26k	1257 / 1141 / 2123 / 2458	5.8 / 3.8 / 14.3 / 15.1	2.57

Decode slows as context grows; ITL stays flat; TTFT scales with prompt length.

Speculative decoding

knob	value
algorithm	NEXTN (MTP in this checkpoint)
steps	3
draft_tokens	4
eagle_topk	1
accept thresholds	1.0 / 1.0 (lossless vs target)

Format

• Routed experts: NVFP4 W4A4 E2M1, group 16
• Attention, shared experts: BF16
• MTP routed experts: FP8, 128x128 block scales
• PLE n-gram embedding: FP8; serve with --ple-offload-embedding for 1x96 GB fit
• KV: --kv-cache-dtype fp8_e4m3
• Hot map: spec/hot_self_32k.pt

License

See LICENSE and NOTICE. Base model: Qwen license / community terms on Qwen/Qwen3.8-Flash-Next.