deepseek-ai

/

DeepSeek-V4-Flash-Vision-Exp

Like 967

Follow

DeepSeek 154k

Finally got DeepSeek-V4-Flash-Vision-Exp serving properly on my box. No released vLLM supports vision yet, so this is a working recipe built from source + a patched flashinfer — sharing in case it saves anyone the three walls I hit.

Box: 2× RTX PRO 6000 (SM120, 96 GB each), Threadripper 9965X, 128 GB DDR5, Ubuntu 24.04.4, driver 595.84.

Model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (~285B/13B active, FP4+FP8 mixed weights, 43-layer backbone, fp8 KV).

The three problems and the fixes:

1. Vision not in released vLLM → build from the vision + SM120 tree (PR #54566 / #41834).

2. flashinfer SM120 sparse-MLA doesn't handle the vision shapes. Two small patches:

   • Prefill: vision widens the causal-SWA candidates from 128 to 512 and uses 64-token (global/vision) or 2-token (compact text) blocks; the dispatcher only had topk=128. Added the topk=512 dual-layout cases in csrc/sparse_mla_sm120_prefill.cu.
   • Decode: indices come in as strided views; the custom op needs contiguous → added .contiguous() in mla/_sparse_mla_sm120.py.
     Rebuild the wheel, --force-reinstall --no-deps, and remove the stale flashinfer-jit-cache / AOT artifacts or the patched source gets shadowed.

3. KV capacity accounting was lying. Reported 209,015 tokens / 1.59× at 131k (~55 KB/token) when the real footprint is ~4–5 KB/token. That's upstream issue #51041 — the sliding-window spec's in-flight budget was charged per-request × concurrency. The fix is PR #52495 (steady/transient split), which wasn't merged into my build, so I ported it myself into kv_cache_interface.py + kv_cache_utils.py (reserve the transient once, steady-only for capacity math and the startup gate; runtime admission untouched so it can only under-report, never over-commit). After that: 2,576,869 tokens / 7.06× @365k.

Run command:

docker run -d --name deepseek-v4-vision \
  --network host --ipc host --shm-size 32g \
  --gpus '"device=0,1"' --restart unless-stopped \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e NCCL_P2P_DISABLE=1 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v /path/to/deepseek-v4-flash-vision-exp:/model \
  <vision-image> \
  --model /model --tensor-parallel-size 2 --gpu-memory-utilization 0.96 \
  --max-model-len 365000 --kv-cache-memory-bytes 13609922048 \
  --cudagraph-capture-sizes 1 2 4 8 --max-cudagraph-capture-size 8 \
  --max-num-batched-tokens 4096 --max-num-seqs 6 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --limit-mm-per-prompt '{"image":2,"video":0}' --tokenizer-mode deepseek_v4 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 --host 127.0.0.1 --disable-custom-all-reduce

Flags that actually matter:

• --cudagraph-capture-sizes 1 2 4 8 --max-cudagraph-capture-size 8 — mandatory. Default FULL graph capture OOMs at boot (needed ~180 MiB, had ~11 MiB free on the second GPU).
• --max-num-batched-tokens 4096 — capacity goes 7.06× vs 4.77× at the 8192 default.
• --shm-size 32g — tiny default shm makes init hang on shm_broadcast.
• Don't disable flashinfer autotune. Default ON gave a clean warmup.

Boot ~4 min. First image request after cold start is slow (JIT + encoder warmup), normal. Decode batch >8 runs eager by design — fine for interactive use.