Instructions to use vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4")
model = AutoModelForCausalLM.from_pretrained("vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4

SGLang

How to use vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/vipertsniper/DeepSeek-R1-Distill-Qwen-14B-NVFP4
```

DeepSeek-R1-Distill-Qwen-14B-NVFP4 / README.md

vipertsniper

Update README.md

344e229 verified 10 months ago

preview code

Raw History Blame Contribute Delete

1.4 kB

metadata

license: mit
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
tags:
  - blackwell
  - nvfp4
  - nvidia
  - asus-ascent-gx10
  - sm121
  - deepseek-r1
  - quantization
  - qwen
library_name: transformers

DeepSeek-R1-Distill-Qwen-14B-NVFP4 (Work in Progress)

This repository contains a self-quantized version of DeepSeek-R1-Distill-Qwen-14B using the NVIDIA NVFP4 format. This was produced on an Asus Ascent GX10 (NVIDIA GB10 Grace Blackwell) system using the NVIDIA ModelOptimizer playbook.

Hardware & Architecture

Host System: Asus Ascent GX10 (Desktop AI Supercomputer)
Accelerator: NVIDIA Blackwell (SM121 / GB10)
Memory: 128GB Coherent Unified Memory (LPDDR5X)
Format: NVFP4 (4-bit Floating Point) with two-level micro-block scaling.

Current Performance Status (Jan 2026)

Tested on vLLM, but performance on the GX10 is currently inconsistent.

Stuttering: There is a known rhythmic stutter in current vLLM builds when running NVFP4 on SM121.
Work in Progress: This is a byproduct of early-access software kernels. Native Blackwell acceleration in vLLM is expected to improve in future nv25.x releases.

Deployment Details

Quantized using the standard NVIDIA NVFP4 playbook. This format is designed for hardware-level acceleration on 5th Gen Blackwell Tensor Cores.

License

Original weights by DeepSeek-AI are under the MIT License.