Instructions to use deepseek-ai/DeepSeek-V3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V3", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V3", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V3

SGLang

How to use deepseek-ai/DeepSeek-V3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V3 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V3
```

Fix chat_template crash on tool-call messages without a `content` key

#134

by albertvillanova HF Staff - opened 2 days ago

base: refs/heads/main

←

from: refs/pr/134

Discussion Files changed

-1

albertvillanova

2 days ago

•

edited 2 days ago

Tracked in https://github.com/huggingface/trl/issues/8280, together with the same fix on the other deepseek-ai repos.

Supersedes #122, which changes only the first of the two checks below: with #122 alone, the template still crashes in the second.

Since huggingface/transformers#45422, from transformers 5.23, apply_chat_template drops None-valued message keys. An OpenAI-style tool-call message ("content": None with "tool_calls") then reaches the template without a content key, and this template crashes:

UndefinedError: 'dict object' has no attribute 'content'

Cause

The template reads message['content'] in two checks, is none and is not none. Neither handles a missing key: is none is false and is not none is true, so the template renders the message as plain text and fails to concatenate the missing content.

Fix

Read the content with message.get('content') in both checks, so a missing key behaves exactly like content=None:

- {%- if message['role'] == 'assistant' and message['content'] is none %}
+ {%- if message['role'] == 'assistant' and message.get('content') is none %}
  ...
- {%- if message['role'] == 'assistant' and message['content'] is not none %}
+ {%- if message['role'] == 'assistant' and message.get('content') is not none %}

Reproduction

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
tok.apply_chat_template(
    [
        {"role": "user", "content": "What's the weather in Paris?"},
        {"role": "assistant", "content": None, "tool_calls": [{
            "type": "function",
            "function": {"name": "get_weather", "arguments": '{"city":"Paris"}'},
        }]},
    ],
    tokenize=False,
)

On transformers main, this raises with the current template and with #122, and renders correctly with this PR. It also raises on released transformers if the content key is omitted.

Verification

Rendered with transformers before and after huggingface/transformers#45422: a single tool call, a multi-call tool flow with tool outputs and a final answer, a tool-call message without a content key, a plain chat with reasoning, and an empty assistant message. Wherever the current template renders, the output of this PR is byte-identical to it, so nothing changes for existing users. The cases that crash today render correctly.

Fix chat_template crash on tool-call messages without a `content` key9acbfc6b

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment