Instructions to use local-inference-lab/GLM-5.3-Flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use local-inference-lab/GLM-5.3-Flash-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="local-inference-lab/GLM-5.3-Flash-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("local-inference-lab/GLM-5.3-Flash-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("local-inference-lab/GLM-5.3-Flash-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use local-inference-lab/GLM-5.3-Flash-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "local-inference-lab/GLM-5.3-Flash-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/local-inference-lab/GLM-5.3-Flash-NVFP4

SGLang

How to use local-inference-lab/GLM-5.3-Flash-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "local-inference-lab/GLM-5.3-Flash-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "local-inference-lab/GLM-5.3-Flash-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-Flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use local-inference-lab/GLM-5.3-Flash-NVFP4 with Docker Model Runner:
```
docker model run hf.co/local-inference-lab/GLM-5.3-Flash-NVFP4
```

malaiwah commited on Oct 6

Commit

015fa7c

verified ·

1 Parent(s): 175ae8c

Add original-weights control: zai FP8 tool-eval 92/100 raw, 94.8% matched-88 vs NVFP4 94.3% (n=6)

Browse files

Original-weights control for PR #8's NVFP4 tool-calling receipt: zai FP8 originals on the same Karmic Kraken stack/protocol score 94.8% on the matched 88-scenario subset (n=6, seeds 42/43 x3) vs 94.3% for the NVFP4 checkpoint — quant-neutral, with the deterministic TC-43/51/68 fail trio reproducing 6/6 on both. Includes 6 per-run JSONs under community-evals/.

Files changed (7) hide show

README.md +15 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s42r1.json +0 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s42r2.json +0 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s42r3.json +0 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s43r1.json +0 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s43r2.json +0 -0
community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s43r3.json +0 -0

README.md CHANGED Viewed

@@ -91,3 +91,18 @@ NVFP4/MXFP8 ModelOpt layout, including per-expert activation scales.
 The tokenizer, upstream chat template, generation settings, vision assets
 and MTP tensors are included. KV cache and runtime buffers require memory
 in addition to the model weights.

 The tokenizer, upstream chat template, generation settings, vision assets
 and MTP tensors are included. KV cache and runtime buffers require memory
 in addition to the model weights.
+## Community tool-calling evaluation — original-weights control (2026-10-06)
+An original-weights control for the community tool-calling receipt (PR #8): the unquantized `zai-org/GLM-5.3-Flash` FP8 release weights, served on the same stack and protocol as the NVFP4 receipt and measured with tool-eval-bench hard mode.
+| Datapoint | Scenario set | Score |
+|---|---|---|
+| NVFP4 checkpoint (this repo, PR #8 receipt, 2026-10-05) | v2.7.0, 88 scenarios | 94/100 (166/176, seeds 42 & 43) |
+| **zai FP8 originals, raw (this control, n=6)** | 2.7.1.dev14, 92 scenarios | 92.3/100 mean (91–94) |
+| **zai FP8 originals, matched-88 subset (n=6)** | 88 shared scenarios | **94.8% mean** (92.6–96.0) |
+Protocol: tool-eval-bench hard mode, seeds 42/43 × 3 reps, temperature 0, parallel 4, max-turns 30, timeout 600; the wrapper reports `"differs_from_leaderboard": []` on every run. Stack: Karmic Kraken `karmic-kraken-beta` digest-pinned `2230db60` (production digest), 4× RTX PRO 6000 TP4, MTP3, FP8 KV cache, GMU 0.86, server-side `reasoning_effort: max` + `clear_thinking: true` — the same mechanism as the PR #8 receipt. Per-run JSONs are included under `community-evals/zai-fp8-originals-kk-20261006/`.
+Interpretation: **the NVFP4 quantization is tool-eval-neutral.** Originals score 94.8% vs the checkpoint's 94.3% on the matched subset — within the documented ±3 single-run spread — and the deterministic fail trio TC-43 / TC-51 / TC-68 reproduces 6/6 on the originals, exactly as on the checkpoint. Note the pinned bench version used here serves 92 scenarios (adds TC-89–92) vs the v2.7.0 reference's 88; raw scores are normalized over the 88 shared scenario IDs for comparability.

community-evals/zai-fp8-originals-kk-20261006/tooleval-p88-zai-fp8-s42r1.json ADDED Viewed