Instructions to use evanrodgers/Nanbeige4.3-33B-W4A16-G128 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use evanrodgers/Nanbeige4.3-33B-W4A16-G128 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="evanrodgers/Nanbeige4.3-33B-W4A16-G128") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("evanrodgers/Nanbeige4.3-33B-W4A16-G128") model = AutoModelForCausalLM.from_pretrained("evanrodgers/Nanbeige4.3-33B-W4A16-G128", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use evanrodgers/Nanbeige4.3-33B-W4A16-G128 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "evanrodgers/Nanbeige4.3-33B-W4A16-G128" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "evanrodgers/Nanbeige4.3-33B-W4A16-G128", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/evanrodgers/Nanbeige4.3-33B-W4A16-G128
- SGLang
How to use evanrodgers/Nanbeige4.3-33B-W4A16-G128 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "evanrodgers/Nanbeige4.3-33B-W4A16-G128" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "evanrodgers/Nanbeige4.3-33B-W4A16-G128", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "evanrodgers/Nanbeige4.3-33B-W4A16-G128" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "evanrodgers/Nanbeige4.3-33B-W4A16-G128", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use evanrodgers/Nanbeige4.3-33B-W4A16-G128 with Docker Model Runner:
docker model run hf.co/evanrodgers/Nanbeige4.3-33B-W4A16-G128
Nanbeige4.3-33B Instruct — W4A16 (INT4 GPTQ, group size 128)
This is a 4-bit weight-only quantization of the instruction-tuned Nanbeige/Nanbeige4.3-33B — the chat / reasoning / tool-use model, not the pretrained Nanbeige4.3-33B-Base. It was made with llmcompressor GPTQ and packaged in compressed-tensors (pack-quantized) for vLLM. Weights are INT4, activations stay BF16, and lm_head is kept in BF16.
- 20.63 GB of weights in 2 shards, down from 66.24 GB for the BF16 instruction-tuned checkpoint (about 3.2× smaller).
- Serving: validated with vLLM 0.31.0 on NVIDIA CMP 170HX (Ampere-class) GPUs. Any Ampere-or-newer GPU with enough VRAM should work.
- Tokenization and chat behavior are unchanged: same tokenizer, chat template, thinking mode, and tool-call format as the upstream instruction-tuned model.
This is a quantized derivative, not a new model. Its capabilities, biases, and limitations come from the upstream instruction-tuned model; see the model card for the full benchmark table and intended use.
Model details
| Item | Value |
|---|---|
| Base model | Nanbeige/Nanbeige4.3-33B (instruction-tuned chat / thinking) |
| Architecture | LlamaForCausalLM, 52 layers, hidden 7168, 56 attention heads / 8 KV heads, head_dim 128, FFN 22016, vocab 166,912, 524,288 max positions, RoPE θ = 1.2e8 |
| Parameters | 33.12B (all retained; quantization changes precision, not parameter count) |
| Quantized modules | All Linear layers except lm_head |
| Scheme | GPTQ W4A16, INT4 symmetric, group size 128, static activation ordering |
| Format | compressed-tensors pack-quantized (auto-detected by vLLM; no --quantization flag needed) |
| Calibration | 512 windows × 2048 tokens = 1.05M tokens, rendered through the model's own chat template; code 40% (Magicoder-OSS-Instruct-75K), tool 20% (glaive-function-calling-v2), chat 20% (ultrachat_200k), Chinese 20% (evol-instruct-chinese), shuffled |
| GPTQ settings | block_size=128, dampening_frac=0.01, actorder="static", sequential per-decoder-layer calibration with CPU offload |
| Output | 2 shards, 20.63 GB of weight data |
| Quantization cost | 2.94 h on one 64 GB CMP 170HX |
| Toolchain | llmcompressor 0.14.0.1, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.13.0 |
Quickstart (vLLM)
Recent vLLM versions auto-detect the compressed-tensors config. This build was validated with vLLM 0.31.0:
pip install "vllm==0.31.0"
vllm serve evanrodgers/Nanbeige4.3-33B-W4A16-G128 \
--served-model-name nanbeige4.3-33b \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Notes:
- Add
--tensor-parallel-size Nto shard across GPUs. With 8 KV heads, values up to 8 divide cleanly. --max-model-len 32768is a safe starting point for a 40 GB card. The model supports 524,288 positions, but KV cache is208 KiB per token in BF16 (104 GiB for a full 512K request), so full context needs multiple large GPUs. We validated 32K on a single 40 GB card and a 262K configuration on 2×64 GB GPUs (PP=2); we did not run full-length quality evaluations.- FP8 KV cache did not work in our validation:
--kv-cache-dtype fp8was unreliable, so leave the KV cache at the default BF16 and do not add that flag. --reasoning-parser qwen3splits the<think>…</think>trace intoreasoning_contentand leaves the answer incontent.--tool-call-parser qwen3_codermatches the base model's recommended tool-calling setup (tool_call_format="xml").
Python client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="nanbeige4.3-33b",
messages=[{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],
temperature=0.6,
top_p=0.95,
max_tokens=32768,
extra_body={
"chat_template_kwargs": {
"enable_thinking": True, # set False for non-thinking mode
"preserve_thinking": False, # True for multi-turn agent/tool workflows
}
},
)
print(resp.choices[0].message.reasoning_content) # thinking trace, if enabled
print(resp.choices[0].message.content) # final answer
Recommended sampling settings from the base model:
| Scenario | Temperature | Top-p | Max new tokens |
|---|---|---|---|
| Agentic and tool use | 1.0 | 0.95 | 65,536 |
| Reasoning and chat | 0.6 | 0.95 | 262,144 |
For tool use, pass tools in the request and keep chat_template_kwargs.tool_call_format at its default "xml" for best tool-calling performance ("json" is also supported for compatibility).
Quantization recipe
The exact llmcompressor recipe is included as recipe.yaml:
default_stage:
default_modifiers:
GPTQModifier:
targets: [Linear]
ignore: [lm_head]
scheme: W4A16
block_size: 128
dampening_frac: 0.01
actorder: static
The calibration set is intentionally deployment-shaped: code, tool-call trajectories, general chat, and Chinese instructions, all rendered with this model's chat template rather than raw text. Calibration used 512 windows of 2048 tokens, shuffled, with one decoder layer resident on the GPU at a time (CPU offload between layers) to fit a single 64 GB card.
Minimal reproduction sketch:
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.recipe import Recipe
recipe = Recipe.from_modifiers(
GPTQModifier(
targets="Linear",
scheme="W4A16",
ignore=["lm_head"],
block_size=128,
dampening_frac=0.01,
actorder="static",
)
)
oneshot(
model=model, # Nanbeige4.3-33B in BF16
tokenizer=tokenizer,
recipe=recipe,
dataset="json",
dataset_path="calib/n512_instruct",
splits="train",
text_column="text",
num_calibration_samples=512,
batch_size=1,
max_seq_length=2048,
shuffle_calibration_samples=True,
sequential_targets=["LlamaDecoderLayer"],
sequential_offload_device="cpu",
tie_word_embeddings=False,
save_compressed=True,
output_dir="Nanbeige4.3-33B-W4A16-G128",
)
Limitations
- Weight-only 4-bit quantization can regress relative to BF16, especially on long-context reasoning, agentic tool use, and code repair. Test your workload.
- Long context is memory-bound. The weights are ~20.6 GB, but a 512K-token KV cache needs ~104 GiB in BF16. Use
--max-model-lenappropriate for your hardware. - No full-suite validation. We did not run the upstream agentic benchmark suite or full-context evaluations on this build; validate on your own workload.
- Transformers
generate(): the upstreamconfig.jsonships withuse_cache: false. This does not affect vLLM, but if you use Transformers directly, passuse_cache=Truetogenerate()(or set it in the config) to avoid recomputing the prefix on every step. - Intended for vLLM / compressed-tensors. Loading the packed INT4 weights through plain Transformers requires a recent
transformers+compressed-tensorsbuild.
License
Apache-2.0, inherited from the base model. See LICENSE. The base model's terms continue to apply.
Citation
If you use this quantized build, please cite the base model and GPTQ:
@misc{nanbeige43chat2026,
title = {Nanbeige4.3-33B},
author = {Nanbeige Team},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}
}
@article{frantar2022gptq,
title = {GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},
author = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},
journal = {arXiv:2210.17323},
year = {2022}
}
- Downloads last month
- 182
Model tree for evanrodgers/Nanbeige4.3-33B-W4A16-G128
Base model
Nanbeige/Nanbeige4.3-33B-Base