Nanbeige4.3-33B Instruct — W4A16 (INT4 GPTQ, group size 128)

This is a 4-bit weight-only quantization of the instruction-tuned Nanbeige/Nanbeige4.3-33B — the chat / reasoning / tool-use model, not the pretrained Nanbeige4.3-33B-Base. It was made with llmcompressor GPTQ and packaged in compressed-tensors (pack-quantized) for vLLM. Weights are INT4, activations stay BF16, and lm_head is kept in BF16.

  • 20.63 GB of weights in 2 shards, down from 66.24 GB for the BF16 instruction-tuned checkpoint (about 3.2× smaller).
  • Serving: validated with vLLM 0.31.0 on NVIDIA CMP 170HX (Ampere-class) GPUs. Any Ampere-or-newer GPU with enough VRAM should work.
  • Tokenization and chat behavior are unchanged: same tokenizer, chat template, thinking mode, and tool-call format as the upstream instruction-tuned model.

This is a quantized derivative, not a new model. Its capabilities, biases, and limitations come from the upstream instruction-tuned model; see the model card for the full benchmark table and intended use.

Model details

Item Value
Base model Nanbeige/Nanbeige4.3-33B (instruction-tuned chat / thinking)
Architecture LlamaForCausalLM, 52 layers, hidden 7168, 56 attention heads / 8 KV heads, head_dim 128, FFN 22016, vocab 166,912, 524,288 max positions, RoPE θ = 1.2e8
Parameters 33.12B (all retained; quantization changes precision, not parameter count)
Quantized modules All Linear layers except lm_head
Scheme GPTQ W4A16, INT4 symmetric, group size 128, static activation ordering
Format compressed-tensors pack-quantized (auto-detected by vLLM; no --quantization flag needed)
Calibration 512 windows × 2048 tokens = 1.05M tokens, rendered through the model's own chat template; code 40% (Magicoder-OSS-Instruct-75K), tool 20% (glaive-function-calling-v2), chat 20% (ultrachat_200k), Chinese 20% (evol-instruct-chinese), shuffled
GPTQ settings block_size=128, dampening_frac=0.01, actorder="static", sequential per-decoder-layer calibration with CPU offload
Output 2 shards, 20.63 GB of weight data
Quantization cost 2.94 h on one 64 GB CMP 170HX
Toolchain llmcompressor 0.14.0.1, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.13.0

Quickstart (vLLM)

Recent vLLM versions auto-detect the compressed-tensors config. This build was validated with vLLM 0.31.0:

pip install "vllm==0.31.0"
vllm serve evanrodgers/Nanbeige4.3-33B-W4A16-G128 \
  --served-model-name nanbeige4.3-33b \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Notes:

  • Add --tensor-parallel-size N to shard across GPUs. With 8 KV heads, values up to 8 divide cleanly.
  • --max-model-len 32768 is a safe starting point for a 40 GB card. The model supports 524,288 positions, but KV cache is 208 KiB per token in BF16 (104 GiB for a full 512K request), so full context needs multiple large GPUs. We validated 32K on a single 40 GB card and a 262K configuration on 2×64 GB GPUs (PP=2); we did not run full-length quality evaluations.
  • FP8 KV cache did not work in our validation: --kv-cache-dtype fp8 was unreliable, so leave the KV cache at the default BF16 and do not add that flag.
  • --reasoning-parser qwen3 splits the <think>…</think> trace into reasoning_content and leaves the answer in content.
  • --tool-call-parser qwen3_coder matches the base model's recommended tool-calling setup (tool_call_format="xml").

Python client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="nanbeige4.3-33b",
    messages=[{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],
    temperature=0.6,
    top_p=0.95,
    max_tokens=32768,
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": True,      # set False for non-thinking mode
            "preserve_thinking": False,   # True for multi-turn agent/tool workflows
        }
    },
)

print(resp.choices[0].message.reasoning_content)  # thinking trace, if enabled
print(resp.choices[0].message.content)            # final answer

Recommended sampling settings from the base model:

Scenario Temperature Top-p Max new tokens
Agentic and tool use 1.0 0.95 65,536
Reasoning and chat 0.6 0.95 262,144

For tool use, pass tools in the request and keep chat_template_kwargs.tool_call_format at its default "xml" for best tool-calling performance ("json" is also supported for compatibility).

Quantization recipe

The exact llmcompressor recipe is included as recipe.yaml:

default_stage:
  default_modifiers:
    GPTQModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: W4A16
      block_size: 128
      dampening_frac: 0.01
      actorder: static

The calibration set is intentionally deployment-shaped: code, tool-call trajectories, general chat, and Chinese instructions, all rendered with this model's chat template rather than raw text. Calibration used 512 windows of 2048 tokens, shuffled, with one decoder layer resident on the GPU at a time (CPU offload between layers) to fit a single 64 GB card.

Minimal reproduction sketch:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.recipe import Recipe

recipe = Recipe.from_modifiers(
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=["lm_head"],
        block_size=128,
        dampening_frac=0.01,
        actorder="static",
    )
)

oneshot(
    model=model,                      # Nanbeige4.3-33B in BF16
    tokenizer=tokenizer,
    recipe=recipe,
    dataset="json",
    dataset_path="calib/n512_instruct",
    splits="train",
    text_column="text",
    num_calibration_samples=512,
    batch_size=1,
    max_seq_length=2048,
    shuffle_calibration_samples=True,
    sequential_targets=["LlamaDecoderLayer"],
    sequential_offload_device="cpu",
    tie_word_embeddings=False,
    save_compressed=True,
    output_dir="Nanbeige4.3-33B-W4A16-G128",
)

Limitations

  • Weight-only 4-bit quantization can regress relative to BF16, especially on long-context reasoning, agentic tool use, and code repair. Test your workload.
  • Long context is memory-bound. The weights are ~20.6 GB, but a 512K-token KV cache needs ~104 GiB in BF16. Use --max-model-len appropriate for your hardware.
  • No full-suite validation. We did not run the upstream agentic benchmark suite or full-context evaluations on this build; validate on your own workload.
  • Transformers generate(): the upstream config.json ships with use_cache: false. This does not affect vLLM, but if you use Transformers directly, pass use_cache=True to generate() (or set it in the config) to avoid recomputing the prefix on every step.
  • Intended for vLLM / compressed-tensors. Loading the packed INT4 weights through plain Transformers requires a recent transformers + compressed-tensors build.

License

Apache-2.0, inherited from the base model. See LICENSE. The base model's terms continue to apply.

Citation

If you use this quantized build, please cite the base model and GPTQ:

@misc{nanbeige43chat2026,
  title        = {Nanbeige4.3-33B},
  author       = {Nanbeige Team},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}
}

@article{frantar2022gptq,
  title   = {GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},
  author  = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},
  journal = {arXiv:2210.17323},
  year    = {2022}
}
Downloads last month
182
Safetensors
Model size
6B params
Tensor type
BF16
·
I32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for evanrodgers/Nanbeige4.3-33B-W4A16-G128

Quantized
(105)
this model

Paper for evanrodgers/Nanbeige4.3-33B-W4A16-G128