Nanbeige4.3-33B — GGUF Quantized Models

Public repo: moeshawky/Nanbeige4.3-33B-GGUF


Model

Property Value
Base model Nanbeige/Nanbeige4.3-33B
Architecture LlamaForCausalLM (dense, GQA)
Layers 52
Hidden size 7168
Intermediate (FFN) size 22016
Attention heads 56 (query), 8 (key/value, 7:1 GQA)
Head dim 128
Vocab size 166912
Max position embeddings 524288
Rope theta 120000000
RMS norm eps 1e-05
Source dtype bfloat16 (14 shard safetensors, ~62 GB)
tie_word_embeddings false

Source checkpoint: /kaggle/input/models/mohamedshawky4340/nanbeige4-3-33b/pytorch/nanbeige4-3-33b/1/ (bfloat16 safetensors).


Quantized files (GGUF, llama.cpp)

All GGUFs were converted from the bf16 safetensors via convert_hf_to_gguf.py --outtype f16 (explicit f16, not auto) to an F16 intermediate, then quantized with /kaggle/working/llama.cpp/build/bin/llama-quantize on 4 CPU threads. Every tensor converted at the requested type — 0 fallbacks in all three quant runs (each of the 471 tensors matched the target block size).

File Size BPW Type Source
Nanbeige4.3-33B.F16.gguf 66.24 GB 16.00 F16 Intermediate baseline (kept so lower quants can be re-derived)
Nanbeige4.3-33B.Q8_0.gguf 35.20 GB 8.50 Q8_0 quantized from F16
Nanbeige4.3-33B.Q6_K.gguf 27.17 GB 6.56 Q6_K quantized from F16
Nanbeige4.3-33B.Q4_K_M.gguf 20.05 GB 4.84 Q4_K_M quantized from F16

Quantization command (example, Q4_K_M):

/kaggle/working/llama.cpp/build/bin/llama-quantize \
  Nanbeige4.3-33B.F16.gguf \
  Nanbeige4.3-33B.Q4_K_M.gguf \
  Q4_K_M \
  -t 4

Usage

llama.cpp CLI

/kaggle/working/llama.cpp/build/bin/llama-cli \
  -m Nanbeige4.3-33B.Q4_K_M.gguf \
  -p "The capital of France is" \
  -n 128 \
  -t 4

llama.cpp server

/kaggle/working/llama.cpp/build/bin/llama-server \
  -m Nanbeige4.3-33B.Q6_K.gguf \
  --host 0.0.0.0 --port 8080 \
  -t 4 \
  --ctx-size 4096

Python (llama-cpp-python)

from llama_cpp import Llama
llm = Llama(model_path="Nanbeige4.3-33B.Q4_K_M.gguf", n_ctx=4096, n_threads=4)
print(llm("The capital of France is", max_tokens=128))

Quantization notes

  • The F16 intermediate is retained in this repo because every lower-precision quant is derived from it; deleting F16 would force re-conversion from the 62 GB bf16 source.
  • Q8_0 is the 8-bit integer quantization (largest of the sub-16-bit options); Q6_K and Q4_K_M are K-quants with mixed block sizes.
  • Sizes measured after actual quantization (llama_model_quantize_impl reports quant size = reported file size). No tensor required fallback demotion in any run.
  • Per the pipeline contract, perplexity measures fluency, not fidelity; no quality claims are made for these quants beyond what llama.cpp reports.

Source of truth

  • Conversion/quant binaries: /kaggle/working/llama.cpp/ (checked out at this host)
  • Source checkpoint: bf16 safetensors at the kaggle path above
  • This README last verified: 2026-10-25 (all four GGUFs present in repo, sizes match llama-quantize output)

Verification commands

# Confirm all four GGUFs are present
hf models info moeshawky/Nanbeige4.3-33B-GGUF --expand siblings | grep -E 'Q8_0|Q6_K|Q4_K_M|F16'

# Re-derive a quant from the F16 intermediate (CPU, 4 threads)
/kaggle/working/llama.cpp/build/bin/llama-quantize \
  Nanbeige4.3-33B.F16.gguf /tmp/test.gguf Q4_K_M -t 4
# Expected: quant size ≈ 20.05 GB, 4.84 BPW, 471/471 tensors converted, 0 fallbacks

Documentation produced under the Documentarian skill: every size/BPW figure measured from actual llama-quantize output; no adjectives without a ratio.

Downloads last month
505
GGUF
Model size
33B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

4-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for moeshawky/Nanbeige4.3-33B-GGUF

Quantized
(105)
this model