Instructions to use moeshawky/Nanbeige4.3-33B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use moeshawky/Nanbeige4.3-33B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use moeshawky/Nanbeige4.3-33B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "moeshawky/Nanbeige4.3-33B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "moeshawky/Nanbeige4.3-33B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
- Ollama
How to use moeshawky/Nanbeige4.3-33B-GGUF with Ollama:
ollama run hf.co/moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use moeshawky/Nanbeige4.3-33B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use moeshawky/Nanbeige4.3-33B-GGUF with Docker Model Runner:
docker model run hf.co/moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
- Lemonade
How to use moeshawky/Nanbeige4.3-33B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Nanbeige4.3-33B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use moeshawky/Nanbeige4.3-33B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use moeshawky/Nanbeige4.3-33B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "moeshawky/Nanbeige4.3-33B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Nanbeige4.3-33B — GGUF Quantized Models
Public repo: moeshawky/Nanbeige4.3-33B-GGUF
Model
| Property | Value |
|---|---|
| Base model | Nanbeige/Nanbeige4.3-33B |
| Architecture | LlamaForCausalLM (dense, GQA) |
| Layers | 52 |
| Hidden size | 7168 |
| Intermediate (FFN) size | 22016 |
| Attention heads | 56 (query), 8 (key/value, 7:1 GQA) |
| Head dim | 128 |
| Vocab size | 166912 |
| Max position embeddings | 524288 |
| Rope theta | 120000000 |
| RMS norm eps | 1e-05 |
| Source dtype | bfloat16 (14 shard safetensors, ~62 GB) |
| tie_word_embeddings | false |
Source checkpoint: /kaggle/input/models/mohamedshawky4340/nanbeige4-3-33b/pytorch/nanbeige4-3-33b/1/ (bfloat16 safetensors).
Quantized files (GGUF, llama.cpp)
All GGUFs were converted from the bf16 safetensors via convert_hf_to_gguf.py --outtype f16 (explicit f16, not auto) to an F16 intermediate, then quantized with /kaggle/working/llama.cpp/build/bin/llama-quantize on 4 CPU threads. Every tensor converted at the requested type — 0 fallbacks in all three quant runs (each of the 471 tensors matched the target block size).
| File | Size | BPW | Type | Source |
|---|---|---|---|---|
Nanbeige4.3-33B.F16.gguf |
66.24 GB | 16.00 | F16 | Intermediate baseline (kept so lower quants can be re-derived) |
Nanbeige4.3-33B.Q8_0.gguf |
35.20 GB | 8.50 | Q8_0 | quantized from F16 |
Nanbeige4.3-33B.Q6_K.gguf |
27.17 GB | 6.56 | Q6_K | quantized from F16 |
Nanbeige4.3-33B.Q4_K_M.gguf |
20.05 GB | 4.84 | Q4_K_M | quantized from F16 |
Quantization command (example, Q4_K_M):
/kaggle/working/llama.cpp/build/bin/llama-quantize \
Nanbeige4.3-33B.F16.gguf \
Nanbeige4.3-33B.Q4_K_M.gguf \
Q4_K_M \
-t 4
Usage
llama.cpp CLI
/kaggle/working/llama.cpp/build/bin/llama-cli \
-m Nanbeige4.3-33B.Q4_K_M.gguf \
-p "The capital of France is" \
-n 128 \
-t 4
llama.cpp server
/kaggle/working/llama.cpp/build/bin/llama-server \
-m Nanbeige4.3-33B.Q6_K.gguf \
--host 0.0.0.0 --port 8080 \
-t 4 \
--ctx-size 4096
Python (llama-cpp-python)
from llama_cpp import Llama
llm = Llama(model_path="Nanbeige4.3-33B.Q4_K_M.gguf", n_ctx=4096, n_threads=4)
print(llm("The capital of France is", max_tokens=128))
Quantization notes
- The F16 intermediate is retained in this repo because every lower-precision quant is derived from it; deleting F16 would force re-conversion from the 62 GB bf16 source.
- Q8_0 is the 8-bit integer quantization (largest of the sub-16-bit options); Q6_K and Q4_K_M are K-quants with mixed block sizes.
- Sizes measured after actual quantization (
llama_model_quantize_implreports quant size = reported file size). No tensor required fallback demotion in any run. - Per the pipeline contract, perplexity measures fluency, not fidelity; no quality claims are made for these quants beyond what llama.cpp reports.
Source of truth
- Conversion/quant binaries:
/kaggle/working/llama.cpp/(checked out at this host) - Source checkpoint: bf16 safetensors at the kaggle path above
- This README last verified: 2026-10-25 (all four GGUFs present in repo, sizes match
llama-quantizeoutput)
Verification commands
# Confirm all four GGUFs are present
hf models info moeshawky/Nanbeige4.3-33B-GGUF --expand siblings | grep -E 'Q8_0|Q6_K|Q4_K_M|F16'
# Re-derive a quant from the F16 intermediate (CPU, 4 threads)
/kaggle/working/llama.cpp/build/bin/llama-quantize \
Nanbeige4.3-33B.F16.gguf /tmp/test.gguf Q4_K_M -t 4
# Expected: quant size ≈ 20.05 GB, 4.84 BPW, 471/471 tensors converted, 0 fallbacks
Documentation produced under the Documentarian skill: every size/BPW figure measured from actual llama-quantize output; no adjectives without a ratio.
- Downloads last month
- 505
4-bit
6-bit
8-bit
16-bit
Model tree for moeshawky/Nanbeige4.3-33B-GGUF
Base model
Nanbeige/Nanbeige4.3-33B-Base