GLM-OCR ยท GGUF (llama.cpp)

GGUF quantizations of zai-org/GLM-OCR for llama.cpp / Ollama / LM Studio. Produced by axteam.

Files

File Size Notes
GLM-OCR-Q4_K_M.gguf 0.5 GB 4-bit k-quant (medium) โ€” recommended size/quality balance.
GLM-OCR-Q8_0.gguf 1.0 GB 8-bit โ€” closest to the original.

Converted with llama.cpp convert_hf_to_gguf.py (recognized as glm4 (text backbone)); the original chat_template is embedded. The vision encoder is not included in these GGUF files โ€” this is the text-backbone LLM only, so it runs only as a text model and cannot do the OCR/image pipeline; it is not a general chat model either (see the check under Requirements). For end-to-end OCR use the original model or the FP8 build axteam/GLM-OCR-FP8-dynamic with vLLM.

Quantization method

These GGUF files are post-training quantization (PTQ) made with llama.cpp (convert_hf_to_gguf.py โ†’ llama-quantize): the BF16 weights are quantized block-wise (k-quants: per-block scales with mixed bit allocation), with no retraining. The original chat_template is embedded in each file.

Measured on GPU (llama.cpp)

llama.cpp b12005 (commit e0f142d, CUDA build) on an NVIDIA B200, 2026-11-02, all layers on the GPU: perplexity over 16 windows of 2,048 tokens from the WikiText-2 test set and from a Korean Wikipedia sample (lower is better). This repo has no BF16 file, so percentages are relative to Q8_0, the file closest to the original.

File WikiText-2 PPL Korean wiki PPL vs Q8_0 (EN / KO)
GLM-OCR-Q8_0.gguf 21.32 18.19 reference
GLM-OCR-Q4_K_M.gguf 22.41 18.39 +5.1% / +1.1%

16 windows per text is a small sample: differences of a few percent between files are within its sampling noise (re-measuring two repos with 64 windows moved such differences by up to ~19 points). This is an OCR model measured on plain text; the absolute values are not comparable with general language models โ€” use them only to compare the files with each other.

Requirements

llama.cpp build โ‰ฅ b8093 โ€” architecture glm4 (GLM-OCR text decoder) needs โ‰ฅ b8093, and the commands on this card use -fa on (โ‰ฅ b6325). Run here on 2026-11-02 with b12005 (commit e0f142d, CUDA build) on an NVIDIA B200: every file in this repo loaded with all layers on the GPU, its perplexity was measured (see Measured on GPU above) and two short chat questions (capital of France / of Korea, one-word answer requested) were asked with every file: every file answered the French one (โ€œParisโ€) but only repeated the Korean question back; this is the text backbone of an OCR model, not a general chat model.
Memory the chosen GGUF file (sizes in the table) plus KV cache for the context (-c); --cache-type-k/v q8_0 -fa on halves the KV cache.
Input text only โ€” no mmproj in this repo, so llama.cpp cannot do OCR / image input with it.
Ollama / LM Studio not tested here.

Run with llama.cpp

# pull one quant straight from the Hub and serve an OpenAI-compatible API on :8080
llama-server -hf axteam/GLM-OCR-GGUF:Q4_K_M -c 8192 --jinja

# or download a single file and run it in the terminal
hf download axteam/GLM-OCR-GGUF GLM-OCR-Q4_K_M.gguf --local-dir .
llama-cli -m GLM-OCR-Q4_K_M.gguf -c 8192 --jinja

These GGUF files hold only the OCR model's text backbone, so they are not general chat models: in our check they answered only the simple French question and repeated the Korean one back (see Requirements). The commands above and below only show how to load and call a file.

Query the server:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "๋Œ€ํ•œ๋ฏผ๊ตญ์˜ ์ˆ˜๋„๋Š” ์–ด๋””์ธ๊ฐ€์š”?"}],
  "max_tokens": 512
}'

Reduce KV-cache memory (optional, run-time)

KV-cache type is chosen at run time in llama.cpp (it is not baked into the file):

llama-server -hf axteam/GLM-OCR-GGUF:Q4_K_M -c 8192 --jinja --cache-type-k q8_0 --cache-type-v q8_0 -fa on

Ollama

ollama run hf.co/axteam/GLM-OCR-GGUF:Q4_K_M

LM Studio

Search for axteam/GLM-OCR-GGUF in the model browser and pick a quant (e.g. Q4_K_M).

License

Quantized derivative of zai-org/GLM-OCR (mit). Original weights & tokenizer belong to Z.ai.


ํ•œ๊ตญ์–ด ์š”์•ฝ

zai-org/GLM-OCR์˜ GGUF ์–‘์žํ™” ๋นŒ๋“œ (llama.cppยทOllamaยทLM Studio์šฉ) ยท ์ œ์ž‘ axteam

๊ฐœ์š”

ํ•ญ๋ชฉ ๋‚ด์šฉ
์›๋ณธ ๋ชจ๋ธ zai-org/GLM-OCR
ํ˜•์‹ GGUF (llama.cpp)
๋ณ€ํ™˜ ๋„๊ตฌ llama.cpp convert_hf_to_gguf.py โ†’ llama-quantize
ํฌํ•จ ํ…์ŠคํŠธ ๋ฐฑ๋ณธ(์–ธ์–ด ๋ชจ๋ธ)๋งŒ โ€” ๋น„์ „ ์ธ์ฝ”๋” ๋ฏธํฌํ•จ, ์ด GGUF๋กœ๋Š” OCR ๋ถˆ๊ฐ€
ํŒŒ์ผ ์ˆ˜ 2๊ฐœ

์–‘์žํ™” ์ƒ์„ธ

  • ์–‘์žํ™” ์ ˆ์ฐจ: PTQ(ํ•™์Šต ํ›„ ์–‘์žํ™”, ์žฌํ•™์Šต ์—†์Œ), llama.cpp ๋ธ”๋ก ๋‹จ์œ„ ์–‘์žํ™”(k-quant Q*_K, Q8_0)
  • chat template: GGUF ํŒŒ์ผ์— ๋‚ด์žฅ
  • ํ’ˆ์งˆ(์‹ค์ธก, 2026-11-02): llama.cpp PPL(WikiText-2ยทํ•œ๊ตญ์–ด ์œ„ํ‚ค ํ‘œ๋ณธ, 2,048ํ† ํฐร—16, B200 GPU, ๋‚ฎ์„์ˆ˜๋ก ์ข‹์Œ, ๋ฆฌํฌ์— BF16 ํŒŒ์ผ์ด ์—†์–ด ์›๋ณธ์— ๊ฐ€์žฅ ๊ฐ€๊นŒ์šด Q8_0 ๊ธฐ์ค€) โ€” GLM-OCR-Q8_0.gguf ์˜์–ด 21.32ยทํ•œ๊ตญ์–ด 18.19 ยท GLM-OCR-Q4_K_M.gguf ์˜์–ด 22.41ยทํ•œ๊ตญ์–ด 18.39(Q8_0 ๋Œ€๋น„ +5.1%ยท+1.1%) โ€” OCR ๋ชจ๋ธ์ด๋ผ ์ ˆ๋Œ€๊ฐ’์€ ์ผ๋ฐ˜ LLM๊ณผ ๋น„๊ตํ•  ์ˆ˜ ์—†๊ณ  ํŒŒ์ผ๋ผ๋ฆฌ ๋น„๊ต์šฉ โ€” ํ…์ŠคํŠธ๋‹น 16์ฒญํฌ๋ผ ํŒŒ์ผ ์‚ฌ์ด ๋ช‡ % ์ฐจ์ด๋Š” ํ‘œ๋ณธ ํ”๋“ค๋ฆผ ๋ฒ”์œ„

์ œ๊ณต ํŒŒ์ผ

ํŒŒ์ผ ํฌ๊ธฐ ์„ค๋ช…
GLM-OCR-Q4_K_M.gguf 0.5 GB 4๋น„ํŠธ k-quant โ€” ํฌ๊ธฐยทํ’ˆ์งˆ ๊ท ํ˜• (๊ถŒ์žฅ)
GLM-OCR-Q8_0.gguf 1.0 GB 8๋น„ํŠธ โ€” ์›๋ณธ์— ๊ฐ€์žฅ ๊ฐ€๊นŒ์›€

์š”๊ตฌ์‚ฌํ•ญ

  • llama.cpp: b8093 ์ด์ƒ (glm4 ๊ตฌ์กฐ(GLM-OCR ํ…์ŠคํŠธ ๋””์ฝ”๋”) ์ง€์› b8093๋ถ€ํ„ฐ, ์˜ˆ์‹œ ๋ช…๋ น์˜ -fa on ์˜ต์…˜ b6325๋ถ€ํ„ฐ)
  • ์‹คํ–‰ ํ™•์ธ: 2026-11-02 b12005(์ปค๋ฐ‹ e0f142d) CUDA ๋นŒ๋“œ๋กœ NVIDIA B200์—์„œ ์ด ๋ฆฌํฌ์˜ ๋ชจ๋“  ํŒŒ์ผ์„ ์ „ ์ธต GPU์— ์˜ฌ๋ ค ์‹คํ–‰ โ€” PPL ์ธก์ •(์œ„ ์˜์–ด ๋ณธ๋ฌธ ํ‘œ), ๋ชจ๋“  ํŒŒ์ผ์— ์งง์€ ์งˆ๋ฌธ 2๊ฐœ(ํ”„๋ž‘์Šคยท๋Œ€ํ•œ๋ฏผ๊ตญ ์ˆ˜๋„, ํ•œ ๋‹จ์–ด ๋‹ต ์š”์ฒญ)๋ฅผ ๋ฌผ์Œ, ํ”„๋ž‘์Šค ์งˆ๋ฌธ์€ ๋ชจ๋‘ โ€œParisโ€๋กœ ๋งžํ˜”์œผ๋‚˜ ํ•œ๊ตญ์–ด ์งˆ๋ฌธ์€ ๋ชจ๋‘ ๋‹ต ๋Œ€์‹  ์งˆ๋ฌธ์„ ๊ทธ๋Œ€๋กœ ๋˜ํ’€์ดํ•จ, OCR ๋ชจ๋ธ์˜ ํ…์ŠคํŠธ ๋ฐฑ๋ณธ์ด๋ผ ์ผ๋ฐ˜ ๋Œ€ํ™”์šฉ ์•„๋‹˜
  • ๋ฉ”๋ชจ๋ฆฌ: ์„ ํƒํ•œ GGUF ํŒŒ์ผ ํฌ๊ธฐ + ์ปจํ…์ŠคํŠธ(-c)๋งŒํผ์˜ KV ์บ์‹œ (KV ์บ์‹œ ์ค„์ด๋Š” ๋ฐฉ๋ฒ•์€ ์•„๋ž˜ ์‚ฌ์šฉ๋ฒ• ์ฐธ๊ณ )
  • ์ž…๋ ฅ: ํ…์ŠคํŠธ ์ „์šฉ(mmproj ์—†์Œ) โ€” OCR์€ ์›๋ณธ ๋ชจ๋ธ์ด๋‚˜ FP8 ๋นŒ๋“œ(axteam/GLM-OCR-FP8-dynamic)๋ฅผ vLLM์œผ๋กœ ์‹คํ–‰
  • OllamaยทLM Studio: ์ œ์ž‘ ์ธก์—์„œ ์‹คํ–‰ ๋ฏธํ™•์ธ

์‚ฌ์šฉ๋ฒ•

  • ์ฃผ์˜: OCR ๋ชจ๋ธ์˜ ํ…์ŠคํŠธ ๋ฐฑ๋ณธ๋งŒ ๋“ค์–ด ์žˆ์–ด ์ผ๋ฐ˜ ๋Œ€ํ™”์šฉ์ด ์•„๋‹˜(์ ๊ฒ€์—์„œ ํ”„๋ž‘์Šค ์งˆ๋ฌธ๋งŒ ๋งžํžˆ๊ณ  ํ•œ๊ตญ์–ด ์งˆ๋ฌธ์€ ๋˜ํ’€์ดํ•จ, ์š”๊ตฌ์‚ฌํ•ญ์˜ ์‹คํ–‰ ํ™•์ธ ์ฐธ๊ณ ) โ€” ์ด ์ ˆ์˜ ๋ช…๋ น์€ ํŒŒ์ผ์„ ๋ถˆ๋Ÿฌ ํ˜ธ์ถœํ•˜๋Š” ๋ฐฉ๋ฒ• ์˜ˆ์‹œ
  • ํŒŒ์ผ ๋ฐ›๊ธฐ: hf download axteam/GLM-OCR-GGUF GLM-OCR-Q4_K_M.gguf --local-dir . (์•„๋ž˜ -m ๋ช…๋ น์— ํ•„์š”, ๋‹ค๋ฅธ ์–‘์žํ™”๋Š” ํŒŒ์ผ๋ช…์˜ Q4_K_M๋งŒ ๋ณ€๊ฒฝ)
  • ์„œ๋ฒ„ ์‹คํ–‰: llama-server -hf axteam/GLM-OCR-GGUF:Q4_K_M -c 8192 --jinja โ€” OpenAI ํ˜ธํ™˜ API
  • ํ„ฐ๋ฏธ๋„ ์‹คํ–‰: llama-cli -m GLM-OCR-Q4_K_M.gguf -c 8192 --jinja
  • Ollama: ollama run hf.co/axteam/GLM-OCR-GGUF:Q4_K_M
  • LM Studio: ๋ชจ๋ธ ๊ฒ€์ƒ‰์—์„œ axteam/GLM-OCR-GGUF ์ž…๋ ฅ ํ›„ ์–‘์žํ™” ๋‹จ๊ณ„ ์„ ํƒ
  • KV ์บ์‹œ ์ ˆ๊ฐ(์„ ํƒ): --cache-type-k q8_0 --cache-type-v q8_0 -fa on โ€” KV ์บ์‹œ ์•ฝ ์ ˆ๋ฐ˜, GGUF ํŒŒ์ผ๊ณผ ์ƒ๊ด€์—†์ด ์‹คํ–‰ ์‹œ ์ง€์ •

๋ผ์ด์„ ์Šค

  • ๋ผ์ด์„ ์Šค: MIT โ€” ์›๋ณธ ๋ชจ๋ธ์˜ ๋ผ์ด์„ ์Šค๋ฅผ ๊ทธ๋Œ€๋กœ ๋”ฐ๋ฆ„
  • ๊ถŒ๋ฆฌ: ์›๋ณธ ๊ฐ€์ค‘์น˜ยทํ† ํฌ๋‚˜์ด์ €๋Š” Z.ai ์†Œ์œ , ์ด ๋ฆฌํฌ๋Š” ์–‘์žํ™” ํŒŒ์ƒ๋ฌผ
Downloads last month
433
GGUF
Model size
0.9B params
Architecture
glm4
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for axteam/GLM-OCR-GGUF

Base model

zai-org/GLM-OCR
Quantized
(41)
this model

Collection including axteam/GLM-OCR-GGUF