kar-say/SmolVLM-500M-Instruct-GGUF

SmolVLM-500M-Instruct GGUF

GGUF conversion of HuggingFaceTB/SmolVLM-500M-Instruct for llama.cpp. Edge deployable on Raspberry Pi 4 (4 GB).

Files

File	Type	Quantization	Size
SmolVLM-500M-Instruct_Q4_K_M.gguf	Language model	Q4_K_M (4-bit)	289 MB
mmproj-SmolVLM-500M-Instruct.gguf	Vision projector	F16	190 MB

Both files are required for image + text inference. Text-only mode needs only the language model.

Model Info

	Original (HF)	BF16 GGUF	Q4_K_M GGUF
Format	SafeTensors	GGUF	GGUF
Precision	BF16	BF16	4-bit mixed
Text model size	1015 MB	820 MB	289 MB
Vision projector	(included)	190 MB (separate)	190 MB (separate)

• Architecture: VLlama3ForCausalLM (text) + SmolVLM Vision Tower
• Parameters: ~500M
• Context length: 8192
• License: Apache 2.0

Usage

# Server (web UI with image upload)
llama-server \
  -m SmolVLM-500M-Instruct_Q4_K_M.gguf \
  --mmproj mmproj-SmolVLM-500M-Instruct.gguf \
  -c 2048 -t 4 --host 0.0.0.0 --port 8080

# CLI (single image)
llama-cli \
  -m SmolVLM-500M-Instruct_Q4_K_M.gguf \
  --mmproj mmproj-SmolVLM-500M-Instruct.gguf \
  --image photo.jpg \
  -p "Describe this image." \
  -c 2048 -t 4

Raspberry Pi 4 (4 GB)

Total RAM usage ~1.0 GB (text + vision + KV cache). Text generation ~8-14 tok/s. Image encoding ~5-10s per image.

Use 64-bit Raspberry Pi OS, build llama.cpp natively, set -t 4.

Conversion

python convert_hf_to_gguf.py SmolVLM-500M-Instruct/ --outfile SmolVLM-500M-Instruct.gguf
python convert_hf_to_gguf.py SmolVLM-500M-Instruct/ --mmproj --outfile mmproj-SmolVLM-500M-Instruct.gguf
llama-quantize SmolVLM-500M-Instruct.gguf SmolVLM-500M-Instruct_Q4_K_M.gguf Q4_K_M