Instructions to use axteam/Qwen3.5-0.8B-AWQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/Qwen3.5-0.8B-AWQ-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="axteam/Qwen3.5-0.8B-AWQ-4bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("axteam/Qwen3.5-0.8B-AWQ-4bit")
model = AutoModelForMultimodalLM.from_pretrained("axteam/Qwen3.5-0.8B-AWQ-4bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/Qwen3.5-0.8B-AWQ-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/Qwen3.5-0.8B-AWQ-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-0.8B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/Qwen3.5-0.8B-AWQ-4bit

SGLang

How to use axteam/Qwen3.5-0.8B-AWQ-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/Qwen3.5-0.8B-AWQ-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-0.8B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/Qwen3.5-0.8B-AWQ-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-0.8B-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use axteam/Qwen3.5-0.8B-AWQ-4bit with Docker Model Runner:
```
docker model run hf.co/axteam/Qwen3.5-0.8B-AWQ-4bit
```

Qwen3.5-0.8B · AWQ INT4 (W4A16)

4-bit quantization of Qwen/Qwen3.5-0.8B, produced for efficient vLLM serving. Quantized by axteam.

Revision 2026-11-01: un-smoothed RMSNorm weights restored bit-exactly from the original. llm-compressor's offset-norm handling had left BF16 rounding noise (at most 0.4% in the effective gain) in the norms that AWQ does not smooth. Quantized weights and the AWQ-smoothed norms are unchanged. The measurements on this card were taken before this change.


Base model	`Qwen/Qwen3.5-0.8B`
Method	AWQ INT4 (W4A16)
Format	compressed-tensors `pack-quantized`
Weights / activations	W4A16 — INT4 asymmetric weights, group size 32 (BF16 scale + INT4 zero-point), BF16 activations
Architecture	Hybrid (linear-attention + full-attention) · Dense · multimodal (vision) · MTP head
Size on disk	~1.0 GB
Calibration	256 samples (Korean-heavy mixed corpus)
License	Apache-2.0

Quantization details

Scheme — W4A16 — INT4 asymmetric weights, group size 32 (BF16 scale + INT4 zero-point), BF16 activations, stored as compressed-tensors pack-quantized.
Kept in BF16 — the vision tower & projector (model.visual.*), the multi-token-prediction head (mtp.*), all RMSNorms, token embeddings, the LM head, the SSM non-linear parts (linear_attn.conv1d/A_log/dt_bias). linear_attn.A_log and the gated linear_attn.norm, FP32 in the original, are stored in BF16 — losslessly for A_log (every original value is exactly representable in BF16), and for the norm the same values vLLM and Transformers (default dtype) load at runtime. Left unquantized to preserve quality (about 43% of all parameters in this build).
Tokenizer / chat_template / preprocessor configs are identical to the original (the original ships no generation_config.json, so none is included here). config.json is the original plus a quantization_config block.

This is post-training quantization (PTQ) — no retraining. AWQ uses a small calibration set (no gradient) to protect salient weight channels before rounding.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	AWQ-8bit	NVFP4A16	NVFP4	AWQ-4bit (this repo)
MMLU (5-shot, generative)	50.56	50.50 (99.9%)	50.31 (99.5%)	48.39 (95.7%)	46.36 (91.7%)	48.55 (96.0%)
GSM8K (5-shot, strict)	32.68	29.80 (91.2%)	32.30 (98.8%)	29.49 (90.3%)	22.82 (69.8%)	28.20 (86.3%)
ARC-Challenge (0-shot)	37.54	36.52 (97.3%)	37.46 (99.8%)	35.07 (93.4%)	33.11 (88.2%)	35.15 (93.6%)
Winogrande (5-shot)	56.27	55.49 (98.6%)	56.67 (100.7%)	55.80 (99.2%)	55.64 (98.9%)	55.88 (99.3%)
KMMLU (5-shot, direct)	34.54	34.22 (99.1%)	34.20 (99.0%)	35.02 (101.4%)	33.51 (97.0%)	35.74 (103.5%)
HAE-RAE (0-shot)	37.03	35.29 (95.3%)	36.30 (98.0%)	34.10 (92.1%)	27.31 (73.8%)	34.28 (92.6%)
Average	41.44	40.30 (97.3%)	41.21 (99.4%)	39.65 (95.7%)	36.46 (88.0%)	39.63 (95.6%)

⚠️ Note: average recovery is 95.6% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed English/Korean set (100 prompts, BF16 greedy answers scored under each build — same harness for every row).

Metric	This model	BF16 original
Perplexity (lower = closer)	2.2372	2.0174 (+10.9%)
Top-1 next-token agreement	84.8%	98.7%

These measure fidelity to the base model, not absolute task scores.

⚠️ Note: this small dense model is sensitive to 4-bit quantization (+10.9% perplexity here). For production we recommend the 8-bit builds (FP8 or AWQ INT8) of this model.

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.17.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 1.0 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=5.2.0` + `compressed-tensors>=0.15.0` (checked on CPU with Qwen3.5-0.8B FP8 / AWQ-4bit / NVFP4A16 / NVFP4).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/Qwen3.5-0.8B-AWQ-4bit --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/Qwen3.5-0.8B-AWQ-4bit",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/Qwen3.5-0.8B-AWQ-4bit", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

# image input (URL or data: URI)
messages = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}},
    {"type": "text", "text": "Describe this image."},
]}]
out = llm.chat(messages, params)
print(out[0].outputs[0].text)

Thinking mode is off by default in this model's chat template; to turn it on pass chat_template_kwargs={"enable_thinking": True} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": true} in the request body). Text-only serving: add --limit-mm-per-prompt '{"image": 0, "video": 0}' to skip loading the vision encoder's memory budget.

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above. This is a hybrid model: only its 6 full-attention layers keep a KV cache, while its 18 linear-attention (Gated DeltaNet) layers keep a fixed-size state per request, which --kv-cache-dtype does not change. FP8 halves the full-attention KV cache, so the saving grows with context length and is small for many short requests:

vllm serve axteam/Qwen3.5-0.8B-AWQ-4bit --kv-cache-dtype fp8

License

Quantized derivative of Qwen/Qwen3.5-0.8B (Apache-2.0). All original weights, tokenizer and chat template belong to the Qwen team.

한국어 요약

Qwen/Qwen3.5-0.8B의 AWQ INT4 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	`Qwen/Qwen3.5-0.8B`
양자화 방식	AWQ INT4 (W4A16)
저장 형식	compressed-tensors `pack-quantized`
모델 구조	하이브리드 어텐션(선형 어텐션 + 풀 어텐션) · Dense · 멀티모달(비전) · MTP 헤드
용량	약 1.0 GB
보정 데이터	샘플 256개 (한국어 위주 혼합 코퍼스)

양자화 상세

가중치: INT4 비대칭, 그룹 크기 32 (BF16 스케일 + INT4 영점)
활성화: BF16 (양자화하지 않음)
양자화 절차: PTQ(학습 후 양자화, 재학습 없음), AWQ 방식 — 보정 데이터로 활성화가 큰 가중치 채널을 찾아 채널별 스케일로 보호한 뒤 INT4로 반올림
BF16 유지: 비전 타워·투영층(model.visual.*), MTP 헤드(mtp.*), 모든 RMSNorm, 토큰 임베딩, LM 헤드, SSM의 선형층 외 파라미터(linear_attn.conv1d·A_log·dt_bias) — 원본에서 FP32인 linear_attn.A_log·게이트 linear_attn.norm은 BF16으로 저장(A_log는 원본 값이 모두 BF16으로 정확히 표현돼 손실 없음, norm은 vLLM·Transformers(기본 dtype)가 실행 때 쓰는 BF16 값과 같음) — 전체 파라미터의 약 43%
원본 파일: 토크나이저·chat template·전처리 설정은 원본과 동일(원본에 generation_config.json이 없어 이 리포에도 없음), config.json은 원본에 quantization_config 블록만 추가

품질 (실측)

원본 대비 충실도: PPL 2.2372 (원본 2.0174, +10.9%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 84.8% (원본 자체 기준값 98.7%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
벤치마크 평균: 39.63 (원본 41.44, 95.6%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
한국어 벤치마크: KMMLU 35.74 (원본 34.54, 103.5%) · HAE-RAE 34.28 (원본 37.03, 92.6%)
MMLU 방식: lm-eval 생성형 변형(mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가
주의: ⚠️ 이 정밀도에서 품질 저하가 뚜렷함 (PPL +10.9%, 벤치마크 평균 원본 대비 95.6%) — 품질이 중요하면 8비트(FP8·AWQ-8bit) 빌드 권장

요구사항

실행 엔진: vLLM 0.17.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 — 가중치 전용 정수 형식이라 FP8·FP4 전용 텐서코어 불필요 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
메모리: 가중치 1.0 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
Transformers로 직접 사용: transformers 5.2.0 이상 + compressed-tensors 0.15.0 이상 (Qwen3.5-0.8B의 FP8·AWQ-4bit·NVFP4A16·NVFP4 빌드로 CPU에서 확인)
trust_remote_code: 불필요

사용법

서버 실행: vllm serve axteam/Qwen3.5-0.8B-AWQ-4bit --max-model-len 32768 — OpenAI 호환 API
Python: LLM(model="axteam/Qwen3.5-0.8B-AWQ-4bit", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
thinking 모드: 기본 꺼짐 — 켜려면 chat_template_kwargs={"enable_thinking": True} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": true})
이미지 입력: 메시지에 image_url 항목 사용
텍스트만 사용할 때: --limit-mm-per-prompt '{"image": 0, "video": 0}' — 비전 타워용 메모리 예약을 생략해 메모리 절약
KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 하이브리드 구조라 KV 캐시는 풀 어텐션 6개 층에만 있고, 선형 어텐션(Gated DeltaNet) 18개 층의 요청별 고정 크기 상태는 이 옵션으로 바뀌지 않음 → 풀 어텐션 KV 캐시만 절반으로 줄어 긴 컨텍스트에서 효과가 크고 짧은 요청이 많을 때는 작음 (가중치 양자화와 별개인 실행 옵션)

참고

수정 이력 (2026-11-01): AWQ가 스무딩하지 않는 RMSNorm 가중치를 원본과 비트 단위로 같게 복원 — llm-compressor가 남긴 BF16 반올림 오차(실효 가중치 기준 최대 0.4%) 제거, 양자화 가중치와 AWQ로 스무딩된 RMSNorm은 변경 없음, 품질 항목의 측정값은 복원 전 기준