license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- text-generation
base_model:
- Qwen/Qwen3.5-0.8B
base_model_relation: quantized
Qwen3.5-0.8B — ExecuTorch XNNPACK 8da4w + 8-bit embedding
qwen3_5_0_8b_xnnpack_8da4w_e8.pte (646.9 MB — down from 1413 MB without embedding quantization; the
248320x1024 fp32 embedding was about a gigabyte of the first export)
- Source: Qwen/Qwen3.5-0.8B
- License: Apache-2.0
- Quantization: 8da4w linear + 8-bit embedding (
embedding_quantize: "8,0") - Export: executorch 1.4.0
export_llm, static shape (seq_len=1), max_seq_length 2048, XNNPACK extended_ops - Config:
llm_params/qwen3_5_0_8b_xnnpack_8da4w_e8.yaml
Verification (Mac arm64, 2026-08-21)
llm_params/gen_static.py, token-by-token prefill then greedy decode:
| prompt | output | decode |
|---|---|---|
| capital of France? | "The capital of France is Paris." | 38.06 tok/s |
| 日本の首都は? | 「日本の首都は 東京 です。」 | 38.06 tok/s |
Prefill tok/s is a sequential-prefill reference only, this being a static export. Chat template: ChatML, bos 248045, eos [248046, 248044].
The iPhone number below predates the use_sdpa_with_kv_cache change and was measured on
the earlier build, so it is not this file's.
iPhone 17 Pro (ETBench, XNNPACK CPU, default threads), earlier build: decode 10.5 tok/s, ttft 0.58 s, load 0.7 s — the same speed as the 1413 MB fp32-embedding export (9.2-11.1 tok/s) at 46% of its size. A hot device throttles to about 7 tok/s; that was a cool run.
The setting that doubles it
use_sdpa_with_kv_cache: True. Upstream's own qwen3_5 config has it off with no reason
given, while the equally hybrid lfm2 config has it on. Measured on the 2B in one run:
8.20 tok/s off, 16.64 on.
That it is only a kernel swap and not a different model was checked three ways: three of four prompts came out token-identical; the first-step logits agree at corr 0.995 with the same top-1; and 338 tokens deep they agree at corr 0.990, same top-1 and top-2. The fourth prompt differs where a thinking model decides how long to think, which is a near-tie either way.