---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- text-generation
base_model:
- Qwen/Qwen3.5-0.8B
base_model_relation: quantized
---
# Qwen3.5-0.8B — ExecuTorch XNNPACK 8da4w + 8-bit embedding

`qwen3_5_0_8b_xnnpack_8da4w_e8.pte` (646.9 MB — down from 1413 MB without embedding quantization; the
248320x1024 fp32 embedding was about a gigabyte of the first export)

- **Source**: Qwen/Qwen3.5-0.8B
- **License**: Apache-2.0
- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`)
- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, XNNPACK extended_ops
- **Config**: `llm_params/qwen3_5_0_8b_xnnpack_8da4w_e8.yaml`

## Verification (Mac arm64, 2026-08-21)

`llm_params/gen_static.py`, token-by-token prefill then greedy decode:

| prompt | output | decode |
|---|---|---|
| capital of France? | "The capital of France is **Paris**." | 38.06 tok/s |
| 日本の首都は? | 「日本の首都は **東京** です。」 | 38.06 tok/s |

Prefill tok/s is a sequential-prefill reference only, this being a static export.
Chat template: ChatML, bos 248045, eos [248046, 248044].

The iPhone number below predates the `use_sdpa_with_kv_cache` change and was measured on
the earlier build, so it is not this file's.

iPhone 17 Pro (ETBench, XNNPACK CPU, default threads), earlier build: decode **10.5 tok/s**,
ttft 0.58 s, load 0.7 s — the same speed as the 1413 MB fp32-embedding export
(9.2-11.1 tok/s) at 46% of its size. A hot device throttles to about 7 tok/s; that was a
cool run.

## The setting that doubles it

`use_sdpa_with_kv_cache: True`. Upstream's own `qwen3_5` config has it off with no reason
given, while the equally hybrid `lfm2` config has it on. Measured on the 2B in one run:
**8.20 tok/s off, 16.64 on**.

That it is only a kernel swap and not a different model was checked three ways: three of
four prompts came out token-identical; the first-step logits agree at **corr 0.995** with
the same top-1; and 338 tokens deep they agree at **corr 0.990**, same top-1 and top-2. The
fourth prompt differs where a thinking model decides how long to think, which is a near-tie
either way.