kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-GGUF

Agnes-3.0-Flash Preview — ROCmFP4 for AMD Strix Halo (gfx1151)

ROCmFP4 / ROCmFPX quantizations of Agnes-AI/Agnes-3.0-Flash — the open-weight Preview checkpoint (33B dense, hybrid gated-delta-rule + global attention, 262,144-token context, text + image) — built and measured on an AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151).

• First ROCmFP4 build of this model — a Hub search on 2026-09-16 found no other ROCm or Strix Halo build of Agnes (21 Agnes repos checked).
• MTP head built into every file. One GGUF per tier; --spec-type draft-mtp finds the head inside the model. It adds 258–270 MiB to a 4-bit file — the separate head file it replaces was 1.91 GiB (why).
• Prompt caching works with MTP on, with the small llama-server patch in recipe/patches/ — without it, a build of this ROCmFPX line reuses nothing while the head is loaded (measured).
• Vision projector included.
• No Agnes-specific patch needed to load the weights — the parallel FFN branch is folded exactly into a standard qwen35 MLP (details).
• Importance-matrix builds of the 4-bit tiers: kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-imatrix-GGUF.

⚠️ This is the Preview checkpoint. Agnes AI states that its production/API model (1M context) is a different checkpoint and that published benchmark results for that model do not apply to these weights.

Which file should I use?

Ryzen AI Max+ 395 (MAX-1), ROCm 7.2.4, patched llama-server (see Quick start), -c 65536, batch 1, greedy (temp 0, top_k 1), ignore_eos so every arm generates exactly 256 tokens after a ~7.1K-token prompt, a unique nonce per request and cache_prompt: false (cache_n = 0 asserted on every timed request), 1 warm-up then the median of 3. Decode numbers are the server's own predicted_per_second. Box iced: no other model loaded.

File	ftype	Size	BPW	KLD vs BF16 ↓²	Same top-1 ↑	PPL (× BF16)	TG MTP ROCm0	TG MTP Vulkan0
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	106	16.82 GiB	4.42	0.0438 ± 0.0010	91.29 %	6.4877 ± 0.0792 (×1.0250)	26.64	27.31
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-COHERENT.gguf	102	17.77 GiB	4.67	0.0385 ± 0.0009	91.39 %	6.4290 ± 0.0780 (×1.0157)	24.76	23.51
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX-AGENT.gguf	115	31.91 GiB	8.39	0.0074 ± 0.0007	96.73 %	6.3014 ± 0.0758 (×0.9955)	16.97	14.97
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX.gguf	111	31.46 GiB	8.27	0.0108 ± 0.0008	96.04 %	6.3176 ± 0.0761 (×0.9981)	16.42	16.17
BF16 reference	32	60.85 GiB³	16	0	100 %	6.3296 ± 0.0762¹	—	—

TG = decode tokens/s after a ~7.1K-token prompt, in-file MTP head, --spec-draft-n-max 4, default (non-strict) verification. ¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.3314 ± 0.0763. ² Quality columns: see Where the quality numbers come from below. ³ BF16 conversion including the MTP layer (the graded BF16 trunk without it is 60.00 GiB); not published.

The two 4-bit tiers trade speed for quality: with MTP, STRIX_LEAN decodes within 7.6 % of COHERENT on ROCm0 (26.64 vs 24.76 tok/s; per-run ranges 26.63–26.99 and 23.03–26.12) and 16 % faster than it on Vulkan0 (27.31 vs 23.51), and prefills 13 % faster than it on ROCm0 (261.3 vs 231.8 tok/s). COHERENT is 12 % lower KLD for 967 MiB more. Decode speed differences under 8.3 % are treated as a tie: files that do identical work per token (each imatrix file and its standard twin) measured up to 8.3 % apart, because draft acceptance varies with the prompt. Prefill barely varies (twins within 2.0 %), so prefill gaps above 3.0 % are treated as real.

The imatrix build narrows the choice. Its STRIX_LEAN matches this repo's COHERENT within noise (KLD 0.0387 vs 0.0385, 0.2σ) at STRIX_LEAN's size (26.31 tok/s on ROCm0), and its COHERENT is the lowest-KLD 4-bit file (0.0322, 23.30 tok/s).

The two 8-bit builds read below BF16 on perplexity (×0.9955 / ×0.9981). That is noise at this resolution, not an improvement — their KLD is still non-zero. Between them, AGENT is 31 % lower KLD than plain (3.3σ), for 457 MiB more.

Quality is graded against the BF16 GGUF on a held-out corpus (wikitext-2 test, -c 2048, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. KLD is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity, whose error bar here is wider than most of the differences.

Where the quality numbers come from. Measured directly on the files in this repo, against BF16 logits recomputed in the same session, and the STRIX_LEAN grade was run twice: all 40 per-chunk rows matched (fresh BF16 logits: recipe/logs/R2_*; grades: R3_*; repeat: R4_*). Today's 4-bit KLD values are 7.8–9.4 % higher than an earlier grading of the same weights on the same machine, before a reboot, while the 8-bit files reproduced their earlier output exactly; BF16's own logits also changed across that reboot, which is why every file here is graded against BF16 logits recomputed today. Those earlier numbers are not used. Grading again with a 16-thread CPU job running alongside (9.49 s per chunk, close to the earlier session's 9.66 s; today's quiet run took 7.53 s) gave today's rows exactly, so CPU load does not change the output.

Quick start

1. Download

hf download kingjones777/Agnes-3.0-Flash-Preview-MTP-ROCmFP4-GGUF --local-dir ~/models/agnes

2. Build llama-server with the prompt-cache patch — ROCmFPX at the measured commit (ROCm and Vulkan prerequisites: the project's build guide)

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout d3ca53726109b46fd5cd6fa42f688c707079b44e
git apply ~/models/agnes/recipe/patches/mtp-prompt-cache-fix.patch
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
cmake --build build --target llama-server -j

(The CMake options of the measured build are listed in Reproduction.)

d3ca537 is also in the history of the official ROCmFPX/ROCmFPX repository; the patch was verified against d3ca537 only.

3. Serve — the MTP head is inside the model file, so there is no --model-draft:

Text and tool calls, with MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/agnes/Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

Images (--mmproj), without MTP:

env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
build/bin/llama-server \
  -m ~/models/agnes/Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj ~/models/agnes/mmproj-Agnes-3.0-Flash-Preview-BF16.gguf \
  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
  -c 65536 --host 127.0.0.1 --port 8080

⛔ Images: with the MTP head loaded, image requests fail on this build (measured). For image input, drop the two --spec-* lines.

(LD_LIBRARY_PATH avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The benchmarks used the MTP command without --mmproj; the exact measured argv is in Reproduction.

Flag	Why
--spec-type draft-mtp	Uses the MTP layer inside the model file; no second file to download or place.
--spec-draft-n-max 4	llama.cpp's default is 16. On this model the measured optimum is about 4 — see the table.
--spec-draft-p-min 0.0	The default 0.75 gate throttles drafting.
--jinja	Already on by default in this build; keep it on — the thinking controls (chat_template_kwargs) and tool calling rely on the Jinja chat template.
-fit off	Autofit reads MemAvailable on integrated GPUs and can silently shrink context or push tensors to CPU.
-cram <MiB>	Not set above (default 8 GiB of host RAM for saved prompts). Set it on a shared box — see Known issues.

⚠️ On an unpatched build the in-file head still works, but prompt caching turns off while it is loaded — every turn re-prefills the whole conversation (measured). For multi-turn / agentic use there, drop the two --spec-* lines.

Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.

Reasoning controls

⛔ Do not pass reasoning_effort: "high".

The upstream model card lists high as the default effort. The chat template shipped with the checkpoint does not accept it. It defaults to xhigh and accepts only xhigh, medium, low; anything else raises:

Unexpected reasoning effort high. Supported types are xhigh (default), medium, and low.

Verified by rendering the template with each value (xhigh / medium / low render; high raises; medium injects no instruction at all). Use:

goal	request field
default deep reasoning	nothing, or "chat_template_kwargs": {"reasoning_effort": "xhigh"}
shorter thinking	"chat_template_kwargs": {"reasoning_effort": "low"}
thinking off	"chat_template_kwargs": {"enable_thinking": false}

With thinking on and a small max_tokens, the whole budget can go to reasoning and content comes back empty — raise max_tokens before concluding the model is broken.

⚠️ preserve_thinking defaults to true in this template: earlier turns' reasoning is re-rendered into the prompt (Qwen3.5's template drops it). Multi-turn and agentic contexts grow faster than you may expect; pass "preserve_thinking": false in chat_template_kwargs to get the Qwen3.5 behaviour.

Speculative decoding (MTP)

The MTP layer is stored inside each model file (15 blk.72.* tensors). It uses the trunk's own 248,320-token embedding and output matrices, so it adds only 258 MiB to STRIX_LEAN, 270 MiB to COHERENT and 461 / 449 MiB to the 8-bit tiers. A separate --model-draft head file has to carry its own copies of those two matrices: the Q4_0 head file this layout replaces was 1.91 GiB, 86 % of it vocabulary matrices the model already had. At run time the difference is 1.85 GiB at load (27.32 → 25.47 GiB) and 1.65 GiB after a ~7K-token request (33.57 → 31.92 GiB) — same tier (imatrix COHERENT), same server, 65,536 context, q8_0 KV cache, vision projector loaded, measured as the drop in MemAvailable.

Tier	Backend	separate head file: tok/s (acceptance)	built-in: tok/s (acceptance)	change
STRIX_LEAN	ROCm0	26.25 (0.704)	26.64 (0.724)	+1.5 %
STRIX_LEAN	Vulkan0	28.25 (0.785)	27.31 (0.766)	-3.3 %
COHERENT	ROCm0	24.09 (0.634)	24.76 (0.660)	+2.8 %
COHERENT	Vulkan0	23.73 (0.647)	23.51 (0.653)	-0.9 %
Q8 AGENT	ROCm0	17.48 (0.681)	16.97 (0.673)	-2.9 %
Q8 AGENT	Vulkan0	15.81 (0.622)	14.97 (0.622)	-5.3 %
Q8	ROCm0	16.66 (0.630)	16.42 (0.642)	-1.4 %
Q8	Vulkan0	16.88 (0.614)	16.17 (0.618)	-4.2 %

Against the separate head file (measured earlier with the same protocol on the unpatched server — the patch does not touch token generation, and decode without a draft head read 11.06 vs 11.04 tok/s on the two), the built-in block is within 3.3 % on the 4-bit tiers, where both drafts use the same Q6_K output matrix. On the 8-bit tiers it was slower in all four measurements (-5.3 % to -1.4 %). Each gap is inside the 8.3 % noise floor on its own, but the direction is consistent and the draft step does cost more there: the built-in block is 8-bit and drafts through the tier's Q8_0 output matrix, where the head file was 4-bit with a Q6_K one.

Measured on Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf (patched server; code prompts of 7,096–7,102 tokens — each request carries a unique nonce):

Config	Decode tok/s (min–max)	Draft acceptance	Prefill tok/s
ROCm0 · no draft head	11.04 (11.04–11.05)	—	279
Vulkan0 · no draft head	11.20 (11.15–11.21)	—	200
ROCm0 · in-file MTP, n-max 3	26.32 (26.31–26.63)	0.797	262
ROCm0 · in-file MTP, n-max 4	26.64 (26.63–26.99)	0.724	261
Vulkan0 · in-file MTP, n-max 4	27.31 (27.19–28.18)	0.766	201
ROCm0 · in-file MTP, n-max 5	26.11 (25.62–26.11)	0.638	260
ROCm0 · in-file MTP, n-max 4, --spec-mtp-strict-qwen	27.02 (26.66–27.03)	0.731	261
Vulkan0 · in-file MTP, n-max 4, --spec-mtp-strict-qwen	27.86 (27.44–28.21)	0.780	198

On ROCm0, n-max 4 is the fastest of the three depths checked (n-max 3: 26.32, 4: 26.64, 5: 26.11 tok/s): 26.64 tok/s, 2.41× the 11.04 tok/s no-draft baseline, with 0.724 of drafted tokens accepted. Vulkan0 speculates correctly on this model: 27.31 tok/s at n-max 4 (2.44× its own 11.20 baseline, acceptance 0.766). The two backends decode 2.5 % apart at n-max 4, inside the noise floor.

Prefill at ~7.1K tokens with the head loaded: ROCm0 261 tok/s, Vulkan0 201 — ROCm0 is 1.30× faster at prompt processing, which dominates multi-turn agentic use. Loading the head costs prefill: without it the same prompt runs at 279 (ROCm0) / 200 (Vulkan0) tok/s (-6.3 % / +0.6 % with MTP).

For reference, the separate Q4_0 head file measured 26.25 (ROCm0) / 28.25 (Vulkan0) tok/s at the same depth on the unpatched server (acceptance 0.704 / 0.785); the in-file block is +1.5 % / -3.3 % from that. The block is quantized with its tier's own types rather than Q4_0, so its predictions differ slightly.

Workload range (imatrix STRIX_LEAN, ROCm0, n-max 4): code 26.31 tok/s (acceptance 0.697) · prose 16.57 tok/s (acceptance 0.352). Speculative speed depends on how predictable the text is — quote a range, not a single number.

An earlier, wider sweep with the separate head file (not shipped any more, unpatched server) found the same shape on ROCm0 — n-max 1: 17.90, 2: 22.84, 3: 25.85, 4: 26.25, 5: 25.28, 6: 24.59 tok/s, peak at n-max 4 — so the in-file check above only brackets that peak. Raw rows: recipe/raw/separate_head_phase_b.jsonl.

Is the output the same as without MTP?

Greedy output with MTP matched greedy output without a draft head on 6 of 9 prompts (default mode), and on 5 of 9 with --spec-mtp-strict-qwen — every pair run cold at temperature 0 (without a draft head, the unpatched and patched servers produced identical output). The 9 prompts are the identity gate's fixed prompt (1) plus the 3 and 5 prompts that the two cache gates also ran without a draft head (c2-* / c3-* rows; cold reply compared with cold reply). A mismatch is not a quality regression: with a draft head loaded the server evaluates the same math in different batch shapes (drafts are verified several tokens at a time, and the head needs the hidden state of every prompt position), so float rounding can differ and a near-tie can break the other way; from that token on, the two replies diverge. If you need output that is byte-for-byte reproducible against a no-MTP run, serve without --spec-type.

This ROCmFPX build also has --spec-mtp-strict-qwen, which caps each draft so the whole verification batch stays inside one 256-cell block of the KV cache — the attention width one-token decoding would use. It measured 27.02 vs 26.64 tok/s on ROCm0 (+1.4 %) and 27.86 vs 27.31 on Vulkan0 (+2.0 %). It did not remove the mismatches above (4 with strict, 3 without). In this mode the server also reprocesses the whole prompt when a request repeats the cached prompt exactly (a regenerate; it logs reason=strict-qwen-exact-hit), because the recurrent state it would need is not saved — so this card uses the default mode.

Prompt caching with MTP

recipe/patches/mtp-prompt-cache-fix.patch (from kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF, sha256 412a0173f60faa937018f0153efe9b5d9230c1606735502485e7e520fffb68dd) applies to d3ca537 cleanly (git apply --check):

    common/common.cpp               |  6 ++++
    common/common.h                 |  5 +++
    tools/server/server-context.cpp | 72 +++++++++++++++++++++++++++++++++--------
    tools/server/server-task.cpp    | 33 +++++++++++++++----

The MTP head's state between steps is the target model's hidden row at the cached prompt's exact end position. The unpatched server cannot rebuild that row for a shorter common prefix, so it refuses partial reuse and reprocesses the whole prompt. The patch stores the row inside each context checkpoint (common_prompt_checkpoint::data_spec), captured at the same moment as the checkpoint's KV state, and on a partial-prefix hit restores the newest checkpoint at or before the common prefix — target state, draft state and MTP row together. If no such checkpoint exists it logs reason=spec-checkpoint-missing and reprocesses cold. The token-generation path is not modified.

Measured: a second turn that shares the first turn's ~7K-token prefix and differs only in its closing instruction, each exchange run warm (cache_prompt: true) and then cold (cache_prompt: false) and the two replies compared byte for byte.

server	draft head	second-turn prompt tokens reused	processed	warm reply = cold reply
d3ca537, unpatched	in-file MTP	0 of 7,067	7,067	—
d3ca537 + patch	in-file MTP	6,011 of 7,039 (all 10 exchanges)	1,028	10/10
d3ca537 + patch	in-file MTP, strict	6,011 of 7,039 (all 5 exchanges)	1,028	5/5
d3ca537 + patch	none (control)	6,011 of 7,039 (all 5 exchanges)	1,028	5/5

⛔ Unpatched, loading the draft head turns prompt reuse off. On the unmodified d3ca537 server with the in-file head, a second turn sharing a ~7K-token prefix reused 0 tokens and reprocessed all 7,067.

✅ Patched, reuse works. In 10 two-turn exchanges at a fixed prompt length of 7,039 tokens, every second turn resumed from the checkpoint 1,028 tokens before its end — 6,011 tokens reused (85 %), 1,028 processed — median prefill 4.2 s instead of 26.8 s cold (6.3× faster), with the draft head still loaded.

Resuming does not change the output: each warm reply was byte-identical to a cold run of the same prompt in 10/10 exchanges with default MTP, 5/5 with --spec-mtp-strict-qwen, and 5/5 with no draft head (control).

One behaviour to know about, which belongs to llama-server rather than to this patch: the server processes the last n_ubatch + 4 tokens of every prompt as two separate batches so it can checkpoint there (1,024 + 4 = 1,028 tokens with the -ub 1024 used in these measurements, where the server default is -ub 512; upstream PR #20288). A turn that resumes from a checkpoint left by a prompt of a different length therefore splits its tail differently from a cold run, and float rounding can flip a greedy token. In an earlier gate whose prompts differed in length by one token, warm and cold replies differed in 1 of 3 exchanges with MTP — and in 2 of 3 with no draft head at all. Both replies are ordinary greedy decodes; they are just not bit-identical to each other.

Memory

Measured footprint (drop in MemAvailable) of imatrix COHERENT with the in-file head, patched server, q8_0 KV cache, vision projector and -cram 512, one slot:

context	after load	after one ~7K-token request
65,536	25.47 GiB	31.92 GiB
131,072	28.80 GiB	35.08 GiB
262,144	36.23 GiB	42.12 GiB

imatrix STRIX_LEAN at 65,536: 24.67 GiB after load, 31.05 GiB after one request.

All three context sizes loaded on this box (124 GiB RAM) with no other model running; the long-context rows were not decode-benchmarked.

Tool calling

The template emits the XML-style <tool_call><function=…><parameter=…> format, which llama.cpp parses natively (through the Jinja chat template, on by default). Suite run through llama-server in the deployed configuration (in-file MTP on), at the checkpoint's recommended sampling (temperature 1.0, top-p 0.95, top-k 20; file and sampling are set in recipe/pipeline/vplan_b.py and recipe/pipeline/agnes_harness.py):

14/14 on native tool_calls (raw XML left in content counts as a failure), run on Agnes-3.0-Flash-Preview-MTP-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf with the in-file head on and the patched server.

check	thinking ON (reasoning_effort: low)	thinking OFF (enable_thinking: false)
multi-arg	✅	✅
nested-object	✅	✅
enum	✅	✅
correct-decline	✅	✅
multi-turn	✅	✅
streaming	✅	✅
parallel	✅	✅

Every check passed in both thinking modes.

Vision

mmproj-Agnes-3.0-Flash-Preview-BF16.gguf is the 27-layer vision tower (shape-identical to Qwen3.5-27B's), loaded with --mmproj. Its attention follows the server's -fa setting, so both settings were checked, with and without the MTP head.

Probe: a synthetic 448×448 image with a red circle and a blue square (a model that ignores the image cannot name both), sent to Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf on the patched server with --mmproj, temperature 0. Pass = the reply names all four of red, blue, circle, square.

MTP head	-fa on	-fa off
loaded	❌ server stopped	❌ server stopped
not loaded	✅ 4/4	✅ 4/4
loaded, unpatched server (control)	❌ server stopped	—

⛔ With the MTP head loaded, image requests fail with -fa on and with -fa off — the server aborted (process: missing MTP boundary for seq_id=0 pos=17 (current=3/1 previous=2/1) → update_slots: failed to process speculative batch). Without the head they work. For image input, serve without --spec-type. The unmodified d3ca537 server aborts the same way, so this is not caused by the prompt-cache patch; the server log points to the MTP state not being advanced across an image's positions, which jump (M-RoPE).

Reply (no head, -fa on):

The image displays two distinct geometric shapes arranged side by side on a plain white background. On the left is a solid red circle, and on the right is a solid blue square. Both shapes are filled with their respective colors — vibrant red for the circle and deep blue for the square — and are clea …

How the parallel FFN was handled

Agnes-3.0-Flash Preview is a Qwen3.5-style dense hybrid (72 layers: 54 gated-delta-rule + 18 global attention, interval 4) with one addition: every layer runs a second, narrower SwiGLU (width 2048) in parallel with the main one (width 17408), and the two outputs are summed. The reference implementation is literally

y = down(act(gate(x)) * up(x)) + parallel_ffn(x)

Because it is a plain sum through the same activation, the branch folds exactly into one wider SwiGLU: gate/up concatenated along the output dimension and down along the input dimension, giving a stock qwen35 MLP of width 19456. That is the same transformation Agnes AI's own SGLang patch applies. The result loads on any llama.cpp/ROCmFPX build with qwen35 support — no Agnes-specific patch (some other GGUFs of this model keep the branch as separate ffn_*_par tensors and require a patched fork).

Three details that are easy to get wrong:

1. Rename delta_attn.* → linear_attn.*. The converter reorders the 48 value heads into the tiled order ggml expects only for tensors named linear_attn.*. Skipping the rename produces a model that loads and runs and is wrong.
2. The MTP layer has no parallel branch (17408 wide), so its MLP is zero-padded to 19456. Zero SwiGLU rows contribute exactly zero.
3. Agnes's RMSNorm is one-centred, x / rms(x) · (1 + w), so the converter's +1 weight offset is correct; the gated delta-rule norm is a plain scale and is correctly left alone.

The fold boundary sits at column 17408 = 544 × 32 = 68 × 256, so no quantization block mixes main and parallel weights.

Verification (recipe/verify_fold.py, which re-derives every tensor from the source independently): PASS. 1305 output tensors, every one bit-identical to its source slice (216 folded, 3 MTP-padded, 1086 passed through), and the folded MLP reproduces the reference forward pass to float32 rounding (max relative error: layer 0: 8.5e-10, layer 36: 5.4e-08, layer 71: 2.5e-08, MTP: 3.0e-08). The full log is in recipe/verify_fold.log.

Quantization methodology

# 0. fold the checkpoint (exact) and verify it
python recipe/fold_agnes.py   Agnes-3.0-Flash  agnes-qwen35
python recipe/verify_fold.py  Agnes-3.0-Flash  agnes-qwen35      # must print RESULT: PASS

# 1. convert: model WITH its MTP layer (the converter's default), and the vision projector
python convert_hf_to_gguf.py agnes-qwen35 --outtype bf16 --model-name Agnes-3.0-Flash-Preview --outfile Agnes-3.0-Flash-Preview-MTP-BF16.gguf
python convert_hf_to_gguf.py agnes-qwen35 --outtype bf16 --mmproj --model-name Agnes-3.0-Flash-Preview --outfile mmproj-Agnes-3.0-Flash-Preview-BF16.gguf

# 2. quantize from BF16 only; the LM head is forced up on every tier and read back by exact tensor name
llama-quantize --output-tensor-type q6_K                            Agnes-3.0-Flash-Preview-MTP-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 16
llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K Agnes-3.0-Flash-Preview-MTP-BF16.gguf OUT Q4_0_ROCMFP4_COHERENT 16
llama-quantize --output-tensor-type q8_0 --token-embedding-type q8_0 Agnes-3.0-Flash-Preview-MTP-BF16.gguf OUT Q8_0_ROCMFPX_AGENT 16
llama-quantize --output-tensor-type q8_0 --token-embedding-type q8_0 Agnes-3.0-Flash-Preview-MTP-BF16.gguf OUT Q8_0_ROCMFPX 16

# 3. grade each shipped file against BF16 logits from the same session (the trunk-only BF16: same trunk weights)
llama-perplexity -m Agnes-3.0-Flash-Preview-BF16.gguf -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 40 --kl-divergence-base bf16.kld \
  -ngl 999 -fa on -dio -dev ROCm0
llama-perplexity -m OUT --kl-divergence-base bf16.kld --kl-divergence -c 2048 -b 2048 -ngl 999 -fa on -dio -dev ROCm0

Receipts (the built file is the receipt — exact tensor names, never a substring match; recipe/logs/):

File	output.weight	token_embd.weight	tensors	trunk vs trunk-only build	MTP block
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	Q6_K	Q5_K	972	identical (957 tensors)	258 MiB
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-COHERENT.gguf	Q6_K	Q6_K	972	identical (957 tensors)	270 MiB
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX-AGENT.gguf	Q8_0	Q8_0	972	identical (957 tensors)	461 MiB
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX.gguf	Q8_0	Q8_0	972	identical (957 tensors)	449 MiB

trunk vs trunk-only build: every trunk tensor's type, shape and sha256 compared with the same tier built without the MTP layer (tensor_identity.py).

Reproduction

server   : https://github.com/charlie12345/ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e
           + recipe/patches/mtp-prompt-cache-fix.patch (sha256 412a0173f60faa937018f0153efe9b5d9230c1606735502485e7e520fffb68dd), applied cleanly (git apply --check)
           built target llama-server with the same CMake options as below; sha256 14e9f1299240c891478c820b8c20811ca5721044d2e63e23799eaa9be26b30bd
tools    : same commit, unpatched (quantize / imatrix / perplexity; the patch touches only common/ and tools/server/)
           build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
           (this CMake forwards AMDGPU_TARGETS to GPU_TARGETS, the spelling used in Quick start)
           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7   (unpatched; used for the unpatched control rows: prompt cache and vision)
source   : Agnes-AI/Agnes-3.0-Flash  revision 891ce4f9ffb89b22888aa7fcc2bb2f3618867684
           folded with recipe/fold_agnes.py (sha256 b1e6dbcf16bbac48fbc95b763efeb85efb21ce1d1fdfa825a6ae01187f044486)
model    : Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf
argv     : /opt/llama-rocm/rocmfpx-724-mtpcache/build-hipvk/bin/llama-server -m /mnt/models/agnes-3.0-flash/out-mtp/Agnes-3.0-Flash-Preview-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
env      : LD_LIBRARY_PATH=<patched build>/bin:/opt/rocm-7.2.4/lib
           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
box      : aimax — AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 124 GiB, GTT 131072 MiB,
           kernel 6.17.6-061706-generic, ROCm 7.2.4
power    : no platform_profile interface on this box; cpufreq governor=performance, EPP=performance;
           GPU measured 84-85 W at 95-100% busy under load
protocol : ~7.1K-token prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
           1 warm-up + median of 3, no co-resident models (box iced)
measured : 2026-09-16, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)

Files

File	Size	sha256
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-STRIX_LEAN.gguf	16.82 GiB	c30ef2cc5e0ecab44613e8e45cc66e05f3916fe4f7d74d7543ad76080fb20074
Agnes-3.0-Flash-Preview-MTP-Q4_0-ROCmFP4-COHERENT.gguf	17.77 GiB	fe4c2134ad5944ddb8ca3f4f1aea8a3d3a025408ad489c4c34514a4113346438
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX-AGENT.gguf	31.91 GiB	3ee9066e547606b3a708918271fc96711ab760adf2d4aae6f7b4997f4acfd9e9
Agnes-3.0-Flash-Preview-MTP-Q8_0-ROCmFPX.gguf	31.46 GiB	d478e7926cfdc56bb80217751fce5c73a0469832df16d25fc616db87b19ba787
mmproj-Agnes-3.0-Flash-Preview-BF16.gguf	0.87 GiB	d35b469bce95e80dfa5928e7547089ceefe8e9bb77452349a3af02c5679d7026

SHA256SUMS covers every model file. recipe/ holds the fold and verification scripts, the prompt-cache patch (recipe/patches/), the full measurement pipeline (recipe/pipeline/), raw per-run results (recipe/raw/), build and receipt logs (recipe/logs/), and results_summary.json with every measured value on this card plus the binary hashes. Architecture facts (layer counts, vocabulary, vision depth) come from the checkpoint's config.json.

Known issues and limits

• Preview checkpoint. Agnes AI's published benchmark results are for a different (production/API) checkpoint.
• reasoning_effort: "high" raises in the shipped chat template — see Reasoning controls.
• Prompt caching with MTP needs the included server patch. Without it the in-file head still drafts, but every turn re-prefills the whole conversation; for multi-turn use on an unpatched build, drop the --spec-* flags.
• A resumed turn can diverge from a cold run of the same prompt (from one flipped greedy token onward) when it resumes from a checkpoint left by a prompt of a different length — a llama-server property, measured with and without a draft head (see Prompt caching).
• llama-server's host-RAM prompt cache defaults to 8 GiB (-cram 8192); each saved ~7K-token prompt costs ~0.7 GiB plus ~0.2 GiB per recurrent-state checkpoint. On a shared box, set -cram explicitly.
• Measured on Linux. The legacy charlie12345/ROCmFPX line has known separate-model draft-mtp bugs on Windows; the in-file path and the patch were not tested on Windows. The patch was verified against d3ca537; it was not tested against current ROCmFPX/ROCmFPX main.
• Not measured: decode beyond a 7,450-token prompt (the 262,144-token context was loaded in the memory test, not benchmarked), long-context quality, video input, concurrency above 1, and task-level accuracy. Perplexity/KLD measure next-token fidelity to BF16 on prose, not reasoning or code correctness.

License and attribution

Apache-2.0, inherited from the base model. Weights and architecture: Agnes AI (Agnes-AI/Agnes-3.0-Flash). ROCmFP4 / ROCmFPX quantization format and runtime: the ROCmFPX project. MTP prompt-cache patch: kingjones777. Folding, quantization, and measurements: kingjones777.