minicx/flint-IQ3_S-mix-GGUF

Flint 40B-A2.5B — custom IQ3_S mix (GGUF)

A single-file GGUF quantization of RecursiveSI/flint (39.8B total / ~2.5B active, 49 layers, 256 routed experts + 1 shared, vocab 150144).

3.56 bpw, 16.5 GiB. The point of this file is not a new quantization method — it is a bit allocation. Experts stay at 3 bits; attention, embeddings, the shared expert and the two dense layers get 5–6. That costs almost nothing in bytes and is what keeps the model coherent at this size.

Requirements — read this first

1. Flint needs llama.cpp from master. Support was added in PR #30928, merged 2026-10-21. Release v0.7.0 (2026-10-15) predates it and will not load this file.
2. --jinja will fail with the built-in chat template. The template embedded in the GGUF (same as upstream and as other flint GGUFs) contains replace("\x00", ""), and llama.cpp's minijinja rejects it with lexer: unknown escape character \x. Use the fixed copy shipped here:

llama-server -m flint-40B-A2.5B-IQ3_S-mix.gguf --jinja \
             --chat-template-file flint-fixed.jinja

The only change in flint-fixed.jinja is the removal of that one filter.

Why this file exists

Measured against the other public quants of the same model, on 2×10 GiB (NVIDIA CMP 90HX):

quant	size	15 coding tasks, detailed prompts	same, terse prompts	max context
this file	16.5 GiB	83%	77%	131072
unsloth UD-Q3_K_XL	17.2 GiB	83%	77%	65536
unsloth UD-IQ3_XXS	13.5 GiB	80%	73%	262144
stock IQ3_S ftype	16.1 GiB	33%	3%	—

Two things to take from that table.

Quality is the same as UD-Q3_K_XL, not better. The differences per task are within noise. The gain is that the same quality arrives in a smaller file, and the ~0.7 GiB saved buys KV cache: twice the context on the same two cards.

But see the perplexity section before choosing this file. On wikitext-2 the much smaller UD-IQ3_XXS scores better than this quant, and the two metrics point in opposite directions. This is an allocation with measured trade-offs, not a strict upgrade over everything public.

The last row is the warning. The stock IQ3_S ftype puts 3 bits on everything, attention and the 150k-token embedding table included. That is 3.8% of the file by size, and losing it collapses the model — it degenerates into repeating a phrase mid-sentence and fails tasks every other quant solves. A naive llama-quantize ... IQ3_S of this model is 2.6 GiB larger than UD-IQ3_XXS and far worse. Do not do it.

The recipe

attn_v\.weight=q6_K
attn_k\.weight=q5_K
attn_q\.weight=q5_K
attn_output\.weight=q5_K
ffn_down_shexp\.weight=q6_K
ffn_(gate|up)_shexp\.weight=q5_K
ffn_down\.weight=q6_K
ffn_(gate|up)\.weight=q5_K
token_embd\.weight=q5_K

llama-quantize --tensor-type-file mix.txt --imatrix <imatrix> \
               flint-BF16-00001-of-00003.gguf out.gguf IQ3_S 7

Three things about --tensor-type-file that are easy to get wrong:

• Matching is std::regex_search over the full tensor name, i.e. substring. ffn_up=q5_K also hits ffn_up_exps and ffn_up_shexp — on this model that would raise 141 expert tensors to 5 bits and produce a ~55 GiB file. Escape the dot. The first match wins, so narrow patterns must come first.
• Comments are not supported; a # line aborts with malformed tensor type '#'.
• ffn_gate_inp (the router) is never quantized, so assigning it a type does nothing.

Use --dry-run first. It prints the full per-tensor plan and the final size in under a second, instead of hours.

Measurements and their limits

Perplexity, wikitext-2 test, 300 chunks at -c 512, f16 KV cache (so this isolates weight quantization), identical settings for both files:

quant	size	PPL
this file	16.5 GiB	25.01 ± 0.37
unsloth UD-IQ3_XXS	13.5 GiB	24.36 ± 0.36

The two metrics disagree, and this is the honest result: perplexity favours unsloth's smaller file. The gap is 0.65 at ±0.37 — not noise. On the coding tasks the ordering is the other way round (83/77 vs 80/73).

The reason is visible in the allocations. UD-IQ3_XXS spends q6_K (6.5 bits) on every non-expert tensor and drops experts to iq2_s (2.5 bits); this file spends q5_K (5.5 bits) on attention Q/K/O and the embedding table, and puts the bits into experts at iq3_s (3.44 bits) instead. Perplexity on general prose is very sensitive to the embedding/output path, which unsloth keeps at higher precision; the coding tasks appear to reward expert precision more.

So: pick by what you do. If you want general-purpose next-token quality in the smallest file, UD-IQ3_XXS is the better choice and it is 3 GiB smaller. If you want the coding behaviour and the long context, use this one.

The obvious next step — q6_K on all non-expert tensors, keeping experts at iq3_s, about 180 MiB larger than this file — would likely beat both. It has not been built: the BF16 source was deleted before this perplexity result came in. The recipe for it is mix-a below with every q5_K changed to q6_K, if someone wants to try.

The task numbers above come from 15 Python coding tasks (write-from-spec and fix-the-bug), each run in a detailed and a terse phrasing, 2 repetitions — 60 generations per quant, graded by executing unit tests. That is enough to see a collapse like the IQ3_S row and enough to establish parity with UD-Q3_K_XL; it is not enough to claim a small quality edge in either direction. Single language, single domain.

Context was measured by actually loading the model, not by llama-fit-params, at -ngl 50 -ts 27,23 with -ctk kvarn5 -ctv kvarn4 --kv-tail-tokens 1024 — a KV cache type that exists in the beellama fork and not in upstream llama.cpp. At 131072 the cards hold 9870 / 8826 MiB; 163840 does not fit. With a standard KV type the absolute numbers will be lower for every file in the table, but the relative ordering follows from file size and should hold.

Credits

• Base model: RecursiveSI/flint, Apache-2.0.
• Importance matrix: unsloth (unsloth_calibration_flint-preview.txt, 687 chunks). This quantization reuses it as-is; it would not work without it. Note the calibration set is English-only — behaviour in other languages is not covered by it, and was not benchmarked here beyond informal use.
• The source BF16 GGUF was also taken from the unsloth repository.

License and modifications

Apache-2.0, inherited from the base model; the full text is in LICENSE.

Per Apache-2.0 §4(b): the weights in this repository are modified relative to RecursiveSI/flint. The modification is quantization from BF16 to the mixed GGUF types listed under The recipe, performed with llama-quantize; no training, merging, pruning or fine-tuning was done, and no tensors were added or removed. flint-fixed.jinja is a modified copy of the base model's chat template with one filter removed, as described above.

"Flint" is used here only to identify the model this file is derived from. This repository is not affiliated with or endorsed by Recursive Superintelligence, Inc.