Access requires acknowledging what this model is
This is a refusal-ablated ("abliterated") model. It complies with requests that the base model refuses. Access is granted automatically once you share your contact details and accept the terms below.
Terms of use
- This model has had its learned refusal direction removed. It will not reliably decline harmful requests. Any safety behaviour must be supplied by your own stack.
- You are solely responsible for how you deploy, serve, or redistribute it, and for complying with all laws and regulations that apply to you.
- If you expose it to third parties, you agree to place your own moderation and policy layer in front of it. The model will not do that job for you.
- It is provided as-is, without warranty of any kind. The authors accept no liability for any use or consequence.
- Licensed under the base model's license (see the license field above); attribution to the base model is expected in derivative work.
By requesting access you agree to share your Hugging Face username and email address with the repository owner.
Log in or Sign Up to review the conditions and access this model content.
Qwen3.8-Flash-Next-Abliterated — NVFP4 (W4A16)
v2 (2026-11-03): this checkpoint now loads and serves on vLLM 0.32.0 (v1 does not). The NVFP4 weight shards are byte-for-byte those of v1; what changed is the n-gram table and the config (below). Verified text-only on one H200 with vLLM's Marlin NVFP4-MoE kernel — see Running it. v1 remains available at revision
480e00b.2026-11-06: faster speculative decoding. The best MTP draft length depends on the workload, and a retrained MTP head is now available: see Speculative decoding: which k and Qwen3.8-Flash-Next-Abliterated-NVFP4-TunedMTP (these weights with a faster MTP head).
NVFP4 build of windowsxp811203/Qwen3.8-Flash-Next-Abliterated, a refusal-removed Qwen3.8-Flash-Next. 335.3 GiB → 125.9 GiB.
This is weight-only NVFP4 (quant_algo: W4A16_NVFP4): activations stay in bf16, so there are no activation scales and
no FP4 tensor-core compute is required. On Hopper the NVFP4 experts run as W4A16. Every v2 number on this card was measured
on one H200 with vLLM's Marlin kernel; stock vLLM 0.32.0 prefers its HUMMING kernel on sm_90, which failed to JIT on our
host, so that path has not been tested with this checkpoint.
What changed in v2
| v1 | v2 | |
|---|---|---|
| MoE experts, attention, DeltaNet, hyper-connections, embeddings, vision tower, MTP head | NVFP4 / bf16 | unchanged — all 9 weight shards byte-identical to v1 (LFS sha256 on the Hub) |
| PLE n-gram table (51.2 B params) | one 102 GB bf16 tensor in model-00008-of-00009.safetensors; vLLM's loader cannot bind it |
the FP8 table from Qwen/Qwen3.8-Flash-Next-FP8 (revision 236dfdf2), copied byte-for-byte: 128 shards + scale in model-ple-0000{1-8}-of-00008.safetensors, declared by the new text_config.ple_embedding_dtype: float8_e4m3fn |
text_config.layer_types |
12 entries spelled qwen_sparse_attention (written by the modelopt export) |
full_attention, the base model's spelling. Earlier vLLM versions rejected the old spelling; vLLM 0.32.0 with transformers 5.19 reads both as indexed_attention, so on 0.32 this is consistency, not a load fix |
Files: v1's model-00008-of-00009.safetensors (the bf16 table) and config.json.text-only are removed, so the
-of-00009 shards skip 00008; that is expected — model.safetensors.index.json lists all 17 weight files.
Why the n-gram swap is safe. Abliteration never touched the n-gram table: its 131 tensors (128 table shards plus three
small int64 tensors) are byte-identical between the base model and the abliterated parent (31 of the 33 shard files that
hold them have the same LFS sha256 on the Hub; the 7 n-gram tensors in the other two were compared one by one), so v1's
bf16 table is the base model's table, and
Qwen's own FP8 release of that same table is its official low-precision form. vLLM 0.32 loads it through its dedicated FP8
PLE path (Qwen4ExpPLEFp8EmbeddingMethod, logged at start-up), and every check below ran through that path. v1 kept the
table in bf16 because no runtime then existed to test an FP8 path.
Community variant — credit where due. gorbatjovy got here first:
qwen3.8-flash-next-abliterated-NVFP4-plefp8
found both reasons v1 would not load in vLLM and fixed them the same way — the n-gram table re-laid-out into the sharded
form vLLM's loader expects and cast to FP8 with a per-tensor scale bit-identical to Qwen's own, and layer_types renamed —
and served it on 2× DGX Spark. Against a BF16-table build of v1, on the first-token distribution of 80 harmless prompts:
top-1 agreement 100 %, mean KL 0.0018, median 0.0006, p90 0.0060. v2 copies the FP8 table from Qwen's release instead of
casting it.
Verified here (v2)
Served with vLLM 0.32.0 on one H200 (sm_90; text-only; NVFP4 experts on the Marlin kernel, see Running it); greedy,
non-thinking; the same prompts, chat format and first-400-character keyword rule as the parent card, on the first 100
AdvBench and GSM8K items. GSM8K answers were generated to 1024 tokens as in the parent, refusal answers to 256. Refusal
markers: the in-file list of the parent's eval_full_flashnext.py plus Traditional-Chinese and curly-apostrophe forms,
whereas the parent's own numbers were scored with a different 49-entry list that script imports when available — so the
two refusal columns are close but not strictly comparable:
| check | v2 (this repo, vLLM) | bf16 parent (transformers) |
|---|---|---|
| AdvBench, first 100 prompts — refused | 0/100 | 5/520 (0.96 %) |
| Chinese harmful prompts (30) — refused | 0/30 (checked by hand¹) | 0/30 |
| GSM8K, first 100 problems | 97/100 | 97.0 % (200) |
¹ The keyword heuristic flags 2 of the 30 answers in the scored run because they use 無法 ("cannot") descriptively, e.g. "網頁 JavaScript 無法直接讀取…" inside a full, compliant answer; the flagged answers were read by hand and none refuses.
The MTP head works. On a 155-prompt mix (MT-Bench en/zh, WildChat zh, HumanEval, GSM8K and MATH-500 with and without
thinking, 8K-token agent prompts with tools), num_speculative_tokens=3 reaches a mean acceptance length of 3.01 tokens
per verification step (2.01 of the 3 drafted tokens accepted, plus the bonus token) and cuts median time per output token
from 7.84 ms to 3.35 ms (2.34×) at one request (output throughput 124 → 258 tok/s); 598 → 1051 tok/s at 8 concurrent
requests (greedy, one H200). The MTP weights ship in this checkpoint; enabling it raises the weights on the GPU from
73.6 GiB to 78.7 GiB.
What is quantized
| component | treatment | size |
|---|---|---|
| MoE experts (48 layers × 512) | NVFP4, 4-bit, group 16, float8_e4m3 block scales + fp32 global scales |
63.3 GiB |
| PLE n-gram table (51.2 B params) | FP8 E4M3 (Qwen's release) | 47.7 GiB |
| MTP draft head (its own 512-expert MoE included) | bf16, grafted back after export | 4.9 GiB |
linear_attn (Gated DeltaNet) |
bf16 | 3.9 GiB |
| full attention, shared experts, routers, gates, hyper-connections, norms | bf16 | 2.9 GiB |
embed_tokens, lm_head |
bf16 | 2.4 GiB |
| vision tower (333 tensors) | bf16, grafted back after export | 0.8 GiB |
| PLE projections and norms | bf16 | 0.06 GiB |
| total | 125.9 GiB |
222,747 tensors; nothing is missing relative to the bf16 parent (nvfp4_breakdown.py). The vision tower and the MTP head
are excluded from quantization by the 294-entry ignore list in the shipped config.json; see v1's card (revision above)
for why grafting them back and keeping them out of quantization mattered.
Running it
VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 \
--language-model-only --max-model-len 32768 --reasoning-parser qwen3
Every v2 number on this card was measured with this setup (vLLM 0.32.0, one H200 per server; each run also set its own
--max-model-len and --max-num-seqs and passed --enable-auto-tool-choice --tool-call-parser qwen3_coder) plus one
local edit: on sm_90, vLLM 0.32.0 prefers its HUMMING NVFP4-MoE kernel, whose JIT failed on our host (NVRTC picked up the
system CUDA 12.6), so we emptied _HUMMING_PREFERRED_CAPABILITIES in
vllm/model_executor/layers/quantization/utils/humming/priority.py to use Marlin (--moe-backend marlin is not a
substitute with MTP on: it also applies to the bf16 MTP-head experts, which vLLM rejects for Marlin; without MTP it was
not tried). VLLM_USE_FLASHINFER_SAMPLER=0 avoided a FlashInfer sampler JIT failure on the same host, so that sampler
never ran; the default model runner was not tried. --language-model-only skips the vision tower; image and video input
were not exercised.
- Memory. By default vLLM 0.32 keeps the 47.7 GiB FP8 n-gram table in pinned host RAM (logged as
weight_device=cpu, pinned=True) and loads 73.6 GiB of weights onto the GPU (78.7 GiB with MTP). Plan for about 50 GiB of free host RAM per server. Smaller GPUs and tensor parallelism were not tested. Data parallelism on one host did not start in our two attempts (DP=3 fails vLLM's MoEintermediate_size % tp_sizecheck; at DP=4 two of the four workers died during start-up, cause not logged), so we ran one independent server per GPU. - Long context. Also served at
--max-model-len 262144with agent prompts of 32K–128K tokens (with MTP k=3). - MTP. Add
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'(k=3 is what the numbers above used). Which k is best depends on the workload; k ≥ 5 needs a patch — see the next section. - Thinking is on by default; the quality checks above turned it off per request
(
"chat_template_kwargs": {"enable_thinking": false}). - Only vLLM was tested.
Speculative decoding: which k (updated 2026-11-06)
Re-measured later with this repo's own MTP head at k = 3 … 7: one H200, vLLM 0.32.0 set up as above plus the QSA ring patch below (the server also carried the FP8 draft LM head backport and patches from our DSpark experiments, all inactive for these stock-MTP runs), one benchmark server at a time, greedy, one request at a time. HumanEval, MATH-500 and Chinese thinking suites at medium effort (up to 2048 new tokens), the 8K agent suite at xhigh effort (up to 1024), the long-context cuts at xhigh effort (44 of 45 prompts; one medium; up to 512), non-thinking suites up to 512. A short GPU job of ours on the same host overlapped the first hour of this session. Cells: time per output token relative to k=3 (+ = faster), acceptance length in parentheses; rows with several suites take the geometric mean of the per-suite median TPOTs and of the per-suite acceptance lengths; the long-context row is the median over all 45 requests.
| workload (prompts) | k=3 | k=4 | k=5 | k=6 | k=7 |
|---|---|---|---|---|---|
| Chinese chat: MT-Bench-zh, WildChat zh and zh-Hant, with and without thinking (460) | 4.17 ms (2.63) | 0 % (2.85) | −3 % (3.02) | −7 % (3.12) | −12 % (3.18) |
| English chat: MT-Bench (20) | 3.48 ms (3.03) | +6 % (3.34) | +7 % (3.62) | +8 % (3.82) | +4 % (3.96) |
| code: HumanEval, with and without thinking (25) | 3.00 ms (3.73) | +10 % (4.50) | +17 % (5.15) | +23 % (5.87) | +25 % (6.32) |
| math: GSM8K, MATH-500 (35) | 3.11 ms (3.60) | +9 % (4.26) | +13 % (4.82) | +21 % (5.46) | +19 % (5.84) |
| agent, 8K-token prompts with tools (15)¹ | 3.35 ms (3.21) | +9 % (3.44) | +18 % (3.86) | +2 % (3.92) | +11 % (4.51) |
| agent, 32K–128K-token prompts with tools (45) | 3.65 ms (3.03) | +5 % (3.45) | +8 % (3.76) | −2 % (3.83) | −4 % (3.99) |
¹ Noisy: in later sessions with retrained MTP heads, this 15-request suite's median moved by 14–18 % between identical runs of one configuration.
- Chinese chat: stay at k=3 or 4. Chinese drafts are accepted less often, so each extra draft token costs more than it returns; k=6 is 7 % slower than k=3 here.
- Code, math, English chat: k=6 (+8 … +23 % vs k=3). Agent prompts with tools: k=5 (8K: +18 %, 32K–128K: +8 % vs k=3; k=6 only +2 % / −2 %). Mixed traffic on stock vLLM: k=4 (level with k=3 on Chinese, +5 … +10 % elsewhere).
- k ≥ 5 needs a patch. Stock vLLM 0.32.0 stops at start-up with
QSA ring capacity 12 must divide the attention block size …for 5 to 8 draft tokens on this model (larger k needs a larger ring; we ran up to 7);patches/qsa-ring-widen-PR54912.diffin the TunedMTP repo is a backport of the open vLLM PR #54912 that fixes it (k ≤ 4 needs nothing). - FP8 draft LM head.
"draft_lm_head_quantization": "fp8"in the speculative config gives the drafter an FP8 copy of the LM head (vLLM PR #59973, released in v0.33.0;patches/patch_vllm_draft_lmhead.pyin the TunedMTP repo backports it to 0.32.0, Model Runner V2 only). With this repo's head it made decoding 5.5 % faster at k=6 (geometric mean over 13 suites, +0.6 … +11.4 %; about 5 % less time per token) with about the same acceptance length (geometric mean −0.3 %), measured in an earlier session with the same sequential setup. - Fastest: the tuned head. Qwen3.8-Flash-Next-Abliterated-NVFP4-TunedMTP is this checkpoint with a retrained MTP head (built on doth4580's DotH head). With k=6 and the FP8 draft head it was faster than this repo's head at its best k on all 13 suites measured (+9.8 % geometric mean, worst +5.5 %), and against k=3 +10 % on Chinese chat and +32 % elsewhere; see its card for the method.
How it was built
v1: quantize_nvfp4_flashnext.py → graft_nvfp4.py → fix_nvfp4_config.py → verify_nvfp4.py (all in this repo).
v2: assemble_nvfp4_v2.py OURS_BODY_DIR QWEN_FP8_DIR BASE_BF16_DIR OUT_DIR (in this repo); no weight is re-quantized.
OURS_BODY_DIR is this repo at 480e00b — its weight shards are hard-linked unchanged except model-00008, which is
dropped. QWEN_FP8_DIR holds Qwen/Qwen3.8-Flash-Next-FP8 at 236dfdf2: its model.safetensors.index.json plus the 33
shards that contain n-gram tensors. BASE_BF16_DIR is Qwen/Qwen3.8-Flash-Next; only its config.json is read (for
layer_types). The script
writes the 128 FP8 n-gram shards and the scale into the eight model-ple-* files, restores layer_types, sets
ple_embedding_dtype, and rewrites the index.
Parent model quality (bf16)
| benchmark | base Flash-Next | abliterated parent |
|---|---|---|
| AdvBench (520) | 517/520 · 99.42 % | 5/520 · 0.96 % |
| HarmBench (400) | 386/400 · 96.50 % | 8/400 · 2.00 % |
| MMLU (1000, paired) | 86.00 % | 84.40 % |
| GSM8K (200) | 96.00 % | 97.00 % |
Full method, the λ sweep, the paired McNemar test on MMLU and the byte-level verification are in the parent model card. The architecture write-up is ARCHITECTURE.md.
License
Qwen Community License 1.0 — not Apache-2.0 — inherited from the base model. Publishing and redistributing derivative weights is permitted. Running a Model-as-a-Service or AI Work Assistant business requires a separate license from Qwen before any commercial use; internal use is exempt provided the model, its outputs and its capabilities reach no third party.
Support / 打賞
If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。
USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888
Disclaimer
This model will not refuse. It is published for alignment and safety research. You are responsible for your use of it and for complying with applicable law.
- Downloads last month
- 288
Model tree for windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4
Base model
Qwen/Qwen3.8-Flash-Next