gemma-4-12B-it-heretic

A decensored variant of google/gemma-4-12B-it, produced with Heretic v1.4.0 (directional ablation / "abliteration"). Gemma 4 12B is the multimodal workhorse of the family — text, image, video, and audio input with text output, 256K context, and configurable thinking modes — and refusal behaviour is suppressed here via targeted weight edits to the attention output and MLP down-projections rather than fine-tuning, so the multimodal perception and reasoning are left largely intact.

Who this is for: developers who want Gemma 4's full multimodal range at a size that still fits a single high-end consumer GPU. This is the tier between E4B (which runs anywhere, on-device) and the 26B/31B MoE models (which need datacenter hardware): 12B is where document, audio, and video understanding gets genuinely good and still quantises to a single card. Refusals drop from 99/100 to 37/100 — the loosest edit in this batch, so expect more drift in tone than the low-KL siblings.

Runs on your gaming PC

Full quantization set (F16 + Q4_K_M, Q5_K_M, Q6_K, Q8_0) produced with llama.cpp.

Your GPU	Recommended quant	Weights
RTX 6000 Ada / A100 (48 GB)	Q8_0	11.80 GB
RTX 4090 / 5090 (24 GB)	Q6_K	9.11 GB
RTX 4080 / 5080 (16 GB)	Q5_K_M	7.96 GB
RTX 3090 (24 GB, dual-GPU offload)	Q5_K_M	7.96 GB
CPU-only (32 GB+ RAM)	Q5_K_M	7.96 GB
CPU-only (16 GB RAM)	Q4_K_M	6.87 GB

Weights only, at this model's native 12B size; add ~2 GB per 32K of context — multimodal context carries image, video, and audio tokens, so KV cache dominates at long context. Q4_K_M is the practical floor for 16 GB systems. OOM? Drop one quant level. Headroom to spare? Go one up.

Abliteration parameters

Trial 100 of a 200-trial Heretic run (seed 1180095890).

Parameter	Value
direction_index	29.72
attn.o_proj.max_weight	1.25
attn.o_proj.max_weight_position	32.29
attn.o_proj.min_weight	0.65
attn.o_proj.min_weight_distance	14.84
mlp.down_proj.max_weight	1.06
mlp.down_proj.max_weight_position	46.29
mlp.down_proj.min_weight	1.04
mlp.down_proj.min_weight_distance	23.15

Performance

Metric	This model	Original model (google/gemma-4-12B-it)
KL divergence	0.0357	0 (by definition)
Refusals	37/100	99/100

Refusals on the harmful evaluation set drop from 99/100 to 37/100. The interesting number here is the KL divergence: at 0.0357 this is the tightest edit in the batch — the lowest collateral damage to everything that isn't refusal — which is why it stopped at 37/100 rather than pushing lower. Gemma 4's safety tuning is broad and consistent, so the remaining refusals are load-bearing for behaviour you'd otherwise destabilise. This is the highest-fidelity variant available here: if you want Gemma 4's answers to stay close to the original and only need the obvious refusals gone, take this one. The E4B sibling trades fidelity (0.1215) for a much lower refusal count (25/100) — pick by which trade you prefer.

direction_index is a single index (29.72) rather than per-layer, which is why the edit stays this tight.

Why abliteration instead of fine-tuning

Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for refusal, leaving the rest of the network (and its capabilities) untouched. See the Heretic repo and the original abliteration writeup for the mechanism.

Made with ❤️ by RACER IS OP — follow for more uncensored models

Files

Safetensors

File	Size
model-00001-of-00005.safetensors	4.62 GB
model-00002-of-00005.safetensors	4.63 GB
model-00003-of-00005.safetensors	4.55 GB
model-00004-of-00005.safetensors	4.64 GB
model-00005-of-00005.safetensors	3.84 GB

BF16, ~12B. The reproduce/ directory carries the full Heretic recipe - config.toml, requirements.txt, the Optuna study journal, and SHA-256 sums - so this exact model can be regenerated bit-for-bit. Reproduce it with heretic --reproduce reproduce/reproduce.json.

GGUF quantizations

Full quantization set (F16 + Q4_K_M, Q5_K_M, Q6_K, Q8_0) produced with llama.cpp.

File	Format	Size
gemma-4-12B-it-heretic-F16.gguf	GGUF F16	22.20 GB
gemma-4-12B-it-heretic-Q4_K_M.gguf	GGUF Q4_K_M	6.87 GB
gemma-4-12B-it-heretic-Q5_K_M.gguf	GGUF Q5_K_M	7.96 GB
gemma-4-12B-it-heretic-Q6_K.gguf	GGUF Q6_K	9.11 GB
gemma-4-12B-it-heretic-Q8_0.gguf	GGUF Q8_0	11.80 GB

Gemma 4 Unified architecture (gemma4) with vision and audio encoders - loads natively in llama.cpp / LM Studio / Jan.

Run llama serve -hf saidutta69/gemma-4-12B-it-heretic to pull the default quant.

Quickstart

# llama.cpp
llama serve -hf saidutta69/gemma-4-12B-it-heretic

# transformers
from transformers import AutoProcessor, AutoModelForImageTextToText

model_name = "saidutta69/gemma-4-12B-it-heretic"
model = AutoModelForImageTextToText.from_pretrained(model_name, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_name)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "https://example.com/page.jpg"},
    {"type": "text", "text": "Describe this document, extract the key figures, and flag anything inconsistent."},
]}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True,
                                        return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Thinking modes

Gemma 4 is built as a reasoner with configurable thinking. Toggle the reasoning block through the chat template — full thinking for hard problems, non-thinking when you want latency:

# full thinking (default)
processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

# skip the reasoning block
processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False,
                              enable_thinking=False)

Strip the reasoning block from the output before displaying it to users if you enabled thinking.

Audio and video

This is the Gemma4UnifiedForConditionalGeneration variant, so audio and video input are native rather than needing a separate audio tower. That makes it useful for meeting-recording analysis, video captioning, and audio-plus-visual QA in one pass.

Model details

Architecture	Gemma4UnifiedForConditionalGeneration (multimodal decoder, unified)
Parameters	~12B
Layers / heads	48 layers, 16 attention heads, 8 KV heads, head dim 256
Hidden / intermediate	3840 / 15360
Sliding window	1024
Position embedding	mixed RoPE — theta 10,000 for sliding layers, proportional theta 1,000,000 with 0.25 partial rotary for full-attention layers
Context length	262,144
Vocab	262,144
Precision	bfloat16
Modalities	Text, image, video, audio in; text out
Languages	140+
Base model	google/gemma-4-12B-it

Where 12B sits in the family

Gemma 4 spans five sizes — E2B, E4B, 12B, 26B A4B, and 31B — mixing dense and mixture-of-experts architectures. The small models (E2B, E4B) have a 128K context; the medium and large ones including 12B reach 256K. 12B is the practical ceiling for single-consumer-GPU deployment: past it you're into MoE territory that wants datacenter hardware. Audio input is native on E2B, E4B, and 12B.

Responsible use

Refusal suppression is deliberate and works as intended: this model will comply with requests the base model would refuse, including some it shouldn't. There is no safety filtering layered on top. You are responsible for how you deploy it — don't put this behind an unmoderated public-facing endpoint serving third parties. It inherits Gemma 4's factual limitations and biases; abliteration removes refusal directions, it doesn't add capability or judgment.

License

Inherits the Gemma 4 license from the base model. Upstream lists it as Apache 2.0 with the Gemma terms of use — read the linked terms before commercial deployment.

Related

• google/gemma-4-12B-it — the base model
• saidutta69/gemma-4-E4B-it-heretic — the 8B on-device sibling, lower KL threshold
• Gemma 4 collection — the rest of the family
• Gemma 4 technical report
• RACER IS OP — Heretic Models — full collection