Instructions to use DevelopingDad/Gemma4-12B-it-Fast with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DevelopingDad/Gemma4-12B-it-Fast with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="DevelopingDad/Gemma4-12B-it-Fast")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast")
model = AutoModelForMultimodalLM.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DevelopingDad/Gemma4-12B-it-Fast with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DevelopingDad/Gemma4-12B-it-Fast"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast

SGLang

How to use DevelopingDad/Gemma4-12B-it-Fast with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use DevelopingDad/Gemma4-12B-it-Fast with Docker Model Runner:
```
docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast
```

Gemma4-12B-it-Fast / README.md

DevelopingDad

Update weight-only checkpoint and generic DGX Spark serving package

231d9a6 verified about 19 hours ago

preview code

Raw History Blame Contribute Delete

11.7 kB

metadata

license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
base_model:
  - DevelopingDad/gemma-4-12b-it-trial62
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - gemma4_unified
  - gemma4
  - nvfp4
  - fp8
  - compressed-tensors
  - vllm
  - speculative-decoding
  - experimental
datasets:
  - allenai/c4

Gemma4-12B-it-Fast

A mixed NVFP4/FP8 weight-only checkpoint derived from DevelopingDad's Gemma 4 12B IT Trial62, with an optional pinned vLLM recipe for one NVIDIA DGX Spark GB10.

Updated October 23, 2026. The checkpoint now selects higher-precision activations (W4A16 MLP / W8A16 attention). All 1,245 retained tensors are byte-identical to the October 8 export; 144 unused MLP activation-scale tensors were removed. No retraining, new alignment, weight merging or new weight quantization occurred. The configuration and checkpoint must be used together. The previous activation-quantized package remains available at revision 64c6387bcb0976f2a4ba4a7adb71fe1e41cf2b9e.

The current optional recipe uses a native-format compatibility template, 65,536 total tokens, 60 GiB FP8 KV cache, eight sequence slots, MTP8, and prefix caching. It adds no task-specific instructions or automatic retries. Full quality and production qualification remain incomplete; arithmetic, factual fidelity, instruction following and structured formatting can fail.

Provenance and checkpoint format

Item	Identity
Original model family	Google DeepMind / Gemma Team, `google/gemma-4-12B-it`
Immediate source	`DevelopingDad/gemma-4-12b-it-trial62`
Source revision	`568aa9764b4f695fb18881b97f6d4ee8b725f75c`
Quantized ancestor	`DevelopingDad/Gemma4-12B-it-Fast@64c6387bcb0976f2a4ba4a7adb71fe1e41cf2b9e`
MLP weights	NVFP4, groups of 16, importance-weighted MSE calibration
Attention weights	Per-channel FP8
Activations	Higher precision; activation quantization disabled in both config groups
Embedding / output head	Original tied BF16, retained byte-for-byte
KV cache	Calibrated scalar FP8 in the optional recipe
Tensor inventory	1,245 tensors / 9,304,785,760 tensor-data bytes
Checkpoint file	`model.safetensors`, 9,304,948,400 bytes
Native tokenizer, processor and chat template	Preserved

The architecture contains 11,959,730,224 original stored BF16 parameter elements across 48 language layers. Packed U8 storage can cause the Hub's automatic counter to undercount logical parameters. The model's native position configuration allows 262,144 positions; that is not a tested context claim for this release. The optional recipe sets a smaller 65,536-token prompt-plus-output window. Eight configured slots do not establish eight simultaneous full-window capacity.

The original conversion used 128 English C4 training rows, seed 91027, maximum 1,024 tokens, a 512-row shuffle buffer, and revision 607bd4c8450a42878aa9ddc051a65a055450ef87. Evaluation prompts and answers were excluded from calibration. No new calibration was performed for this update. The source Trial62 modification method is not documented in its repository and is not inferred here.

The model retains Gemma's heterogeneous attention dimensions: 40 sliding layers with eight KV heads and eight full layers with one KV head. Original BF16 media projections are retained. The generic byte-level verification is in tensor-integrity.json; it establishes checkpoint identity, not behavioral equivalence to the BF16 source.

Optional DGX Spark recipe

Tested runtime family: ARM64, NVIDIA GB10, vLLM 0.31.0, Torch 2.13.0+cu130, Transformers 5.17.0, FlashInfer 0.7.0.post1, compressed-tensors 0.17.0. The original conversion used llm-compressor 0.14.0 / compressed-tensors 0.19.0.

Pinned image: vllm/vllm-openai@sha256:3f7dd5b777d34d1724456ce71f87385dca288c3bb23029ab27dee358f5d2b971.
Separately downloaded Google assistant, pinned to 46d4c6f13f0ac0ad827b915669b8df9b81c64c51; MTP8, FP8-per-tensor draft body and optional isolated FP8 draft head.
65,536 total tokens, eight sequence slots, 4,096 batched tokens, chunked prefill, 60 GiB FP8 KV and long-prefill scheduling controls.
Prefix caching enabled. Exact repeated prefixes may be reused; changed inputs require their own prefill. Disable caching for cold-performance comparisons.
Higher-precision target activations and unquantized target components; native reasoning and tool parsers.
Native-format template with system-block whitespace compatibility; thinking off unless explicitly requested. Loopback binding; no automatic container restart.

The isolated draft-head overlay preserves the target's tied BF16 embedding/head. Startup receipts check dtype, finite scales, shape and sharing. Runtime overlays extend one pinned vLLM revision and are experimental. A 60 GiB KV allocation requires substantial unified memory in addition to the weights and runtime. The root native template remains unchanged. The optional template preserves a space or newline in the system block to match the measured rendering; it contains no added instructions. See serving/README.md for downloads, checksum validation and the launcher. The launcher prints its plan unless --run is passed.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="DevelopingDad/Gemma4-12B-it-Fast",
    messages=[{"role": "user", "content": "Explain why rainbows appear."}],
    temperature=0.0,
    max_tokens=512,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

For opt-in reasoning, pass chat_template_kwargs.enable_thinking=true. The pinned vLLM API also accepts a top-level thinking_token_budget; for example:

extra_body={
    "chat_template_kwargs": {"enable_thinking": True},
    "thinking_token_budget": 2048,
}

Allow enough total completion tokens for both reasoning and the final answer. The 2,048-token reasoning budget is a request option, not a baked checkpoint property or a global server default. Truncating reasoning too early can produce incomplete or empty answers or expose unfinished reasoning in visible content.

Performance and validation

A fresh October 23 sequential screen measured approximately 55.0 and 59.7 tok/s on two explanatory prompts and 89.1 tok/s on a code prompt, after first visible content. All three reached a 512-token output cap; these are generation-rate diagnostics, not completed-answer quality scores. First visible content was 0.065–0.123 seconds on the warm server. Four bounded reasoning/JSON/tool protocol checks passed.

See the generic diagnostic and results. The current screen uses public, self-contained prompts and excludes cache reuse for measured requests. It is a bounded API check, not a standardized quality benchmark or a throughput guarantee. Performance depends on output length, reasoning, cache state, concurrent prefills and hardware.

Earlier October 8 rates used activation quantization, a different optional template, 32K context and 8 GiB KV. Those rates do not describe the updated native-template recipe. No claim of a uniform minimum speed, broad model equivalence or fastest-possible configuration is made.

Capabilities and limitations

Native text, streaming, opt-in reasoning, tool-call protocols and JSON-schema output are supported by the pinned runtime. Protocol success does not establish task correctness.
Higher-precision activations avoid additional activation quantization; NVFP4/FP8 weight quantization still introduces loss relative to the original BF16 checkpoint.
Arithmetic, source fidelity, completeness, requested formatting and resistance to untrusted instructions remain imperfect. Independently check consequential outputs.
Plain JSON-only prompting can add Markdown fences; use explicit JSON schemas when strict structure is required.
Native media components are retained. Limited image checks do not establish full image, audio or video quality. No new comprehensive multimodal benchmark is claimed.
Long-context retrieval, sustained load, full-window concurrency and recovery require separate deployment-specific validation. Configured context and cache size are resource settings, not quality guarantees.
No universal refusal-removal or behavior-preservation claim is made. Full quality qualification remains incomplete.

Credits and attribution

Contribution	Credit and source
Original model, architecture, IT training, tokenizer, processor, native template	Google DeepMind / Gemma Team, Gemma 4 12B IT, technical report
Immediate Trial62 checkpoint and this release's conversion, benchmarking and optional runtime packaging	DevelopingDad, Trial62
Native speculative assistant	Google DeepMind / Gemma Team, Gemma 4 12B IT assistant, pinned revision `46d4c6f13f0ac0ad827b915669b8df9b81c64c51`
Serving engine and upstream code extended by the optional overlay	vLLM contributors, vLLM, source `db9527a46873454610df6dbedf79a36d6bf1a7f6`
Quantization and compressed checkpoint tooling	vLLM Project / Red Hat AI and the llm-compressor and compressed-tensors contributors, llm-compressor, compressed-tensors
Calibration corpus	Google/T5 and the Allen Institute for AI, C4, derived from Common Crawl; C4 dataset license is ODC-BY
GPU kernels, framework and hardware	FlashInfer contributors, FlashInfer; NVIDIA, CUTLASS, CUDA and DGX Spark; PyTorch contributors, PyTorch; Hugging Face contributors, Transformers
Preceding reference checkpoints used to compare serving recipes	Unsloth, Gemma 4 12B IT NVFP4, and Red Hat AI, Gemma 4 12B IT NVFP4

DevelopingDad used OpenAI Codex assistance for implementation, audits and documentation. DevelopingDad maintains and publishes this release. Google assistant weights are downloaded separately and are not bundled here. Prior reference publishers' checkpoints were not merged into this model. No upstream organization is represented as endorsing this derivative.

License and citation

The checkpoint derives from Google's Apache-2.0 Gemma 4 release. See LICENSE, NOTICE, MODIFICATIONS.md, CREDITS.md, provenance.json and SHA256SUMS. Upstream copyright/SPDX headers are retained in optional runtime files. C4's separate ODC-BY license and Common Crawl terms apply to the calibration corpus; corpus text is not distributed here.

@misc{gemmateam2026gemma4,
  title={Gemma 4 Technical Report},
  author={Gemma Team},
  year={2026},
  eprint={2607.02770},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2607.02770}
}