Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4, optimized and compiled for SiMa.ai Modalix with MTP speculative decoding. It bundles the target model and its matching MTP assistant.

• Quantization: BF16 activations, INT4 target language weights and INT8 draft/assistant weights. Target weights retain their checkpoint-encoded quantization.
• Maximum context length: 4,096 tokens, including input and generated output.
• Standard decoding model: Gemma-4-E4B-it-TextOnly-GPTQ-a16w4. See its card for the model description and standard-decoding baseline.

Performance

MTP can provide approximately 2–4× higher generation throughput than the standard decoding model, depending on the application, input and draft acceptance. This is an indicative range; actual gains can be lower or higher.

Generation throughput refers to decoding after the first token; the same speedup does not necessarily apply to prefill or total application latency.

Prerequisites

To run this model, you need:

1. A SiMa.ai Modalix device.
2. SiMa.ai Neat Runtime 0.6.0 or later, including LLiMa with Gemma4 MTP support.
3. The Hugging Face CLI for downloading the compiled package.

Installation & Deployment

1. Install or Update Neat Runtime

Follow the SiMa.ai Neat getting started guide to install or update Neat on Modalix. If a compatible runtime is already installed, continue with the model download.

2. Download the Model

Download the complete repository directly on Modalix:

hf download simaai/Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4 \
  --local-dir /media/nvme/llima/models/Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4 --local-dir Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4
scp -r Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Keep both component directories together inside the downloaded parent directory.

Usage

Validate with LLiMa CLI

Run the parent model directory on Modalix:

llima run Gemma-4-E4B-it-TextOnly-GPTQ-MTP-a16w4

For runtime options, use llima run -h.

GenAI Demo Application

Use the GenAI Multimodal Assistant guide to install and run the demo application.

API Usage

For OpenAI- or Ollama-compatible APIs, follow Serve GenAI Models.

Recompilation

Run compile.sh in a compatible SiMa.ai Model Compiler environment with LLiMa Gemma4 MTP compilation support. The script downloads pinned source checkpoints and compiles both components:

bash compile.sh
llima-deploy .recompile/compiled ./deployed-model

To build an 8,192-token context, recompile and deploy both components using:

MAX_NUM_TOKENS=8192 bash compile.sh

The script uses mtp_precision.py to compile the assistant with INT8 weights. See recompile.json for source revisions and settings.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• The downloadable package uses a 4,096-token context; a different context requires recompiling both components.

Troubleshooting

• llima not found: Install or update the Neat runtime.
• Target or draft/assistant not found: Download the complete repository, preserve both component directories, and pass their parent directory to llima run.
• Permission denied: Check read permissions for the model files and access to /media/nvme.

Resources

• Standard decoding model
• GenAI with LLiMa
• Serve GenAI Models
• Target checkpoint
• Assistant checkpoint

See the source repositories for model and license terms.