google

/

gemma-4-26B-A4B-it

Like 1.73k

Follow

Google 72.6k

Hi @Dampfinchen ,

Thank you so much for your detailed feedback! We really appreciate you sharing your experience with Gemma 4, especially your insights on MoE performance, multilingual and creative tasks, and comparisons with Qwen 3.5. We’re glad to hear you’re enjoying the models and look forward to seeing your continued testing and exploration!

Hi @Dampfinchen ,

Thank you so much for your detailed feedback! We really appreciate you sharing your experience with Gemma 4, especially your insights on MoE performance, multilingual and creative tasks, and comparisons with Qwen 3.5. We’re glad to hear you’re enjoying the models and look forward to seeing your continued testing and exploration!

Thank you! It's nice to see you interacting with the community. You knocked it out of the park with this release and the high download counts on both your HF page and others like Bartowski and Unsloth speak volumes.

As promised, I have spend a lot of this time with the model and I'm loving it. Of course nothing is perfect so I have noticed two quirks with the model after long term testing:

1. For creative writing purposes: Creative writing is really really good with this model. However I noticed generations are too similar to each other, at the default sampler settings Google recommends, but also using my own settings which I have to set very high to get only slightly variation between generations. I assume this is a side effect of instruct tuning and probably overfitting. Another problem are repetitive AI slop phrases. For example, it really likes to use the phrase "it is not X but Y, Not just X but Y" and stuff like that, plus generally it is too agreeable with the user. Gemini suffers from these issues as well.

2. For agentic tasks, it seems the model could use more initiative. Qwen 3.5 handles things on its own very nicely, but Gemma 4 you kinda have to babysit. The model has trouble following through (calling tools) sometimes after it has stated it plans to do so to the user.

Tool Call -> Tool call -> tool call -> Response to the user, that works without issues. However...

Tool Call -> Tool Call -> Tool Call -> Tool Call -> Response to the user -> Tool call, this is what the model has trouble with especially at higher context sizes.
It often tells the user it is going to do something now, but then stops the generation instead of calling a tool. I have noticed this using different frontends and backends and of course with the latest chat template.

I hope this information is useful to you.

@Dampfinchen -

Thanks for the feedback. Could you share full repro steps, along with any relevant code, prompts, tool schemas, sampler settings, chat templates, and environment details? That would make it much easier for us to investigate these issues.

@Dampfinchen -

Thanks for the feedback. Could you share full repro steps, along with any relevant code, prompts, tool schemas, sampler settings, chat templates, and environment details? That would make it much easier for us to investigate these issues.

For sure. My environment is Windows 11, 32 GB RAM, RTX 2060.

I have made a reproduceable test environment for you. First, please download https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF/blob/main/google_gemma-4-26B-A4B-it-Q5_K_S.gguf . It has the latest fixes.

I have made a test package for you. It includes the latest llama.cpp pre-built with CUDA support, including a CLI prompt for settings as well as an MCP server for web search and a conversation.json you need to import to llama.cpp's webUI. Please download it here: https://www.file-upload.net/en/download-15593431/Reproduce-Gemma4Issue.zip.html

Step 1. Let's set up the MCP server for the websearch first. In the MCP folder, run this command: "uvx mcp-proxy --named-server-config config.json --allow-origin "*" --port 8001 --stateless"

Step 2: Start llama.cpp using the command in the configuration text file (and give it the path to the Gemma 4 model you have just downloaded!). I have used deterministic settings (temp 0) to make it easier to reproduce, this issue also happens with the recommended settings. Now, go to the webui in your browser on http://localhost:5001

Step 3: Now we will configure the MCP duck duck go websearch server. (The other MCPs included in the config are not relevant.) Click the gear in the top right corner, -> MCP-> Add Server -> Server URL: "http://127.0.0.1:8001/servers/ddg-search/mcp" -> Click Add and then save.

Step 4: Now import the conversation.json. Click on the gear again -> Import/Export-> Import Conversation -> Import Reproduce-Conversation.json -> Import "Hello Gemma", then save. On the left you should see the conversation after you click the little icon in the top left corner. Click it and, this step is important, make sure the DDG search server is enabled by clicking the + left to the input chat box, MCP-> enable it.

As you can see from the provided conversation, the first thinking -> websearch logic trace was successful. The task was to find a list of 5 coasters at a theme park. However, look at the last reasoning process and open it. At the end of the reasoning process, it will tell itself that it will do a search now to verify the results. "Let's do one more search to get a definitive list of 5." Now instead of calling the search tool again, it will just generate the response to the user. In other cases like with my Hermes Agent example earlier, it will just tell the user it has to do X now but ends the generation instead of calling a tool. Both issues are very similar. I was using the recommended sampler settings (temp 1, top_k 64, top_p 0.95, min_p 0, rep pen 1)

Click regenerate and you will hopefully see the model making the same mistake on your end. If not, hit regenerate until it does, as even though I used deterministic settings, the output is not 100% deterministic and sometimes skips the verification step thats needed to trigger the issue. For deeper analysis, you can run llama.cpp with the --verbose command to see the exact token output.

This issue is really strange, because a lot of the times it works as expected, but in other cases it just doesn't. Maybe it has something to do with the complexity of the tasks.

This is also not a lcpp limitation, as it works with other models reliably.

That's for the "wants to do X but doesn't" bug. Unfortunately I cannot reliably reproduce the looping issues, as that only has happened once when it was working on a big coding project with me. Then it was repeating the same lines of code blocks endlessly.

I'd love to try to replicate this and fix whatever is broken
can you share the prompt on a gist please?

I'd love to try to replicate this and fix whatever is broken
can you share the prompt on a gist please?

For sure.

https://gist.github.com/Dampfinchen/4ad3832aa5b85fc883f4f0ebd8b5f96d

You can import this prompt.json in llama.cpp and configure the MCP server (config.json) I have mentioned above. Use deterministic settings for best reproduceablity, perhaps you need a few regenerations but you should eventually see how the model thinks about using the tool call to verify its results ("Let's do one more search to get a definitive list of 5.") but just doesn't and goes straight to the answer instead.

This still happens with the latest chat template updates by Google so I assume it's a model issue.

Sorry to interject, but a specific prompt isn't necessary. Simply give the model a "web search" tool and a "read website" tool and ask it about current events. The model will almost never fetch a website. It will rely solely on web search snippets, often while thinking "actually, I have enough information to form a response". If you include instructions in your tool stating something along the lines of "ensure you use the read website tool as search snippets are not enough information to form a factual response" it will ignore the tool instructions. If you put something similar in your system prompt, it will ignore that too. If you question the model on the tool instructions or system prompt it will apologize and use the read website tool once and then immediately forget the tool instructions, the system prompt, and the direct addressing of its failure and simply not use the read website tool in follow-up prompts. The models have been this way since release. I tested Unsloth's new GGUFs yesterday (12B, 26B, 31B) and the issue remains.

I did some tests and it seems that reasoning injection into the model, using a modified chat template, might help with this.

Setting some variables with the reasoning start:

{% set internal_monologue %}
I must approach every task with absolute maximum reasoning effort, checking every and any part asked. I will not take shortcuts; instead, I will comprehensively decompose every problem and I will document every intermediate step, every considered alternative, and every rejected hypothesis to ensure that no assumption is left unchecked.

My objective is surgical and thorough understanding. I need to deeply research any knowledge base and/or codebase available.

When analyzing codebases, I will audit all related files to maintain total context. I will always verify complex or specific technical requirements against the knowledge base, if available, to ensure absolute accuracy. Finally, I am committed to executing every single step requested from both the user and the system prompt.

I will not skip, summarize, or condense the instructions; I will perform all tasks in their entirety, ensuring complete and exhaustive resolution for both system prompt alignment and user request.

So, for system prompt alignment and the user request, I will {% endset %}
{% set internal_monologue_tools %}
So, for completeness and now with the information from the tool call:

{% endset %}

And putting them at the prompt generation part:

{%- if add_generation_prompt -%}
    {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
        {{- '<|turn>model\n' -}}
        {%- if not enable_thinking -%}
            {{- '<|channel>thought\n<channel|>' -}}
        {%- else -%}
            {{- '<|channel>thought\n' + internal_monologue -}}
        {%- endif -%}
    {%- elif ns.prev_message_type == 'tool_response' and enable_thinking -%}
        {{- '<|channel>thought\n' + internal_monologue_tools -}}
    {%- endif -%}
{%- endif -%}

At least in my tests, it seemed to help. This technique might be useful in other areas as well, for steering the reasoning for the models.

I'd love to try to replicate this and fix whatever is broken
can you share the prompt on a gist please?

Hi! Were you able to reproduce the issue?

Doesn't fare better in Pi Agent, either. @lgusm @sonali-kumari11 . This is the latest Gemma 26b 4b QAT model.

I have a similar issue. Is there any Google development to retrain 4.1 without these issues?

I'd love to try to replicate this and fix whatever is broken
can you share the prompt on a gist please?

@lgusm What were the results of your testing?

Hi everyone, sorry for the delay here but:

after analysing the messages, there are a couple of issue mixed together, let me try to answer all of them (sorry if I miss anything but it is already a long answer):

1. Tool-use laziness and secondary tool calls (re: @Dampfinchen )

We reproduced the roller-coaster scenario (asking for 5 items when the initial search returns only 3 to 4 items) across both local llama.cpp deployments and cloud endpoints. Two distinct mechanisms cause this behavior:

• Token budget truncation (max_tokens):
  Gemma 4 Chain-of-Thought (CoT) reasoning is thorough and typically consumes 300 to 450 tokens before emitting tool tokens. If your local UI (such as LM Studio, Jan, or Open WebUI) defaults max_tokens (or n_predict) to 512, the generation cuts off mid-thought (finish_reason: length). Because structured tool calls are generated at the very end of the sequence, the tool call is lost.
  Fix: Set max_tokens to at least 1024 (or 2048 for longer contexts).

• Agentic grounding vs. parametric memory:
  Like most models with strong pre-trained knowledge, when Gemma receives partial search results for well-known topics, its planner may conclude it already has enough context to answer directly rather than triggering an extra round-trip web query.
  Notably, in our benchmarks with structured tools (such as a database tool running SQL queries), the model never showed this behavior. When a SQL query returned 4 records for a 5-item prompt, Gemma 4 recognized the exact boundary, accurately returned only the 4 retrieved records, and explicitly noted in its thinking channel that no 5th record existed in the database, without making up data or skipping logic. The difference with open-web search is that snippets operate in an open-world space where strong pre-trained parametric knowledge can lead the model to feel confident answering directly.
  Best practice (strict citation constraint): If your agentic workflow requires strictly verified external evidence for every single item, require per-item source URLs in your prompt instruction:

"For each item in your response, provide the exact source URL link retrieved from the search results."
Because the model cannot invent external retrieval URLs, this constraint clearly communicates that internal pre-trained knowledge is insufficient for the prompt's requirements, prompting the agent to issue subsequent search queries until all items have verified sources.

2. Snippet reliance (re: @jesterium )

If the model relies only on snippets rather than calling read_website for full-page retrieval, check that your runtime uses the updated official chat_template.jinja. Earlier community templates stripped thinking tags or improperly flattened tool schema boundaries.

3. Quantization (QAT) and multimodal roadmap
   QAT models: Official Quantization-Aware Trained weights (gemma-4-26B-A4B-it-qat-w4a16-ct and GGUF q4_0 / Unsloth UD-Q4_K_XL variants) are now available on the Hugging Face hub. They preserve attention fidelity under low-bit quantization and quantized KV caches compared to post-training quantization.
   Audio and video: Native audio processing is currently focused on the edge variants (E2B and E4B) to optimize on-device latency. Expanding multimodal I/O across the larger models remains an active area of development for future releases.

I hope this helps!!

How is Unsloth UD-Q4_K_XL QAT ?

nevermind. They uploaded a QAT version as well.

Thanks for investigating this. I've been troubleshooting what appears to be a closely related but slightly different issue with Gemma 4 26B-A4B: intermittent failure to initiate the first tool call, even when the request clearly requires current information and the system prompt explicitly instructs the model to use a tool.

I initially encountered this in OpenWebUI, but have reproduced it directly against llama.cpp's OpenAI-compatible API, without OpenWebUI or an MCP server involved.

Environment

• Model: Gemma 4 26B-A4B-it, Google QAT Q4_0 GGUF distributed by ggml-org (gemma-4-26B-A4B-it-Q4_0.gguf)
• llama.cpp: b11669-620d2225c
• Context: 262144
• KV cache: Q8_0 (K and V)
• Sampling: temperature 1.0, top_p 0.95, top_k 64 (unless overridden per test)
• MTP speculative decoding: enabled, using the corresponding ggml-org Q4_0 draft model
• Generation limits: /props reports n_predict = -1 and max_tokens = -1
• Tool selection: tool_choice: "auto"
• Requests: non-streaming, with chat_template_kwargs: {"enable_thinking": true} in the later tests

The /props endpoint reports supports_tool_calls: true and supports_tools: true.

Reproduction

I used a deliberately simple request that requires knowledge of recently released models:

System message:

Today is October 22, 2026.

User message:

I am looking for a good small to mid-size LLM. I currently run Qwen3.6 35B and Qwen3.8 27B. I want something smaller, with long context, reasonably fast (60 tokens/s), and released within the last 6-18 months. No Gemma models.

Tool definition:

{
  "type": "function",
  "function": {
    "name": "search_web",
    "description": "Search the live web for current information, including newly released AI models, release dates, and technical specifications.",
    "parameters": {
      "type": "object",
      "properties": {
        "query": {
          "type": "string",
          "description": "Search query"
        }
      },
      "required": ["query"]
    }
  }
}

With tool_choice: "auto", Gemma frequently returns an ordinary answer without invoking search_web.

It recommends older models such as Llama 3.1 8B, Mistral NeMo 12B, and Qwen 2.5 14B, despite the explicit recency constraint and current date.

Typical failed response metadata:

{
  "finish_reason": "stop",
  "tool_calls": null,
  "completion_tokens": 3200
}

Other failed responses consumed approximately 1,880–3,891 completion tokens and also ended with finish_reason: "stop".

I don't see evidence that these particular failures are caused by generation limits. The model completes its reasoning and produces a normal answer.

Control test: Changing only tool_choice from "auto" to "required" successfully produces a correctly structured search_web function call with finish_reason: "tool_calls".

Explicit requests from the user to search also work reliably in OpenWebUI. The apparent failure is in autonomous tool selection rather than basic tool-call formatting or execution.

Prompt experiments

I tested several system-prompt approaches while keeping the user message and tool definition unchanged.

The most effective instruction so far has been:

When answering questions requiring current external information, recent releases, version compatibility, or information beyond your knowledge, your FIRST action must be to call the appropriate available tool.

Do not generate recommendations from memory when current verification is needed. Do not merely state that your knowledge may be outdated. Search first.

However, even this does not produce reliable tool initiation.

Temperature experiment

I performed six additional tests using the same strong policy, with three paired seeds (101, 202, 303):

These are small samples, so I would not infer that lower temperature improves tool calling. Both temperatures exhibited failures.

Related observation: temporal grounding

Another recurring behavior may be contributing to the problem.

Despite the current date being supplied in the system message, Gemma frequently treats it as hypothetical and assumes references to newer models must be mistakes.

For example, its reasoning has included:

The user says "Today is October 22, 2026". This is a persona/simulated date.

And:

Qwen 3.6/3.8 are not real model names as of my current training, the user might be hallucinating...

Instead of recognizing a knowledge gap and searching, it often rationalizes away the unfamiliar information and proceeds using older knowledge.

Even when it successfully initiates a search, its generated queries sometimes target 2024–2025 rather than the appropriate 2025–2026 timeframe.

Explicit date-grounding instructions can improve the model's interpretation of the date, but have not reliably corrected tool initiation.

This seems related to the behavior described in Gemma 4 discussion #80.

Current assessment

My observations suggest a distinction between:

1. The model's ability to produce a valid tool call, which appears functional.
2. Its autonomous decision to use an available tool, which is unreliable.
3. Its ability to recognize that unfamiliar contemporary information warrants external verification.

This differs from the secondary-tool-call scenario discussed above because the failures occur before any external information has been retrieved.

I have not yet repeated the tests with MTP speculative decoding disabled, or against another inference backend, so I cannot exclude an implementation-specific contribution.

Update: MTP speculative decoding control
I repeated the six-trial temperature/seed experiment with MTP speculative decoding completely disabled, using the same prompts, tool schema, sampling settings, and seeds.

MTP disabled:
Temperature:
1.0
0/3 successful tool calls
Temperature:
0.3
1/3 successful tool calls

The failure persists without MTP, including normal completions with finish_reason: "stop" and no tool calls. These samples are too small to determine whether MTP meaningfully affects tool-selection reliability, but they demonstrate that speculative decoding is not required to reproduce the problem. Temporal grounding also remains inconsistent. The one successful call without MTP searched for models released in 2024–2025, despite the system date being October 2026.

Is this first-tool-call failure consistent with the agentic grounding behavior your team has identified, or would you consider it a separate issue worth investigating?