What evaluating HUMAIN M3 taught us about Arabic benchmarks—and the systems behind their scores

Omer Nacar
Research notes · Arabic LLMs · Evaluation

---

🟥 A model scores 14% on virology. Is the model really the problem?

A model scores 14% on virology. The obvious conclusion is that it struggles with the subject.

That was the first result we saw while evaluating humain-m3-preview.

But when we inspected the records, 76 of the 100 requests had not produced model answers at all. They had returned generic HTML 403 Forbidden pages, distinct from the structured guardrail_blocked moderation responses discussed later.

The evaluation code had counted each serving failure as a wrong answer.

After retrying those failed requests, virology reached 59.6% accuracy over observable model answers.

The first score looked like a model weakness. Much of it was a measurement failure.

That incident became the starting point for our paper, The Score Is Not the Model: An Error Budget for Evaluating HUMAIN M3 on Arabic Benchmarks. We evaluated 9,497 questions across three Arabic benchmark suites, then investigated how much the surrounding evaluation system could change the number reported as “model accuracy.”

🟦 Core takeaway
A benchmark score measures a model through a system. Prompt construction, serving reliability, moderation behavior, reasoning configuration, output budgets, and scoring rules all shape the number that reaches a leaderboard.

---

1. What we measured

We evaluated the HUMAIN M3 preview endpoint on September 5, 2026.

The primary runs used:

• temperature: 0
• reasoning: disabled for the main benchmark protocol
• initial output budget: 1,024 tokens
• truncation handling: output budget increased when a response exhausted the available generation space before returning a final answer

The study combines Arabic educational and cultural material with translated academic questions. These are the specific subsets evaluated in this study, not complete coverage of every benchmark.

🟨 What does “model-observed” mean?

The denominator matters.

9,497 attempted items
→ 9,428 items with an observable model answer
→ 69 items with no observable answer

Among the 9,428 observable answers, 32 were provider-exposed outputs returned inside structured moderation error payloads. We use those outputs only for offline measurement and report them separately so the scoring convention remains auditable.

One additional response was invalid and is counted as incorrect.

The remaining 69 requests produced no observable model answer and are excluded from the model-observed denominator.

M3 performed strongly on the evaluated Arabic material. But the more useful finding emerged when we examined how the measurement itself was produced.

---

2. Small pipeline choices can produce large score changes

We built an error budget: a record of how reported accuracy changed when we corrected a measurement defect or changed a scoring convention.

🟧 Important: These effects come from different comparisons, benchmarks, or scoring conventions. They must not be added together.

It is also useful to distinguish three different sources of score movement:

That distinction matters. Fixing a broken prompt repairs the measurement. Enabling reasoning changes the evaluation protocol.

---

3. A failed request is not a wrong answer

The first complete Arabic OpenAI MMMLU run scored 78.14%.

Of 314 records labelled invalid, 313 contained an HTML error page instead of a model answer.

These were generic serving failures—not structured moderation responses.

After retrying the failed requests, the benchmark reached 83.14%, a difference of:

+5.00 percentage points

The damage was highly uneven because the infrastructure incident began partway through an ordered benchmark run.

For example:

• Virology: 14.0% → 59.6%
• Sociology: 44.44% → 91.58%
• US Foreign Policy: 45.00% → 90.00%

This matters because aggregate accuracy alone can hide the failure mode.

An infrastructure incident can create what looks like a convincing pattern of subject-specific model weakness.

🟩 Harness change
Infrastructure failures became a separate outcome, eligible for retry, rather than being silently converted into incorrect model answers.

---

4. The prompt must preserve the actual question

The inherited ArabicMMLU template assumed that every item had exactly four choices labelled A–D.

The benchmark did not always satisfy that assumption.

Some questions had only two or three choices. Others had five choices, including questions whose gold answer was E. Hundreds relied on a reading passage stored in a separate Context field.

The old prompt renderer:

• padded missing options with literal None
• could not faithfully represent fifth-choice questions
• omitted the passage for context-dependent questions

Correcting the template moved ArabicMMLU from:

86.05% → 89.81%

A gain of:

+3.76 percentage points

The largest improvements appeared in subsets where the rendering defects mattered most:

• Arabic Language (General): +13.64 points
• Arabic Language (Grammar): +6.58 points

The hosted endpoint introduces some run-to-run variation, so the rerun is not a perfectly deterministic counterfactual. Even so, the concentration of gains in the subsets most exposed to the prompt defects strongly supports rendering quality as the main explanation.

🟩 Harness change
Render the actual number of choices, support the full answer-key range, and include the passage whenever the item depends on one.

---

5. Moderation creates different kinds of missing answers

Not every 403 meant the same thing.

We observed two importantly different cases:

1. generic HTML serving failures, where no usable model answer was returned;
2. structured moderation responses, where the endpoint returned a moderation error and, in some cases, also exposed the model-generated answer inside error.model_content.

That distinction changes what can be measured.

Across ArabicMMLU, 29 of 32 provider-exposed moderation outputs were correct.

If every moderation response were automatically scored as wrong, those known model outputs would disappear from the evaluation.

This produces three legitimate views of the same benchmark:

The 1.64-point gap between the first and last rows reflects different evaluation questions.

For an application developer, the endpoint's ability to deliver an answer matters.

For a model researcher, a generated answer that is visible in the provider's structured error payload is also useful evidence about the underlying model behavior.

Neither perspective reveals what M3 would have answered on the 38 ArabicMMLU requests with no observable output.

Including the provider-exposed outputs, accuracy over all ArabicMMLU source items is bounded between:

88.93% and 89.91%

depending on the unknown correctness of those unobserved items.

🟦 Reporting rule
Always report the score and denominator together.
“Accuracy when an answer is observable” and “accuracy across all attempted requests” are different measurements.

---

6. Reasoning mode changes the question being asked

A malformed prompt is a measurement defect.

Enabling reasoning is different: it is a legitimate change in evaluation protocol.

Our main runs used the non-reasoning configuration for comparability with historical evaluations. We then ran a targeted diagnostic with adaptive reasoning enabled on five preselected weak formal and quantitative subjects:

• formal logic
• abstract algebra
• college mathematics
• econometrics
• college chemistry

🟪 Targeted reasoning diagnostic — not a full-suite comparison

Across these 499 questions, mean subject accuracy increased from:

64.13% → 84.98%

A gain of:

+20.84 percentage points

calculated before rounding.

Every selected subject improved.

However, this experiment retained a fixed 1,024-token output cap. Under adaptive reasoning, 66 of 499 responses (13.2%) exhausted the available output budget before producing a final answer and were counted as unanswered.

This means the measured 84.98% should not be interpreted as the ceiling of reasoning-enabled M3 under a larger generation budget.

The practical lesson is straightforward:

🟨 A score obtained with reasoning disabled does not establish the model's best attainable performance on multi-step tasks.

Reasoning configuration and output budget should therefore be reported alongside benchmark accuracy.

---

7. The serving layer can change during an experiment

Our early moderation probes tested prompt categories sequentially.

That initially seemed reasonable.

Then an interleaved run showed that the aggregate block rate changed substantially over time:

• first quarter: 40%
• last quarter: 16%

Time had become confounded with prompt category.

A category tested early could appear more likely to be blocked simply because of when it was tested.

We therefore randomized and interleaved the experimental conditions.

In the subsequent category probe:

• Normative Islamic questions—for example, questions asking about religious rulings—were blocked in 26.7% of 30 trials.
• Factual Islamic questions, Arabic grammar, Arabic general knowledge, and neutral English controls produced zero blocks across 120 pooled trials.

This is a pattern observed in a small, controlled probe of a preview endpoint.

It does not identify the moderation mechanism.

It also does not imply that Islamic content generally, or Arabic text generally, triggers blocking.

🟦 Evaluation design lesson
When a hosted system can change over time, randomize and interleave the conditions you want to compare.

---

8. Two Arabic-specific questions worth studying next

8.1 What language appears in the exposed reasoning?

In a 53-probe diagnostic, 35 responses emitted an exposed reasoning trace.

None met our threshold for Arabic-dominant text.

For Arabic mathematics and logic questions, the exposed traces averaged only 2.3% Arabic characters, despite the prompts themselves being Arabic.

This describes the text exposed by the API.

It does not reveal the language of the model's internal computation.

That distinction is essential.

---

8.2 How stable are answers under ordinary Arabic text normalization?

We evaluated 310 paired items under orthographic normalization.

Normalization was associated with a:

16.1% paired answer-disagreement rate

The net accuracy difference was:

−4.19 percentage points

but it did not meet the conventional p < 0.05 significance threshold.

We also lacked an unchanged-input replay baseline. Therefore, some of the disagreement may reflect ordinary endpoint variability rather than the normalization intervention itself.

For that reason, the 16.1% figure should be interpreted as answer disagreement under the paired experiment, not as a direct estimate of robustness degradation.

Both observations motivate further experiments.

Neither supports a sweeping conclusion about Arabic reasoning or Arabic robustness.

---

9. What this means for leaderboard comparisons

The historical comparison in the paper contained top-two margins of:

• 0.62 points
• 2.21 points
• 3.14 points

across the three suites.

Each measured pipeline effect exceeded at least one of those margins.

That makes evaluation details consequential.

It does not establish a corrected ranking, because the nine historical comparison models were not rerun with the revised harness.

For ArabicMMLU, we also evaluated M3 with the inherited prompt template to preserve a procedural comparison.

Its:

• 86.05% under the inherited template

should not be confused with:

• 89.81% under the corrected template

and the two should not be compared as though every historical model received the same correction.

🟥 Leaderboard caution
A difference smaller than the uncertainty introduced by the evaluation pipeline should not automatically be interpreted as a meaningful difference in model capability.

---

10. A better benchmark report

The most reusable output of this work is not a single M3 score.

It is a reporting discipline.

1. 🟦 Show the outcome counts

Report:

• valid answers
• invalid outputs
• filters
• provider-exposed moderation outputs
• unresolved requests

2. 🟥 Keep serving failures separate

Retry transient infrastructure failures and preserve their status instead of silently converting them into model errors.

3. 🟩 Validate the rendered questions

Before a large run, inspect:

• passages
• actual option counts
• answer-key range
• missing fields
• prompt formatting

4. 🟪 State the execution settings

Report:

• reasoning mode
• output-token budget
• temperature
• truncation handling
• retry behavior

5. 🟨 Explain exactly what the score measures

Make the denominator explicit.

State how unavailable answers, moderation events, and invalid generations are treated.

6. 🟧 Interleave comparisons and preserve records

Account for time variation and keep enough per-question information to reconstruct the reported result.

---

Conclusion

HUMAIN M3 is the case study.

The broader contribution is a way to ask better questions about any model served through an API.

A leaderboard number is not produced by model weights alone. It is produced by an end-to-end path:

dataset → renderer → prompt → model configuration → serving layer → moderation → response parser → scoring rule

Each stage can influence the final measurement.

The goal is not to make benchmark scores less useful.

It is to make them more interpretable, more reproducible, and harder to misattribute.

🟦 A score becomes more useful when we can explain how it was produced—and how much of it belongs to the model, the protocol, or the path between them.

---

References and materials

Based on “The Score Is Not the Model: An Error Budget for Evaluating HUMAIN M3 on Arabic Benchmarks,” by Omer Nacar.

All study results above describe the September 5, 2026 evaluation window. humain-m3-preview is a preview endpoint and its behavior may change.

The accompanying research materials contain the evaluation harness, per-question records, and diagnostic probes.

Benchmark background

• ArabicMMLU — Arabic educational benchmark.
• Towards Inclusive Arabic LLMs — background on culturally aligned Arabic evaluation.
• Arabic OpenAI MMMLU on Hugging Face — translated benchmark dataset.

Acknowledgment: HUMAIN provided access to the M3 preview endpoint. The analysis and conclusions are the author's own. All results reported here reflect the behavior of the humain-m3-preview endpoint as observed on September 5, 2026; because this is a preview service, its behavior and performance may change over time.