# Decontamination

Contamination suspicion is the most common and most reasonable way to dismiss a
fine-tune's benchmark gains — and the gains in `bfcl-results.md` are large
enough (`simple_java` +42.00, `simple_javascript` +34.00) that the question
deserves a real answer rather than a reassurance. This document gives the
method, the thresholds, the per-source results, and the file hashes needed to
check that the source we screened is the source we trained on.

The numbers here are real and final — they are not placeholders. They describe
the corpus this model was trained from (dataset `l2-1936f4db1167`).

---

## 1. The rule

**The BFCL corpus is never trained on.** It is registered in our source list as
`NEVER TRAIN` and is used only as (a) the evaluation instrument and (b) the
forbidden corpus that everything else is screened against.

Every external source is screened **before** any of its rows enter a dataset,
and the build **fails closed** if a source lacks a current PASS report. A report
is bound to a SHA-256 of the source file; changing the source file invalidates
the report by construction.

---

## 2. Method

**Forbidden corpus.** Built from the pinned BFCL data directory at harness
commit `f7cf7359b7ac615a0b294831c5ba2bc95ee4a000`, plus our own private suite
items:

| Component | Count |
|---|---:|
| BFCL data files | 71 |
| BFCL benchmark items | 10,109 |
| 8-grams extracted | 194,092 |
| Function signatures extracted | 2,467 |
| Private suite items | included (`v2`, 139 items) |

**Screen 1 — task-content 8-gram overlap.** N-grams are taken from **task
content** (the user's ask and the reference behaviour), not from whole rows.
This matters: an earlier whole-row screen flagged 100% of one source on a single
shared prompt idiom. Boilerplate is not contamination. Contamination is task
content matching task content.

**Screen 2 — function-signature matching.** Rows whose tool signatures match a
benchmark signature are dropped regardless of prose overlap.

**Direction of the source-level gate.** The gate is benchmark-side, following
the Tulu-3 orientation: the question is *how much of the benchmark does this
source cover*, not *how many of this source's rows look a bit like the
benchmark*. A large corpus will always contain some incidental overlap; what
disqualifies it is covering a material fraction of the benchmark.

**Thresholds.**

| Rule | Value |
|---|---|
| N-gram size | 8 |
| Row-level coverage drop | a row is dropped if ≥50% of its task-content n-grams appear in the forbidden corpus |
| Source-level fail rate | a source **FAILS** if it covers >2% of benchmark items |

**No threshold re-tuning.** If a source fails, we do not move the threshold. If
the failure is attributable to an identifiable subset, that subset is excluded
at the converter (with provenance recorded) and the source is re-screened
full-corpus; it must then genuinely pass. A gate that gets re-tuned whenever a
source fails is not a gate.

---

## 3. Per-source results

Sources that contributed rows to this model:

| Source | HF repo | Rows scanned | BFCL items covered | Coverage rate | Gate (≤2%) |
|---|---|---:|---:|---:|:--:|
| ToolACE | `Team-ACE/ToolACE` | 13,301 | 137 / 10,109 | **1.36%** | **PASS** |
| hermes-fc | `NousResearch/hermes-function-calling-v1` | 26,274 | 95 / 10,109 | **0.94%** | **PASS** |
| hermes-reasoning | `interstellarninja/hermes_reasoning_tool_use` | 113,259 | 142 / 10,109 | **1.41%** | **PASS** |
| smol-smoltalk | `HuggingFaceTB/smol-smoltalk` | 54,608 | 126 / 10,109 | **1.25%** | **PASS** |

Sources screened that contributed **zero** rows to this model:

| Source | HF repo | Rows scanned | Gate | Why zero rows |
|---|---|---:|:--:|---|
| xlam-irrelevance | `MadeAgents/xlam-irrelevance-7.5k` | 7,500 | PASS | all 7,500 rows routed to a `needsSynthesis` bucket by the converter and none survived selection; now policy-excluded from commercial mixes (CC-BY-4.0) |
| nemotron-chat | `nvidia/Nemotron-Post-Training-Dataset-v1` (chat) | 0 | PASS | source unusable — every user prompt in the v1 chat split is scrubbed empty upstream |

Note on hermes-reasoning: this source is explicitly BFCL-aligned by
construction, which is why its screening was mandatory-strict. It covers only
1.41% of benchmark items after screening.

**Inline re-screen at build time.** The per-source reports are a precondition,
not the removal mechanism. Row exclusion is re-applied inline during dataset
construction against the same forbidden corpus, so the shipped dataset is
screened as-built, not as-reported. For this dataset the inline screen dropped
a further **643 rows** — **642 on signature matching** and **1 on n-gram
coverage**.

---

## 4. Hashes — check that the screened source is the trained source

Each decontamination report is bound to the SHA-256 of the exact source file it
screened. The dataset manifest records the same hash for the file it read.

| Source | Source file SHA-256 | Report SHA-256 |
|---|---|---|
| ToolACE | `ba12c083fca7e8da48c67ad5b895e495447da7c66e39a2e19742c082e6cb537e` | `4989ea87317b467c6ba1f35c47179a63d3360865af39b983eaec4fd62e25537f` |
| hermes-fc | `4e7acc9884fbd1f76f8b38016bd9c6ee1e027631bb26fb087643094cf6790524` | `c78d5c3d0c935bbba10084a951b1736965bb8788a1d63e025acca70955dc6038` |
| hermes-reasoning | `b3730a6ebcd0cf8c9ca679093db2811ce8a66406fa3244d55a7c40fe539915b1` | `893a1145564d2c417ded4257ee61a3e6e9705f582b9b8cadc7522b520f2f210e` |
| smol-smoltalk | `65dabd3d4a300995c12b90af9de0b9de9180416b962cb8e81190e10d75262493` | `2eb2e9cb64c8ce91f59e463209d00b1f2eeb6295e20ed40cb4c103afd30b39f9` |
| xlam-irrelevance (0 rows) | `2e6f3d0adbd40248a592ea001e3f3a4f1624a5d50f434fa1e7d019e93e922327` | — |
| nemotron-chat (unused) | `02a1001b50d4461a29f9c2be11c7dbfbed96c9b442995d12b44a962f9787629f` | — |

Training-set artefact hashes (dataset `l2-1936f4db1167`, built 2026-07-30,
build seed 42):

| Artefact | SHA-256 | Rows |
|---|---|---:|
| `train.jsonl` | `d2dbce906ff05ae54a75da2713dcf1d02243c629076d22b745f84050754355b5` | 14,460 |
| `valid.jsonl` | `f98956e1057c2222779a93598737de36f6bad04b4c7fa118c7eb978897b081a9` | 5 |

The dataset itself is not published (it contains 53 OculusMind internal capture
rows). The hashes are published so that if we ever release it, or disclose it
under NDA to a partner's engineer, it can be shown to be the same bytes. Note
that the Apache-2.0 licence on the weights does not extend to this dataset.

---

## 5. Private-suite leakage control

Separately from BFCL, training rows derived from our own captures may only come
from **train-split** cases. The build asserts this and fails closed. For the
dataset behind this model:

| Check | Result |
|---|---|
| Holdout cases in the suite | 25 |
| Holdout violations in the training set | **0** |
| Item-id scan across 139 suite items | **clean** |

This matters more than usual here, because the private-suite result being
reported is a *regression*. A leak would have inflated the internal score, not
depressed it — so leakage is not an available explanation for the finding in
`internal-results.md`, and the clean scan says the score is not inflated either.

---

## 6. What this screening does and does not prove

**Does:** show that no source used here covers more than 1.5% of the pinned BFCL
benchmark by task content, that BFCL data was never used for training, that
signature-level matches were removed (642 rows at build time alone), that the
screened bytes are the trained bytes, and that our private benchmark's holdout
split was not trained on.

**Does not:** prove the *base model* is uncontaminated. Ministral-3-8B's
pretraining corpus is not published, and any public benchmark may have leaked
into any public base model. This is a limitation shared by every fine-tune of a
public base and it is why the claim here is a **delta measured against that same
base on the same harness** — whatever contamination the base carries, both sides
of our comparison carry it equally, so it cancels out of the delta. It does not
cancel out of the absolute numbers, and we do not claim it does.

**Also does not:** rule out semantic paraphrase contamination that survives an
8-gram screen. Signature matching catches the tool-shaped version of this;
prose paraphrase of a benchmark task with entirely different wording would pass.
We have no method that closes that gap completely, and neither does anyone else
publishing at this scale.

**A note on the direction of the risk here.** Undetected contamination would
inflate the §1 gains. It would *not* produce the irrelevance regression, which
is a loss of a behaviour the base already had. So a sceptic who suspects
contamination should discount the gains and keep the regression — which is the
conservative reading, and the one this card is written to survive.