Decontamination
Contamination suspicion is the most common and most reasonable way to dismiss a
fine-tune's benchmark gains — and the gains in bfcl-results.md are large
enough (simple_java +42.00, simple_javascript +34.00) that the question
deserves a real answer rather than a reassurance. This document gives the
method, the thresholds, the per-source results, and the file hashes needed to
check that the source we screened is the source we trained on.
The numbers here are real and final — they are not placeholders. They describe
the corpus this model was trained from (dataset l2-1936f4db1167).
1. The rule
The BFCL corpus is never trained on. It is registered in our source list as
NEVER TRAIN and is used only as (a) the evaluation instrument and (b) the
forbidden corpus that everything else is screened against.
Every external source is screened before any of its rows enter a dataset, and the build fails closed if a source lacks a current PASS report. A report is bound to a SHA-256 of the source file; changing the source file invalidates the report by construction.
2. Method
Forbidden corpus. Built from the pinned BFCL data directory at harness
commit f7cf7359b7ac615a0b294831c5ba2bc95ee4a000, plus our own private suite
items:
| Component | Count |
|---|---|
| BFCL data files | 71 |
| BFCL benchmark items | 10,109 |
| 8-grams extracted | 194,092 |
| Function signatures extracted | 2,467 |
| Private suite items | included (v2, 139 items) |
Screen 1 — task-content 8-gram overlap. N-grams are taken from task content (the user's ask and the reference behaviour), not from whole rows. This matters: an earlier whole-row screen flagged 100% of one source on a single shared prompt idiom. Boilerplate is not contamination. Contamination is task content matching task content.
Screen 2 — function-signature matching. Rows whose tool signatures match a benchmark signature are dropped regardless of prose overlap.
Direction of the source-level gate. The gate is benchmark-side, following the Tulu-3 orientation: the question is how much of the benchmark does this source cover, not how many of this source's rows look a bit like the benchmark. A large corpus will always contain some incidental overlap; what disqualifies it is covering a material fraction of the benchmark.
Thresholds.
| Rule | Value |
|---|---|
| N-gram size | 8 |
| Row-level coverage drop | a row is dropped if ≥50% of its task-content n-grams appear in the forbidden corpus |
| Source-level fail rate | a source FAILS if it covers >2% of benchmark items |
No threshold re-tuning. If a source fails, we do not move the threshold. If the failure is attributable to an identifiable subset, that subset is excluded at the converter (with provenance recorded) and the source is re-screened full-corpus; it must then genuinely pass. A gate that gets re-tuned whenever a source fails is not a gate.
3. Per-source results
Sources that contributed rows to this model:
| Source | HF repo | Rows scanned | BFCL items covered | Coverage rate | Gate (≤2%) |
|---|---|---|---|---|---|
| ToolACE | Team-ACE/ToolACE |
13,301 | 137 / 10,109 | 1.36% | PASS |
| hermes-fc | NousResearch/hermes-function-calling-v1 |
26,274 | 95 / 10,109 | 0.94% | PASS |
| hermes-reasoning | interstellarninja/hermes_reasoning_tool_use |
113,259 | 142 / 10,109 | 1.41% | PASS |
| smol-smoltalk | HuggingFaceTB/smol-smoltalk |
54,608 | 126 / 10,109 | 1.25% | PASS |
Sources screened that contributed zero rows to this model:
| Source | HF repo | Rows scanned | Gate | Why zero rows |
|---|---|---|---|---|
| xlam-irrelevance | MadeAgents/xlam-irrelevance-7.5k |
7,500 | PASS | all 7,500 rows routed to a needsSynthesis bucket by the converter and none survived selection; now policy-excluded from commercial mixes (CC-BY-4.0) |
| nemotron-chat | nvidia/Nemotron-Post-Training-Dataset-v1 (chat) |
0 | PASS | source unusable — every user prompt in the v1 chat split is scrubbed empty upstream |
Note on hermes-reasoning: this source is explicitly BFCL-aligned by construction, which is why its screening was mandatory-strict. It covers only 1.41% of benchmark items after screening.
Inline re-screen at build time. The per-source reports are a precondition, not the removal mechanism. Row exclusion is re-applied inline during dataset construction against the same forbidden corpus, so the shipped dataset is screened as-built, not as-reported. For this dataset the inline screen dropped a further 643 rows — 642 on signature matching and 1 on n-gram coverage.
4. Hashes — check that the screened source is the trained source
Each decontamination report is bound to the SHA-256 of the exact source file it screened. The dataset manifest records the same hash for the file it read.
| Source | Source file SHA-256 | Report SHA-256 |
|---|---|---|
| ToolACE | ba12c083fca7e8da48c67ad5b895e495447da7c66e39a2e19742c082e6cb537e |
4989ea87317b467c6ba1f35c47179a63d3360865af39b983eaec4fd62e25537f |
| hermes-fc | 4e7acc9884fbd1f76f8b38016bd9c6ee1e027631bb26fb087643094cf6790524 |
c78d5c3d0c935bbba10084a951b1736965bb8788a1d63e025acca70955dc6038 |
| hermes-reasoning | b3730a6ebcd0cf8c9ca679093db2811ce8a66406fa3244d55a7c40fe539915b1 |
893a1145564d2c417ded4257ee61a3e6e9705f582b9b8cadc7522b520f2f210e |
| smol-smoltalk | 65dabd3d4a300995c12b90af9de0b9de9180416b962cb8e81190e10d75262493 |
2eb2e9cb64c8ce91f59e463209d00b1f2eeb6295e20ed40cb4c103afd30b39f9 |
| xlam-irrelevance (0 rows) | 2e6f3d0adbd40248a592ea001e3f3a4f1624a5d50f434fa1e7d019e93e922327 |
— |
| nemotron-chat (unused) | 02a1001b50d4461a29f9c2be11c7dbfbed96c9b442995d12b44a962f9787629f |
— |
Training-set artefact hashes (dataset l2-1936f4db1167, built 2026-07-30,
build seed 42):
| Artefact | SHA-256 | Rows |
|---|---|---|
train.jsonl |
d2dbce906ff05ae54a75da2713dcf1d02243c629076d22b745f84050754355b5 |
14,460 |
valid.jsonl |
f98956e1057c2222779a93598737de36f6bad04b4c7fa118c7eb978897b081a9 |
5 |
The dataset itself is not published (it contains 53 OculusMind internal capture rows). The hashes are published so that if we ever release it, or disclose it under NDA to a partner's engineer, it can be shown to be the same bytes. Note that the Apache-2.0 licence on the weights does not extend to this dataset.
5. Private-suite leakage control
Separately from BFCL, training rows derived from our own captures may only come from train-split cases. The build asserts this and fails closed. For the dataset behind this model:
| Check | Result |
|---|---|
| Holdout cases in the suite | 25 |
| Holdout violations in the training set | 0 |
| Item-id scan across 139 suite items | clean |
This matters more than usual here, because the private-suite result being
reported is a regression. A leak would have inflated the internal score, not
depressed it — so leakage is not an available explanation for the finding in
internal-results.md, and the clean scan says the score is not inflated either.
6. What this screening does and does not prove
Does: show that no source used here covers more than 1.5% of the pinned BFCL benchmark by task content, that BFCL data was never used for training, that signature-level matches were removed (642 rows at build time alone), that the screened bytes are the trained bytes, and that our private benchmark's holdout split was not trained on.
Does not: prove the base model is uncontaminated. Ministral-3-8B's pretraining corpus is not published, and any public benchmark may have leaked into any public base model. This is a limitation shared by every fine-tune of a public base and it is why the claim here is a delta measured against that same base on the same harness — whatever contamination the base carries, both sides of our comparison carry it equally, so it cancels out of the delta. It does not cancel out of the absolute numbers, and we do not claim it does.
Also does not: rule out semantic paraphrase contamination that survives an 8-gram screen. Signature matching catches the tool-shaped version of this; prose paraphrase of a benchmark task with entirely different wording would pass. We have no method that closes that gap completely, and neither does anyone else publishing at this scale.
A note on the direction of the risk here. Undetected contamination would inflate the §1 gains. It would not produce the irrelevance regression, which is a loss of a behaviour the base already had. So a sceptic who suspects contamination should discount the gains and keep the regression — which is the conservative reading, and the one this card is written to survive.