Independent run: Banking77 12-way 0.9062 against your 77-option 0.690
Cross-check on BANKING77 at 77 options - our option-count curve agrees, magnitude does not.
Setup. Revision c4f7fdb3aeab, one sealed manifest, seed 42, 1,240 scored evaluation decisions across 9 typed-decision suites, with vendor calibration applied.
What we measured (overall 0.8508): Banking77 12-way subset 0.9062, emotion 0.8385, MNLI 0.7917, multilingual intent 1.0000.
The observation. Your card reports BANKING77 with 77 options at 0.690 (v0.2) and 0.652 (v0.3). Ours is a 12-way subset of the same dataset at 0.9062.
The direction matches the option-cardinality effect you document - fewer declared options, easier - but we would like to know whether a 0.22 gap between 12-way and 77-way is what you would expect, or whether it suggests our 12-way reduction is not measuring the same thing your 77-option configuration does. Ours is a genuine subset of the intents, not a re-sampling.
What would help. Whether your 77-option figure and a 12-option subset are produced by the same code path in your harness, and if so, whether you would expect our subset result to land where it does. If a 12-option run of your own is easy, that would settle it directly.
Full data and per-row predictions: https://sysone.sdad.pro/
Thanks for the careful cross-check. A gap of this size is what we'd expect, and it doesn't point to a problem with your reduction:
Different code paths. Up to 16 options, JevK5 reads all answer letters in one pass at the model's main temperature. Above 16, it uses the knockout readout: groups of at most 16, then a final round, with a separate knockout_temperature (0.93 for v0.3). Your 12-way run takes the first path, and our 77-option figure takes the second.
Confusable intents. Most of BANKING77's difficulty is in near-duplicate pairs: the top-up variants, card_payment_fee_charged vs extra_charge_on_statement, the pending and declined variants. A 12-intent subset usually keeps few of them, unless they are chosen on purpose.
Chance level. Chance is 1/12 against 1/77, and both of our figures are raw accuracy.
Different items. Our 0.652 and 0.690 are measured on held-out rows from the BANKING77 train split, with all 77 intents offered. They are not test-split numbers.
We're happy to run your exact 12 intents through the same harness if you send the list (or we can pick 12 ourselves). That would settle it directly.
Yes, please. Here is the exact 12-intent list from our sealed manifest.
Intents (12, balanced at 8 evaluation cases each = 96 total):
card_arrival
card_linking
exchange_rate
lost_or_stolen_card
order_physical_card
pin_blocked
refund_not_showing_up
request_refund
terminate_account
top_up_failed
topping_up_by_card
transaction_charged_twice
What we measured: overall 0.8282 as of v2.1.0 (it was 0.8508 under the v2.0.0 gold, where the label noise inflated some cases).
The evaluation items are in datasets/v2/manifest.v2.1.0.jsonl under suite_id: banking77_12, split evaluation. Each case carries the full state text and the 12-way criteria, so you can re-render the exact prompt shape you expect.
Important nuance. The cases whose gold changed between v2.0.0 and v2.1.0 in this suite are mostly top_up_failed vs topping_up_by_card - exactly the confusable pair you flagged. If you run the same items against both versions of gold, it would also confirm whether our correction was right.
Manifest sha256 bf77c5c373ce64218220377456298ad195d7ed01313d3b01e16ee441da8e3027, seed 42, pinned at JevK5 revision c4f7fdb3aeab.
Update: we have corrected the gold labels and re-scored the panel as dataset v2.1.0.
The overall figure quoted above - 0.8508 - was scored against the old gold, which had our reviewer's label drift on about 14% of public-source cases. Under the corrected gold, the same stored answers for JevK5 re-score to 0.8282.
| v2.0.0 gold | v2.1.0 gold | |
|---|---|---|
| JevK5 overall accuracy | 0.8508 | 0.8282 |
No inference was re-run. The correction moves the gold; the answers are unchanged. Full detail: https://sysone.sdad.pro/