A new paper posted on arXiv identifies a methodological gap in how current benchmarks evaluate AI models intended for enterprise deployment. The authors argue that all 18 audited benchmarks score a generic model identifier, even though usable capability actually depends jointly on the model weights, the serving route, numerical precision, the expected output contract, and the execution harness. They frame this conflation as a systematic source of measurement error.
To address it, the team proposes a protocol called IB2, built around three components. First, a gold-blind preflight check verifies that a given serving route can actually execute the evaluation contract before any task is submitted to it. Second, a scoring rule keeps failures reflected in the score while excluding credit for unsupported capabilities. Third, an adjudication mechanism is structurally blind to the scores it is judging. The protocol's reference instantiation includes 128 locked tasks and 987 assertions spanning document, spreadsheet, chart, tool, and database work — though the corpus itself remains sealed, since the authors stress that the procedure, not the dataset, is the contribution.
Applied across eleven systems, the protocol surfaces several findings. Capability availability turns out to be measurable and sometimes unstable: two complete runs on identical weights failed distinct predicates of the final binding gate, while a third passed it before a subsequent run — limitations the advertised model identifier never exposed. The discriminating power of benchmarks also proves uneven: four of seven test suites saturate within a tight band across six systems, with most of the spread traceable to governed database work and multi-tab join tasks.
Perhaps the most notable result concerns how much the choice of serving route alone can shift outcomes: for the same declared model revision and precision, scores moved from 77.38 to 82.54, with a paired interval of [0.11, 10.60]. The authors caveat that the two configurations differed in access mode, harness generation, and the tool-call parser used by the serving stack — with harness generation itself being a property of their own evaluator rather than of any tested endpoint. They also note that excluding failed responses from denominators reorders system rankings, illustrating how accounting choices around reliability can alter a conclusion even when the stated methodology looks unchanged.