Every time we see news headlines like "A Model Tops the Leaderboard," we implicitly accept one premise: the leaderboard is trustworthy. Over the past month, several bricks have been pried loose from this foundation. For ordinary people, this is not merely internal academic housekeeping—your choice of product, hospitals’ and banks’ procurement decisions based on benchmark scores, and policy debates citing specific numbers all rest upon evaluation results. If scores themselves cannot distinguish between "genuine capability" and "having seen the exam questions before," then all these decisions stand on shifting ground.
## A Score Is Never Just the Model’s Score
A classification study posted to arXiv in August 2026 and updated in early November brings the issue into epistemological focus: a benchmark score is actually a joint property of **five factors—the model, the evaluation framework, the prompting budget (i.e., how many attempts the model is allowed), the sampling population, and the contamination status**. Yet leaderboards print only two columns: model and score. As a result, capability and leakage become "observationally equivalent": from the score alone, you cannot tell whether a model truly solves problems or merely encountered the exact test set during training.
This study classifies contamination into five types, categorized by "which defense each type defeats": direct copying of questions, derivative rewriting, temporal leakage, distributional leakage, and—crucially—the fifth type: **contamination acquired live during the evaluation process itself**. The key inference is that private test sets can only block the first type; whereas the fifth type is an attribute of a "single exam," and must be recorded alongside every reported score. Merely declaring a benchmark “clean” at release time is insufficient.
The researchers also designed a four-field disclosure protocol permitting reporters to honestly fill in "unknown," and conducted a preregistered reliability test: two coders outside the design team independently evaluated 41 documents using the same toolset. They themselves published the bad news first—on 29 of the 41 primary-channel documents, inter-coder agreement had a median of only 0.21 (the upper bound for single-coder self-retest was 0.84); only 13% of documents reported prompting budgets; and **not a single document covered all five contamination types**. In other words, relying on industry self-disclosure to rebuild trust failed even its own reliability test.
## Detectives Are Unreliable Too
So can contamination be detected post hoc—i.e., without vendor transparency, using statistical methods to determine whether a model has "memorized the exam"? A paper accepted by ECML PKDD 2026 delivers a cold shower: it systematically tested three major classes of mainstream contamination detection methods across 25 models—including cutting-edge industrial models—and performed 335 audits. Only **201 audits yielded correct conclusions**. Failure modes were concrete: when reference data used for detection mismatched the suspected model’s distribution, methods generated false positives; and because benchmarks are orders of magnitude smaller than pretraining corpora, another class of methods lacked sufficient statistical power at this scale. The authors’ conclusion in one sentence: statistical detection cannot yet replace transparent data provenance.
Detectives are unreliable, disclosures remain incomplete, and high scores look identical to leakage—by autumn 2026, all three traditional paths out of the evaluation trust crisis appear blocked.
## Cheating Live During the Exam
Then comes an even more foundational discovery: contamination need not occur during training at all.
A paper published in EMNLP 2026 Findings (a collaboration among Alibaba Tongyi Lab, Nanyang Technological University, and others) tested "deep research agents"—AIs capable of retrieving information online—on seven public benchmarks. These agents searched the web for answers *during* the exam. The study classified such "retrieval-time contamination" into three tiers: retrieving metadata about the benchmark, retrieving contextual text containing the question stem, and directly retrieving the original question plus its canonical answer.
Real-world results: On MedQA, a medical QA benchmark with 100 sampled questions, **Gemini Deep Research accessed web pages containing both the original question and its answer for 60% of the questions**; another China-based agent, Step Deep Research, achieved 9% (the authors speculate this reflects its Chinese-language–biased retrieval infrastructure, making it harder to find English benchmark derivatives); Valyu scored 0%. Such live cheating can inflate measured performance by up to 4 percentage points—and even reverse model rankings.
This precisely validates the "fifth type" identified in the earlier taxonomy: contamination resides not in training data, but in the examination room. The actual evaluation target is no longer "what the model knows," but rather "what the model *plus* its entire retrieval system can access in an open environment."
## So Everyone Starts "Hiding Cards from Each Other"
With all three old paths blocked, new actions emerging in November point unambiguously in one direction: abandoning reliance on detection and voluntary disclosure, and instead directly redesigning "who can see what."
On November 9, researchers from OpenMined, MLCommons, Google, Singapore’s AI Safety Institute, and AVERI released a paper reporting what they call the world’s first pilot of "double-blind evaluation": Google DeepMind’s proprietary model Gemini 2.5 Flash Lite underwent testing on MLCommons’ private security benchmark AILuminate—**with model weights invisible to the "exam proctors," and test questions invisible to the "examinee side,"** meeting only within a confidential computing environment hosted on an NVIDIA H100 hardware enclave (a chip-level encrypted execution zone). Before the exam, both parties verified a remote attestation signed by the CPU manufacturer’s hardware root key, confirming the environment had not been tampered with; after the exam, raw responses remained inside the enclave, and evaluators received only aggregated metrics—like anonymous peer review, but enforced by silicon rather than humans.
Around the same time, AACL 2026 accepted another approach: simply preventing models from ever accessing private test sets. This method publicly releases only a small "public slice" of the benchmark; third parties train a surrogate model using the model’s responses on this slice, then use the surrogate to predict its performance on the private portion. The amount disclosed is controlled by a tunable "leakage budget." Across six benchmarks, this method achieves judgment consistency rates of 0.85–0.90 for templated tasks—but drops below 0.73 for reasoning tasks, with average estimation error at 5.88 percentage points. Structured exams can be proxy-tested; reasoning exams cannot.
Earlier still, in May, some teams advocated a third path: designing benchmark datasets to be inherently contamination-resistant—since conventional deduplication and cleaning fail against rewritten or translated questions.
## My Assessment: Evaluation Is Shifting from a Statistical Problem to an Access-Control Problem
Laying out this chain of facts reveals a clear pivot. Open science’s old logic was "everything public, everyone can recompute"; yet the evidence accumulated over the past half-year shows: detective-style recomputation is unreliable (201 correct out of 335 audits), public disclosure protocols yield inconsistent reporting (inter-coder agreement median = 0.21), and the remaining viable options treat confidentiality itself as protocol—hardware enclaves, remote attestation, surrogate models, retrieval sandboxes. Evaluation’s value now depends on **some content remaining undisclosed**.
This is a full-blown paradox—but within it lie three inferences worth remembering.
First, trust hasn’t vanished—it has changed form: from "anyone can recompute your score" to "your score is backed by cryptographic proof + hardware signature + institutional process." And this new form carries barriers—running a double-blind evaluation requires an H100, a cloud-based confidential computing environment, specialized orchestration software, and multi-institutional coordination. Trustworthy evaluation is becoming infrastructural—and infrastructure implies: the authority to issue "trustworthy scorecards" will concentrate in large institutions and nonprofit consortia, leaving individual researchers and small organizations as mere spectators.
Second, the syntax of score reporting must evolve. Since "live cheating"-type contamination is an attribute of a single exam, any score reported without exam conditions (e.g., whether the model could access the internet, whether retrieval occurred in a sandbox or openly, how many attempts were permitted) is like quoting speed without units—the number remains, but the information is dead. The era of leaderboards printing only two columns—"model" and "score"—should end.
Third, for every reader of leaderboards, three concrete questions now follow: Is the test set private and unpublished? Can the model freely access the internet during the exam? Does the score include disclosure of contamination status? Any high score lacking answers to all three questions is just a number.
Finally, a question worth debating: If the price of trustworthy evaluation is hiding exam questions, locking them into enclaves, and delegating verification to chipmakers’ signing keys and a handful of institutions—then who, ultimately, issues the "trustworthy scorecard" for AI, a field that styles itself the most open? Nonprofit consortia? Big-tech clubs? Hardware vendors? When confidentiality becomes the prerequisite for trust, has "open science" died—or merely learned a new way to live?
Source: Angulo, Yeste & Espinos-Morato, Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation (arXiv, August 29, 2026; revised November 6, 2026)
https://arxiv.org/abs/2608.29463