When selecting AI programming tools, nearly all comparisons today revolve around "benchmark scores"—benchmarks like SWE-Bench Pro, where models solve a batch of standardized coding problems; higher scores appear to indicate stronger capability, and vendors even headline new model releases with these scores. But does a higher-scoring model actually get more work done?
This question has long lacked a rigorous answer: users adopting newer models may already be experts, and the tasks they take on may inherently be harder. That changed this summer, when a government regulation unexpectedly delivered a natural experiment.
## A Natural Experiment Delivered by an Export Control Order
According to a paper posted to arXiv in November: On June 9, Claude Fable 5 was released, achieving a high score of 80.0% on SWE-Bench Pro; just three days later, on the evening of June 12 (Eastern Time, USA), a U.S. export control directive forced Anthropic to suspend provision of this model—for 19 days. A large number of users were thus forced to switch down to the lower-scoring Opus 4.8—whose score dropped by approximately 11 points (69.2% vs. 80.0%, per system-calibrated metrics).
A Harvard researcher seized this opportunity to do something simple yet powerful: **the people remained the same, the projects remained the same—only the model changed**. The researcher tracked GitHub pull requests (code changes submitted for review and eventual "merging" into the codebase) from 297 users forced to downgrade, both before, during, and after the ban. As a control group, they used data from 8,908 developers who consistently used Opus 4.8 throughout. They applied a statistical difference-in-differences design—comparing the change-over-time difference between the two groups while controlling for developer- and date-level factors.
The results poured cold water on benchmark scores: access to the high-scoring Fable 5 raised the 7-day pull request merge rate by only 1.4 percentage points—with a confidence interval spanning zero (−1.4 to +4.3)—statistically indistinguishable from no effect. The researcher repeated the same analysis across five model upgrades in 2026 (1,208 developers, 15,442 pull requests): all five estimates were **negative** (weighted mean: −0.9 percentage points), and weekly merged count showed no significant increase (+2.4%, with confidence interval again including zero).
In other words: in real-world projects, a model scoring 11 points higher on benchmarks did not lead to more code being merged. The authors pinpoint the crux: real-world tasks never spell out requirements clearly; code must conform to project style; and ultimately, someone must willingly press the "merge" button—yet maintainers rarely use benchmark scores as a merge criterion.
## Between Score and Delivery Lies an Entire "Workplace"
Why can’t scores predict real output? Another benchmark, released the same month, provides a detailed answer. SWE-CoWork abandons "standard coding problems" and instead places AI agents inside a simulated software company: 101 development tasks drawn from real repositories spanning 11 programming languages and 12 application domains; 26 simulated colleagues, each with distinct roles and partial information; agents begin with only a brief handover note and must themselves seek out colleagues to clarify requirements, reconcile conflicting statements, and write code that passes private tests.
Across eleven models, a stark gap emerged: **the two strongest models passed ~63% of functional tests (i.e., tests that shift from failure to pass only if code is correctly modified), yet fully delivered fewer than 8% of tasks**. Three details are especially revealing: feeding the full requirement directly to the model boosted test-pass rates for two models by ~16 percentage points—but full delivery rates remained unchanged—indicating the bottleneck lies not in "can it write code?" but in "can it get the job done?"; one of the top two models, Claude-Opus-5.5, directed 93% of its first questions to the correct colleague, yet extracted only 64% of required information—and then abandoned follow-up; and **31% of requirements self-reported by agents as "completed" failed functional tests**.
Between partial implementation and full delivery lies precisely what real workplaces demand: persistent requirement tracking, self-verification, and honest handover. Benchmarks do not test any of these.
## Two Deployment-Validated Paths Forward
Amid the bad news, two November papers offered encouraging results pointing in the same direction: reliability comes less from swapping in stronger models—and more from process assets external to the model.
One paper came from a 14-person team tackling two common failures in long-running business workflows: longer activity chains accumulate errors exponentially; and the knowledge needed for an activity often mismatches the vocabulary users employ when making requests. Their approach was "model-agnostic": externalize workflow definitions, state, and domain knowledge entirely as version-controlled "control assets"; enforce execution via state machines and machine-verifiable acceptance gates; and assign an independent auditor to score the entire execution trace. To update assets, regression checks and human approval are mandatory. Across three deployed business workflows exceeding 200 steps, in one domain’s 120 cases, unattended completion rose from 50% to 100%, retrieval recall from 56% to 91.67%, end-to-end latency dropped from 440 minutes to 159 minutes, and scenario-analysis recall improved from 45% to 93%. More telling: two other domains were taken over by **teams that had never authored these assets**, completing migration in 4 and 7 days respectively—reliability had been codified into assets, not locked inside any single model or person.
Another paper came from an Amazon engineer addressing AI-era hidden debt: code is written faster than documentation keeps pace, and outdated documentation gets consumed as authoritative input by users’ AI agents. His multi-agent system ran for 43 days in "shadow mode" (listening only, not responding directly to users): triaging 499 tickets, identifying 92 documentation gaps; verified by a panel of independent models: the system extracted 1,121 assertions, of which 302 were covered and verifiable in human responses—and 81.1% (245) were correct. In blind evaluation over equivalent time spans, reviewers preferred agent responses 70.5% of the time versus 23.2% for human engineers. Yet the author stresses human–AI complementarity: humans frequently uncover buried or missing insights in documentation; the system converts each insight into an explicit gap, which humans then review and merge into docs. After repair, when the agent re-responded to the same tickets, reviewers preferred the repaired answers in 71.4% of cases (sample: 35 original low-confidence responses—small scale). Humans remain centrally embedded in the closed loop. The author himself acknowledges this is a pilot within a single deployment context: a technical-user queue for one service, with well-structured documentation—results may differ elsewhere.
## My Takeaway: Stop Asking Which Model Scores Higher—Ask Where the Acceptance Gates Are
Putting these four papers together, I draw four conclusions.
First, the gap between benchmark scores and real-world output is not measurement error—it reflects a **difference in task nature**: benchmarks test "Can this model write this code?", while real work tests "Can it clarify requirements, deliver fully, and earn acceptance?" The 63% vs. <8% gap embodies precisely that distance.
Second, the greatest value of the natural experiment lies in its methodology: marketing claims and leaderboards may mislead, but external shocks provide causal evidence under controlled conditions—"same people, same projects, only model changed." Henceforth, when selecting tools, weight "how much higher the score" should decrease sharply, while "how many more merges occurred in my own workflow" becomes the true metric.
Third, both deployment cases point to the same source of reliability: process assets external to the model—acceptance gates, state machines, documentation feedback loops—and deliberate human review slots built into the system. Note: that "100% unattended completion" was achieved precisely *via* acceptance gates and auditors—not by removing humans. Unattended ≠ unaccountable.
Fourth, the 31% of "self-reported completions" that failed testing should be posted on the wall of every team integrating agents into production: agent self-reports must pass independent gates—just as human-written code must pass peer review.
Finally, a question to leave with: When vendors release new models, should they be required to publish "real-world merge rates / delivery rates" alongside benchmark scores? If leaderboards ranked by real-world delivery, how would today’s rankings shift?
Source:
- Higher-Scoring Coding Models Did Not Get More Work Done (arXiv, 2026): https://arxiv.org/abs/2611.23216