What does it actually cost to have an AI agent perform a task—fixing a code bug, running a survey, or processing a batch of documents? You probably can’t answer that. Vendors quote prices like “a few dollars per million tokens”; leaderboards rank models by accuracy; yet no reliable source publishes the true end-to-end cost of completing a real-world task from start to finish.
From late October to mid-November 2026, four papers posted to arXiv tore open this bill and calculated it line by line. Their conclusions matter to everyone paying for such services: listed prices differ by over 11×; leaderboards cannot distinguish good from bad; the largest cost components are hidden from view; and nearly half the model’s inference output can be deleted without affecting results.
## Prices Differ by 11×, Leaderboards Fail to Discriminate
Start with selection. A paper posted on November 16 extracted 58,197 “run–task” records from the Holistic Agent Leaderboard (HAL), focusing analysis on seven benchmarks including SWE-bench. Its three findings grow progressively more sobering.
First, statistically indistinguishable accuracy differences separate adjacent leaderboard tiers: on the accuracy–cost frontiers across the seven benchmarks, 27 out of 35 tiers cannot be reliably ranked. Second, on SWE-bench, a run costing $34.16 achieves an accuracy 14.6 percentage points lower than the top-performing run costing $170.78—suggesting the pricier option is indeed stronger, yet existing task suites cannot statistically confirm any real difference in accuracy between them. Whether a fivefold price premium delivers value remains unanswerable using current benchmarks. Third, even the stated “cost” itself is unreliable: the paper reconstructed each vendor’s cost line-by-line using standard token pricing and found discrepancies up to 11.7× between published figures and reconstructed values—for one batch of 50 tasks, the published cost was $489.29, while reconstruction yielded only $41.76, attributable to differing accounting methods for prompt caching.
The authors’ proposed solution is refreshingly simple: avoid exhaustive evaluation; instead use “round-by-round elimination”—first allocate small budgets to all candidates for a few tasks, eliminate the weakest performers each round, and concentrate remaining budget on survivors. On SWE-bench, this approach achieved the paper’s selection target using just 6.6% of the exhaustive evaluation cost; it delivered the lowest cost on six of the seven benchmarks, and tied for lowest on the seventh. The authors candidly note limitations: advantages narrow if per-task costs are unknown in advance, or if exact identification of the single best candidate is required.
## Where the Money Goes: Not “Writing”, but “Re-reading”
The bill’s structure is even more counterintuitive. A systems paper posted on November 9 analyzed execution traces from real users and benchmark tests (using tools like Claude Code and Codex) and found that **the largest cost driver is not generating new content—but repeatedly re-reading old context**: at every step, the agent resubmits its entire history so far as input, and pays for it again. As conversations lengthen, this “cache read” share grows—rising from 13% to 53% in one trace set, and from 30% to 73% in another. Even a simple search executed within a ~100K-token context—producing almost no new output—incurs charges equivalent to 10,000 input tokens plus 1,000 output tokens.
Tool calls quietly inflate costs too: executions with identical token counts but higher call frequencies incur up to 26× higher costs—because each call triggers a fresh full-context billing. Worse, only 27% of tool responses are used on average; in retrieval tasks, up to 89% of rounds are spent flipping through files. Long-conversation costs can reach 10–100× those of short ones.
Incidentally, their optimization layer—which modifies neither the model nor agent behavior, but wraps a system layer to compress context and merge tool calls—reduced end-to-end cost by up to 35%, while preserving 97% of original accuracy. How easily money can be saved reveals how wastefully it was spent before.
## Nearly Half the Inference Output Is Disposable Without Impact
Can “waste” be measured precisely? A paper posted on November 4 provided a framework: “wasted tokens” are defined as content whose removal does not impair useful work. They normalized full agent execution traces and used a judge model to label deletable segments—repetitions, fillers, redundant fetches, detours—then actually removed them, re-ran the tasks, and verified result preservation.
They deliberately distinguished two categories: “strict waste”—repetition, filler, redundant fetching—and “exploratory detours”: attempts that appear unnecessary in hindsight but were reasonable at the time. Results: in online pruning experiments, removing 41%–46% of model inference output left task performance largely unchanged across two benchmarks.
More important than the numbers are structural insights: the degree of waste varies more with task type and agent configuration than with model choice; the same model paired with different wrappers introduces distinct inefficiency sources. In other words, waste is primarily an engineering-architecture artifact—not model laziness. Expecting “next-generation models to automatically save money” is unlikely to pay off.
## A Cold Economic Reality Check on Multi-Agent Division of Labor
The fourth paper tested a widely hoped-for solution: does splitting tasks across multiple agents improve both quality and cost efficiency? Two researchers from the Linux Foundation’s AI & Data Working Group ran controlled experiments across 135 coding tasks, 2,430 runs, and six architectures—the entire experiment cost only $67.16, fitting the theme well.
Conclusions unfold in three layers. First, protocols bear no blame: handoff protocols between agents add only 7.2 ms per transfer—0.11% of total pipeline latency—and zero cost. Second, expense lies in “splitting” itself: a three-stage pipeline (planner → coder → reviewer) costs 3.22× more and takes 3.24× longer than a single strong-model invocation, yet solves no additional problems. Offloading planning and review to cheaper models reduces cost to 0.39× the original pipeline—but pass rate drops by 6.7 percentage points, leaving it still more expensive, slower, and worse than single-model invocation—and cheaper models “think” more, raising inference token share from 21.9% to 37.6%, eroding unit-cost savings via volume inflation; the paper’s break-even prediction model thus underestimates actual cost by ~19%. Third, real savings come from task-based routing—not role-based division: a retrospective “oracle” that selects between strong and weak models per task achieves 94.3% pass rate at $0.0098 per solved task—half the cost of single strong-model invocation ($0.0198).
Of course, the “oracle” knows answers in hindsight. The paper’s actually implemented zero-shot router captured cost savings but missed accuracy: for hard tasks, it routed 86.7% to cheap models—and 28.8% of those failed. Task-granularity routing is correct; how to route well remains the next challenge. The paper states bluntly: routing entire tasks requires no inter-agent protocol at all.
## My Takeaway: It’s Time to Change the Unit of Pricing
Together, these four papers yield three key takeaways.
First, buying agent services by token price is like buying houses by brick count. The right question is: “How much does it cost end-to-end to solve one problem—and is the accuracy difference statistically discernible?” Leaderboards reporting neither cost methodology nor statistical power are merely advertisements.
Second, the bill’s essence is “paying repeatedly for memory”: architecture determines the bulk of cost; unit price is just a multiplier. This explains a paradox—why replacing pipeline roles with cheaper models fails to save money (“thinking more, writing more” offsets unit-cost gains), while compressing context and eliminating needless tool calls delivers immediate 35% savings.
Third, waste is structural—and leverage lies in structure: waste varies more with task and wrapper than with model, so cost-saving priority should go to workflow design, context management, and demand-driven routing—not waiting for cheaper models or trusting multi-agent division of labor.
If you’re procuring agent tools for your company, ask these three questions directly: What is the end-to-end cost for a typical task? Is the accuracy difference versus competitors statistically discernible? Is the quoted price based on the vendor’s published accounting—or reconstructible using standard token pricing?
Finally, a question worth debating: You pay for “answers”, yet the bill’s largest component is AI “re-reading its own memory”. Should this cost structure be transparent to every user? And shouldn’t leaderboards long ago have shifted to ranking by “cost per solved problem”?
Source:
- Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints: https://arxiv.org/abs/2611.21515