There is a class of questions that ordinary people—without any specialized training—can answer correctly 90% of the time: looking up a fact, comparing two images, navigating a few clicks through a webpage, and finally delivering a clear answer. The GAIA benchmark, released in November 2023, was designed precisely along this principle—conceptually simple for humans, yet demanding that a system simultaneously reason, interpret images, browse the web, and use tools. The result? Human respondents achieved 92% accuracy, while GPT-4 equipped with plugins managed only 15%.
The authors constructed 466 questions, publicly releasing all question statements—but withholding answers to 300 of them in the backend to power a leaderboard. Unpublished answers mean the leaderboard cannot be gamed by memorizing questions (this interpretation is mine).
This gap remains jarring even today. It is not the old script of "human experts vs. machines"—in domains like law or chemistry, where specialized expertise is required, machines have long been approaching or even surpassing human performance. Instead, the GAIA authors deliberately went the opposite way: rather than chasing problems that grow ever harder for humans, they focused on tasks that ordinary people can reliably solve. Their judgment, stated explicitly in the paper, is that the threshold for general artificial intelligence lies in a system’s ability to demonstrate robustness on such tasks—approaching that of an average person.
Since GAIA’s release, the craft of agent evaluation itself has evolved significantly. Reading several primary sources side-by-side reveals three key shifts: (1) the evaluated object has changed from a single response to an entire work process; (2) test questions themselves must first undergo quality inspection; and (3) scores must be interpreted alongside cost and confidence intervals.
## From "Answering Questions" to "Getting Work Done"
AgentBench is a multidimensional benchmark (first released in August 2023, later published at ICLR 2024) that places "LLMs as agents" into eight distinct interactive environments. It does not assess single-turn responses but instead evaluates multi-turn reasoning and decision-making.
Its observed failure patterns are worth remembering: top-tier commercial models *can* indeed operate in complex environments—but their performance differs markedly from open-source models with ≤70B parameters. Crucially, the main causes of model failures are not knowledge gaps, but weaknesses in long-horizon reasoning, decision-making, and instruction following. In other words, models don’t fail because they “don’t know”—they fail because they “lose track of what they’re supposed to do” or “don’t follow instructions as given.” The paper’s proposed improvement paths follow this diagnosis: enhancing instruction-following capability and training on high-quality, multi-turn alignment data—neither of which involves feeding the model more knowledge.
Another counterintuitive finding in the paper is that code training exerts a dual effect across different agent tasks: it helps in some environments but offers no benefit—or even harms performance—in others. This contradicts the then-popular assumption that “more code training automatically makes better agents.”
This is precisely where “evaluating agents” diverges from “evaluating models”: evaluating models tests a single response; evaluating agents tests an extended work process. As the process lengthens, the limiting factor shifts from knowledge to stability.
## Test Questions Themselves Must Undergo Health Checks
The evaluation community has also learned a hard lesson: scores are not always trustworthy—not necessarily due to model flaws, but sometimes due to flawed questions.
SWE-bench is a well-known benchmark for software-engineering agents. Its questions derive from resolved GitHub issues in 12 open-source Python libraries: given a problem description and a codebase, the model submits a patch and runs the original PR’s built-in tests—the existing functionality must remain unbroken, and new tests must pass; both conditions must hold for a solution to count as “solved.” This sounds objective.
On August 13, 2024, OpenAI and the benchmark’s original authors jointly conducted a large-scale human annotation effort—with discouraging results: 38.3% of samples were judged by annotators to have ambiguous problem descriptions—unclear about what the issue is or how to resolve it; 61.1% of samples had evaluation unit tests that could misclassify perfectly valid solutions as incorrect. A third category of trouble also emerged: development environments sometimes failed to set up properly, causing tests to fail—even for completely reasonable patches.
The annotation was rigorous—not a perfunctory exercise: 93 Python-experienced developers independently scored 1,699 randomly sampled instances, with each instance rated by three annotators; the strictest rating among the three determined the final score. Ratings used a four-point scale (0–3), and any item scoring ≥2 was deemed不合格 (unacceptable). The authors acknowledge that this conservative approach may mistakenly discard some usable questions—but ensures that retained questions are highly trustworthy. Of these 1,699 annotated samples, 68.3% were discarded.
Their illustrative example is telling. One scikit-learn question states: “A user passed the `copy` parameter to a certain function, but the library ignored it.” At first glance, this seems straightforward—just fix it. Yet the evaluation test was extracted directly from the original PR, requiring not only behavioral correction but also emitting a deprecation warning—and the warning text must match *exactly*, character-for-character. That exact wording emerged only after multiple rounds of discussion in the PR thread—a context invisible to the model, which sees only the problem description. This question is nearly impossible to solve—not because it’s difficult, but because the grading criterion is hidden outside the question itself.
In response, the two teams launched SWE-bench Verified: a subset of 500 manually verified “problem-free” samples replaces the original test set. Difficulty is now defined via human time estimation: annotators estimate how long it would take an engineer familiar with the codebase for a few hours to write the patch, categorizing into four tiers—under 15 minutes, 15 minutes to 1 hour, 1 to 4 hours, and over 4 hours. When constructing the subset, they prioritized the latter two tiers, then filled remaining slots randomly to reach 500. For reference, 77.8% of the original test set was estimated to require under one hour. This time-estimation protocol carries two built-in caveats: it assumes the engineer can conceive a solution, and—since large models may have memorized parts of the codebase and PRs—it likely overestimates difficulty for such models.
The impact of this “health check” was immediate: the same GPT-4o scored 16% on the original SWE-bench, but jumped to 33.2% on SWE-bench Verified when paired with the best-performing open-source scaffolding—the paper describes this as “more than doubling.” And this gain wasn’t achieved by simplifying questions: when re-calculating scores stratified by difficulty tier, performance rose *within every tier*, indicating that discarded items were unanswerable ones across all difficulty levels—not just the hardest.
The methodological significance of this far outweighs the numbers themselves: evaluation infrastructure itself requires evaluation. Before interpreting a score, ask first whether the questions have been validated—and only then ask whether the model is strong.
## Cost, Confidence Intervals, and Process All Appear on the Leaderboard
The units of measurement in evaluation are also shifting. Terminal-Bench (current version 4.1) declares on its official website—hosted jointly by Stanford University, Harbor Research Institute, and Laude Research Institute—that it aims to “measure and evolve alongside the frontier of agent work.” Its leaderboard does not report a single number: alongside resolution rate, it displays cost and token consumption, and each row includes error bars representing 95% confidence intervals.
These three added dimensions each address a practical question: resolution rate answers “Can it finish the task?”; cost answers “How much does one execution cost?”; and the confidence interval answers “How much luck is involved in this ranking?”
Why is the confidence interval so critical? An offhand observation in OpenAI’s paper provides evidence: the same GPT-4 achieved only 2.7% resolution on SWE-bench Lite using an early retrieval-based scaffolding, but jumped to 28.3% when switched to the CodeR scaffolding. The model was identical. Every row on the leaderboard is actually a product of “model × scaffolding × luck”; reporting only peak performance hides the other two factors.
## My Three Judgments
The following synthesis emerges from reading the above materials side-by-side—not conclusions drawn from any single source.
First, evaluating agents is fundamentally about evaluating *process capability*. The three major failure modes identified by AgentBench—long-horizon reasoning, decision-making, and instruction following—are none of them knowledge deficits; the suggested improvements are likewise process-oriented. Likewise, GAIA’s 92% vs. 15% gap shows that what stalls machines isn’t professional depth, but the ability to reliably complete an ordinary, mundane task end-to-end. The field’s most important pivot in recent years has been shifting evaluation focus from “how many answers are correct?” to “does the task get completed?”
Second, high scores presuppose fair questions. In the original SWE-bench, over 60% of evaluation tests risked misclassifying valid solutions as incorrect; nearly 70% of annotated samples were discarded. Scores achieved on such flawed questions may measure not competence, but “the ability to game bad questions.” Quality assurance carries real costs: 93 developers, 1,699 samples, triple independent annotation—this pipeline itself demands money and time. Whether it’s worthwhile depends entirely on what decisions you intend to make using those scores.
Third, look at confidence intervals and cost—not just peak scores. Terminal-Bench’s inclusion of confidence intervals, cost, and resolution rate treats “reliability” and “economic efficiency” as intrinsic components of capability. Meanwhile, GPT-4’s jump from 2.7% to 28.3% demonstrates that no single-point score should ever be treated as an inherent property of the model itself. For users, the first question when selecting an agent should not be “What’s its highest score ever?” but rather “What’s the probability it completes this class of tasks in one go—and at what cost?”
## Key Takeaways
If you select tools based on evaluation scores: before trusting any leaderboard claim, ask three questions—Have the questions undergone human verification? Is the evaluation measuring a single response or an extended work process? Are confidence intervals reported—not just peak scores—and is the scaffolding clearly specified?
If you’re developing agents: before building your own task suite, rigorously audit your evaluation tests—those figures of 38.3% and 61.1% show that poor-quality questions are not rare edge cases; log cost and latency from day one; and define difficulty by “how long would a knowledgeable person take?”—a far more useful approach than categorizing by question origin.
If you’re just observing: the next time you see a headline like “Model X Surpasses Humans,” pause to reflect on GAIA’s 15%. Where machines truly lag isn’t in exams humans find difficult—but in the trivial, everyday chores humans consider beneath notice.
What “92% for humans, 15% for AI” chores exist in *your* work? Feel free to share in the comments.
Source: GAIA Benchmark Paper Abstract Page https://arxiv.org/abs/2311.12983