A demo only needs to run once. Production systems must always answer three questions: What did it just do? Why did it do that? And how much did it cost? The first question is the hardest. Traditional software has mature answers—logs, metrics, and distributed tracing—backed by decades of refined tooling. But when the program itself emerges from probabilistic reasoning, its control flow is generated at runtime, and its memory evolves with the conversation, 'a single execution' loses repeatability: identical inputs no longer map to identical code paths, and 'what it just did' no longer has a unique answer. This field is now called AgentOps. This article reads four primary research papers and one industrial standard in parallel to see how far this answer has progressed.
## Two Perspectives, One Chasm
A paper from August 2025 states the problem directly. Its authors are from the University of California, Santa Cruz; ShanghaiTech University; and the eunomia-bpf open-source community. They point out a critical semantic gap in existing tools: either you observe the agent’s high-level intent—i.e., prompts and model traffic—or you observe its low-level actions, such as system calls—but these two perspectives cannot be causally linked. This blindness makes it hard to distinguish among three cases: benign operations, malicious attacks, and expensive failures.
Their proposed solution is boundary tracing: instead of instrumenting application code, monitor the agent externally at stable system interfaces. Use eBPF to intercept TLS-encrypted model traffic and extract semantic intent, while simultaneously monitoring kernel events to observe system-level effects. Then apply a real-time engine plus a secondary model to causally correlate the two streams across process boundaries. The entire process requires no agent-side instrumentation and remains robust against rapid changes in upper-layer interfaces. The paper reports performance overhead below 3%. Evaluation covers three use cases: detecting prompt injection attacks (a separate topic, not elaborated here), identifying resource-wasting inference loops, and exposing hidden coordination bottlenecks in multi-agent systems. The project is open-sourced, with code hosted under the eunomia-bpf community.
## Three Frameworks, Three Cuts
The first framework comes from a paper published in November 2024. Three researchers conducted a systematic mapping study of existing DevOps tools: a single query for 'Agent and DevOps' alone returned 618 repositories on code-hosting platforms; they filtered the top three pages of results, aggregated several queries, and analyzed 17 tools individually for features and limitations. Their output is an entity-relationship model of agent artifacts: user goals sit at the top layer; agent spans handle tasks; knowledge bases supply data and context; reasoning spans generate plans; each plan span invokes one or more model spans; and plans ultimately materialize as workflows. These nested spans constitute the objects that tracing should cover. Their taxonomy is positioned as a reference template for designing and implementing AgentOps infrastructure, supporting monitoring, logging, and analysis capabilities. Their stance is blunt: from a DevOps perspective, enabling observability for agents is a necessary condition for AI safety.
The second framework appears in a survey submitted in June 2026 and updated in September 2026: anomaly-first. It cites Princeton’s HAL leaderboard, noting that success rates across various agent systems remain generally low—and quotes verbatim that SWE-Agent’s success rate remains below 40%—with numerous anomalies impeding task completion. Prior work assumed anomalies occur mainly at specific steps during task execution; this survey pushes the boundary outward in both directions. Before execution, intent recognition and dynamic workflow construction may fail—the workflow is not static but continuously updated as the task evolves. After execution, task completion does not equal task success: a workflow may terminate normally yet produce defective outputs; a root-cause analysis may proceed without visible anomalies yet fail due to premature termination or insufficient granularity. It defines AgentOps as 'using various operational techniques to maintain agent systems' (original text: 'Agent System Operations'), classifying anomalies into intra-monolith and inter-monolith categories, covering the full lifecycle—before, during, and after execution. Inter-monolith anomalies persist throughout: before execution, they primarily involve task specification and orchestration failures, often triggered by ambiguous, incomplete, or malicious inputs and inappropriate sub-agent composition; during execution, communication and trust anomalies may arise when agents exchange information. The framework outlines four stages: monitoring, anomaly detection, root-cause localization, and resolution; resolution is further subdivided by intervention timing into three types: pre-execution prevention, in-execution correction, and post-execution recovery. A section on datasets and benchmarks appears at the end.
The third framework originates from a paper by IBM Haifa Research Lab, published in July 2025: role-first. Developers, testers, site reliability engineers, and business users—four roles interact with agent systems at different lifecycle points. The paper proposes a six-stage automated pipeline: behavioral observation, metric collection, problem detection, root-cause analysis, optimization recommendations, and runtime automation—closing the loop from seeing behavior all the way to automatic action. Its title states its stance plainly: the key role of automation is to manage uncertainty and enable self-improving systems—not to eliminate uncertainty, but to tame it, ensuring safe, adaptive, and effective operation.
The three frameworks cut along mutually exclusive dimensions: one by artifacts, one by anomalies, and one by roles and pipelines. Even the term 'AgentOps' itself carries slightly different meanings across the three papers. This is not about right or wrong—it reflects the field’s lack of consensus on 'what to observe, and for whom.'
## Standards Are Still Moving
Academic consensus is absent; industrial standards are still in flux. OpenTelemetry’s GenAI semantic conventions—whose official documentation page now displays only the word 'Moved'—have been split into a standalone repository, extending OpenTelemetry’s existing semantic conventions to cover GenAI clients, MCP (Model Context Protocol), and provider-specific spans, metrics, and events, managed via the Weaver tool. Notably, the Schema URL in the new repository’s documentation remains marked TODO: the canonical identifier for this convention has not yet been finalized. Semantic conventions define which fields record a model call—this foundational element itself remains in transit.
## Three Judgments
Judgment One: AgentOps is not an extension of APM [this article’s judgment]. APM traces call stacks: identical code follows identical paths, and averaging across multiple runs reveals truth. AgentOps confronts decision paths: plans are generated ad hoc, memory evolves with dialogue, and code may even be written and executed by the agent at runtime. Single executions are non-repeatable; thus, the object of tracing shifts from 'which line is slow' to 'why did it choose this step, what was its basis, and what were the consequences?'
Judgment Two: The three frameworks’ lack of convergence signals youth—not chaos [this article’s synthesis]. Each paper addresses a distinct question: what to trace (artifact model), what constitutes an anomaly (anomaly taxonomy plus four-stage process), and who uses it (roles plus pipeline). Since the questions themselves are misaligned, the answers naturally diverge.
Judgment Three: In the short term, the most telling signal is not new tools—but when standards settle [this article’s judgment]. The full-page 'Moved' notice and the dangling Schema URL for GenAI semantic conventions indicate that 'what counts as an observation' remains under negotiation. Until definitions stabilize, tools cannot be fairly compared, and data cannot interoperate.
## What You Can Take Away
For teams building agents: Before selecting a tracing solution, first clarify which layer of artifacts you need to observe—goals, plans, model calls, or workflows. This nested span entity-relationship model provides an off-the-shelf reference.
For platform SREs: The four-stage cycle—monitoring, detection, localization, resolution—can serve as a closed-loop checklist. Also, set separate monitoring points for 'task completion' and 'task success'; the latter is costlier but more valuable.
For evaluation researchers: Evaluation asks 'how well was it done?'; AgentOps asks 'what did it just do?' Evaluation has been discussed elsewhere; the two are complementary—not interchangeable.
For general readers: Next time you see a demo of an agent product, ask: Does it have a flight recorder? Can the recording be replayed?
If your agent has a flight recorder, which segment would you most want to replay? Let us know in the comments.
## Definitions and Boundaries
All four papers were read in full via their preprint versions and cached locally. Both OpenTelemetry references are preserved at the page level. The June 2026 survey has an earlier arXiv record: arXiv:2508.02121, submitted in August 2025, originally titled 'A Survey on AgentOps', with an abstract nearly identical to the later version; this article cites only the newer record, disclosed here to avoid duplication. The 17 tools are not listed individually to prevent brand listing. 'Below 3%' and 'SWE-Agent success rate below 40%' are quoted verbatim from the respective papers. 'Princeton’s HAL leaderboard' is cited per the original footnote: the main text mentions only 'leaderboard', while the footnote gives the URL hal.cs.princeton.edu—the name 'HAL' derives from that address. 'AgentOps' and 'span' retain their English forms per convention, without forced translation. The stance statement by the IBM authors is attributed to the original authors; this article only paraphrases it, without inferring commercial motives. Prompt injection detection is mentioned in the main text only in passing, as it is not the focus of this article. Agent evaluation is covered in another article; this article explicitly excludes it. This article does not constitute a recommendation or evaluation of any product or project.
## Sources
AgentSight: System-Level Observability for AI Agents Using eBPF, arXiv:2508.02736, full-text level, https://arxiv.org/abs/2508.02736
AgentOps: Enabling Observability of LLM Agents, arXiv:2411.05285, full-text level
Agent System Operations: Categorization, Challenges, and Future Directions, arXiv:2606.01581 (v2), full-text level
Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems, arXiv:2507.11277, full-text level
OpenTelemetry GenAI Semantic Conventions official documentation page ('Moved' notice preserved verbatim), page-level
semantic-conventions-genai standalone repository README, page-level