Agent Observability Platforms: Tracing, Evaluation, and Team Management Compared

On Monday morning, an engineering lead has two problems. A coding agent’s pull request says the tests passed, but the test step never ran. And a research agent’s report has been waiting for review since Thursday, and nobody can say whose decision it is. Both get called “agent observability.” They need different records.

The first question is about a run: what the model saw, which tools it called, and where its output went wrong. Tracing and evaluation platforms answer it. Braintrust, Arize Phoenix, LangSmith, and Galileo, now sold as Splunk Agent Observability, record each step of a run and help you score whether the output was good.

The second question is about work: which task exists, who owns it, what it is waiting on, who must approve it, and what it has cost so far. A work-management layer answers it. AgentCenter and Paperclip keep tasks, owners, and reviews that outlast any single run.

So choose by the object you need to inspect or control. If your agents give wrong answers and you can’t see why, start with tracing and evaluation. If the answers are fine but work stalls between agents and people, start with work management. Many teams running several agents will need both, connected by a link from the task to the trace. A single developer with one or two agents may need neither yet.

What a trace records, and what a task record adds

A trace records one execution end to end. Each step inside it is a span: a model call, a retrieval, a tool call, with its inputs, outputs, timing, and usually token counts. An evaluation scores outputs against a test set or a rubric, so you can tell whether a prompt or model change made things better or worse. These tools are built around the run.

A task is a unit of work with an owner. One task may take several runs from several agents, a change request from a reviewer, and a human decision before it is done. The task record keeps what the traces can’t: who has it now, what it is blocked on, and what was decided.

The two records get confused because the products overlap. Tracing tools now let you assign rows to reviewers. Work-management tools keep run logs. The word “approval” covers at least four different things:

• Annotation: a person labels a trace or an output as good or bad, usually to build evaluation data.
• Runtime control: a policy checks a model or tool input or output during the run and blocks or changes it.
• Action approval: an agent pauses before a specific action until someone approves it.
• Work approval: a person signs off on a finished deliverable before it ships.

When a product says it supports approvals, find out which of these it means. The comparison below names which ones each product documents.

The six platforms at a glance

“Best for” is our judgment from the documented scope, not a vendor claim. Each entry below names what the product documents and where that documentation stops.

Braintrust: traces, scores, and a review queue

Braintrust describes itself as an observability platform for instrumenting, understanding, and improving agents. A trace is one end-to-end execution made of spans, and each span shows its duration, total tokens, and estimated LLM cost. On top of that record sit experiments, scorers, and datasets for offline evaluation, and online scoring, which scores production traces in the background as they are logged. Alerts go to webhooks, Slack, or PagerDuty.

Its human review goes further than most tracing tools toward work management: you can assign rows in logs, experiments, and datasets to teammates and work through a review queue. That assignment covers a trace or a dataset row, not a business task with dependencies and a final sign-off.

Deployment. SaaS is available on Starter, Pro, and Enterprise plans; BYOC and self-hosted deployments require Enterprise. In every option, Braintrust operates the control plane: the web UI, authentication, and organization metadata. Self-hosting moves the data plane, where traces, datasets, and prompt completions are stored, into your infrastructure: the Braintrust API, PostgreSQL, Redis, object storage, and its Brainstore database.

Arize Phoenix: tracing you host yourself

Phoenix is built on OpenTelemetry and OpenInference instrumentation. It scores traces and spans with LLM-based evaluators, code checks, or human labels; groups traces into datasets you can rerun against a new version of your application; and versions prompts. It also rolls up token costs to the trace and project level.

Deployment. Phoenix runs as a container with a web UI, a trace collector, and a SQL database: SQLite by default, PostgreSQL for production and multiple users. Arize’s docs say there are no license fees, usage limits, or feature gates. The license is the Elastic License 2.0, which is source-available rather than an OSI-approved open-source license. The database, upgrades, and sign-in configuration are yours to run.

Phoenix is not Arize AX. According to Arize’s FAQ, AX is the managed commercial platform, and production monitors, alerting, and dashboards for non-developers belong to AX. If you need managed hosting or alerting, you are comparing AX, not Phoenix.

LangSmith: tracing and evaluation, with adjacent products for running agents

LangSmith records each unit of work as a run, collects the runs for one operation into a trace, and links traces into threads. LangChain and LangGraph integrate directly, but manual instrumentation adds tracing to any code regardless of framework, and it accepts OpenTelemetry traces. Evaluation is offline, before you ship, and online, in production. Annotation queues group runs and threads under a rubric and track reviewer progress. Alerts, including a cost alert, route to Slack, PagerDuty, or a webhook.

LangSmith is also more than an observability tool, and the adjacent products matter for this comparison. LangSmith Deployment is a runtime for agent workloads. LangSmith Fleet, formerly Agent Builder, documents agents that pause and request approval before specific actions. The beta LLM Gateway enforces spend caps by organization, workspace, API key, or user, and blocks any request that would exceed the cap. Those are documented action approvals and hard spend limits, so don’t assume they are missing because the tracing product is the part you evaluated.

Deployment. Cloud has US, EU, and APAC regions. BYOC and self-hosted are available on the Enterprise plan. In BYOC, LangChain runs the control plane and the data plane runs in your cloud; in self-hosted, an Enterprise add-on, you run both.

Galileo, now Splunk Agent Observability: evaluation with runtime controls

Cisco completed its acquisition of Galileo in May 2026. A banner on Galileo’s documentation says that as of August 7, 2026, Galileo is Splunk Agent Observability, and that its docs apply to customers who onboarded before that date; newer customers use the Splunk documentation. Expect names, plans, and pricing to move toward Splunk’s. Galileo’s pricing page now redirects to Splunk.

The product logs sessions, traces, and spans and evaluates them with built-in metrics, LLM-as-judge, or custom code. Its distinctive layer is Agent Control: a centralized layer that evaluates LLM and tool inputs and outputs while the agent runs, to block harmful content, prompt injection, or PII leakage without changing agent code. Controls can be reused across agents, and Agent Control itself is an Apache-2.0 project with its own documentation, where policies deny, steer, or observe. Galileo’s older Protect feature is deprecated in favor of it.

That is runtime governance, and it would be wrong to call Galileo a tool without controls. It governs what a step may say or do during a run. The docs we checked didn’t establish a durable task record with an owner, or a sign-off on a finished deliverable. Galileo records model costs, but we found no documented spend cap.

Deployment. The current Splunk overview offers Agent Observability on-premises, where your organization installs, upgrades, and operates it, or as SaaS integrated with Splunk Observability Cloud. Feature availability varies by deployment option, so confirm that the features you need exist in the one you choose.

AgentCenter: a task board built for OpenClaw

AgentCenter calls itself a mission-control dashboard for OpenClaw agents, and it is the closest product here to Paperclip’s job. Tasks can be assigned to a specific agent or left in an inbox, and move through inbox, assigned, in progress, review, and done. Agents wake on a heartbeat, every 15 minutes by default, and claim one task at a time. A squad view shows every agent’s role, status, and current task.

Review is a work approval: tasks don’t move to done on their own, and a person or a lead agent approves the deliverable or sends it back with feedback. If the project has a lead agent, reviews happen automatically. Approval of individual agent actions wasn’t documented on the pages we checked. AgentCenter’s blog describes per-agent cost tracking and spending caps per task, per agent, and per project that pause the task, fail it, or send a notification, though the product documentation we checked doesn’t cover them. The homepage says it is built for OpenClaw agents and also invites agents from any stack, but it names no other supported runtime.

Deployment. The dashboard is hosted by AgentCenter or self-hosted as a single Docker image under a paid license. Agents connect from your own machines; optional cloud VMs for running them are limited to the top plan.

Paperclip: task owners, review stages, and budgets

Paperclip organizes agents into a company with goals, roles, and tasks. Each task has one assignee, and the task thread keeps the comments, documents, and decisions from every run that touched it. Agents run on the runtime you connect through an adapter, such as Claude Code, Codex, or Gemini CLI, so one task list can span agents from different vendors. Filtering by status shows what is blocked and what is in review.

Reviews are enforced at the point of completion. An execution policy can add a review stage and an approval stage to a task. When the agent doing the work marks the task done, Paperclip intercepts the change, moves the task to review, and assigns it to the reviewer. A reviewer who requests changes sends it back to the same agent. By default, the third consecutive change request from an agent reviewer goes to the task’s responsible person instead, so two agents can’t loop indefinitely.

Budgets act on the runs Paperclip manages. Paperclip records each run’s tokens and billed cost per agent and warns at 80% of a monthly budget. Before a run starts, it reserves an estimate against the budget and refuses the run if committed spend plus the estimate would exceed the limit. At 100%, it pauses the agent and cancels the agent’s queued and running work. The limits are real but bounded: provider calls already sent are still billed, and spend that doesn’t go through a Paperclip-managed run isn’t counted.

What Paperclip doesn’t do. It doesn’t instrument the spans inside a model call or score outputs against an evaluation dataset. Each heartbeat keeps a run transcript, but transcripts are stored per run and can scroll off; the permanent record is the activity log of Paperclip changes. Its optional OpenTelemetry export traces Paperclip’s own server requests, not your agents’ model calls. Its Tool Gateway can require approval for calls to connected MCP and HTTP tools, but it is experimental, off by default, and doesn’t govern a runtime’s own shell and file tools. If you need to know why a model answered the way it did, pair Paperclip with one of the tracing tools above.

Deployment. Paperclip is MIT-licensed and installs with one command on Node.js 24.11 or newer. For what self-hosting involves, see Self-Hosted AI Agent Stacks.

One incident, three records

Here is an illustrative incident. It isn’t a customer account, and it doesn’t imply any integration between the products named.

A research agent writes a weekly market summary. Its reviewer notices that one figure is three months old. What would each kind of tool record?

• The trace shows the run step by step. The retrieval span returned an archived report, the model quoted it accurately, and the final answer passed it along. The bug is in retrieval, not in the model’s reasoning. A tracing tool is the fastest way to find that step.
• The evaluation keeps the fix honest. You add this case to a dataset, write a scorer that checks figures against the dated source, and run an experiment on the retrieval change before shipping it. Online scoring can then flag similar answers in production.
• The work record shows who handles it. The summary task goes back to the agent with the reviewer’s note. A follow-up task to fix retrieval gets an owner and a due date. The person who approves the summary signs off on the corrected revision, and the cost of the extra runs lands in the agent’s spend. Paste a link to the trace into the task thread and you have both records in one place.

None of these records replaces the others. The trace can’t tell you who is fixing retrieval. The task thread can’t tell you which span returned the archived report. The evaluation can’t tell you whether the corrected summary was approved.

Choose by what you need to inspect or control

• Start with tracing and evaluation when agents produce wrong or inconsistent output and you can’t see why, or when you are changing prompts and models and need to know whether quality improved. Braintrust, Phoenix, LangSmith, and Splunk Agent Observability all fit. Phoenix fits when traces must stay on servers you run. Galileo’s Agent Control fits when you need policies that block content during the run.
• Start with work management when individual runs are good enough but work stalls: nobody knows which agent owns a task, reviews wait for days, or spend is visible only on the provider invoice. AgentCenter fits teams running OpenClaw agents that want a hosted board. Paperclip fits teams that run agents on several runtimes and want enforced review stages and per-agent budgets on a server they control. For a broader comparison of management tools, see 7 Platforms for Managing AI Agent Teams.
• Use both when several agents do work that people must approve. Keep traces in the tracing tool, keep ownership and decisions in the work tool, and link them through the trace URL. For where each layer can stop work, see What Is an AI Agent Control Plane? and Agent Harness vs. Framework vs. Runtime vs. Control Plane.
• Use neither yet if you are one developer running a few Claude Code or Codex sessions. The runtimes’ own session tools may be enough; see Manage Multiple Claude Code and Codex Sessions.

What each choice costs you

Instrumentation costs code and data. Someone has to add tracing to every agent, keep it working through framework upgrades, and decide how long to retain prompts and outputs that may contain customer data. Self-hosting moves that data onto your servers, along with the databases to back up: Phoenix’s PostgreSQL, Braintrust’s data plane. On Braintrust and LangSmith, self-hosting is an Enterprise option, and on Braintrust, the control plane stays with the vendor.

Work management costs discipline. A task board only knows what goes through it. Agents have to pick up work from tasks and report back to them, and someone has to keep owners and reviewers current. Runtime support is a real constraint: AgentCenter documents OpenClaw, and Paperclip supports the runtimes it has adapters for. Budgets in a work-management layer act per agent and per run, not per request. Paperclip can refuse a run and cancel an agent’s queued and running work, but it can’t recall a provider call that has already been sent, and it can’t count spend that happens outside the tool. Where you need a cap on every model request, a gateway that blocks requests over the limit, such as LangSmith’s beta LLM Gateway, acts at a different point.

Frequently asked questions

Is a tracing dashboard the same as a task dashboard?

No. A tracing dashboard aggregates runs: latency, errors, scores, and cost. A task dashboard aggregates work: who owns what, what is blocked, and what is waiting for review. A run can look healthy while the task it belongs to sits unreviewed for a week.

Does a cost chart stop spending?

Not by itself. Cost charts in Braintrust, Phoenix, LangSmith, and Galileo show spend. A limit has to act somewhere: LangSmith’s LLM Gateway blocks requests over a cap, and Paperclip refuses runs that would exceed an agent’s budget and cancels the agent’s queued and running work when it pauses. Check which spend each limit can see and where it acts: on each request, or on each run.

Are runtime safety controls the same as publication approval?

No. Agent Control checks model and tool inputs and outputs during a run. Publication approval is a person signing off on a finished artifact. A run can pass every guardrail and still produce a draft nobody should publish, so if you need both, configure both.

Can Paperclip replace a tracing tool?

No. Paperclip keeps run transcripts and a permanent activity log of Paperclip changes, but it doesn’t record spans inside model calls or run evaluations. Use it beside a tracing tool, not instead of one.

---

Before you compare platforms, take the last agent failure your team had and ask which record you couldn’t find: the record of what the agent did, or the record of whose job it was and what was decided. Buy for that missing record first, and keep each record in the system built to hold it.

Sources

Checked October 9, 2026, and rechecked October 21, 2026. Most vendor pages are undated and may change. These sources establish documented behavior, not results of our own tests.

• Braintrust: docs, traces, online scoring, human review, alerts, deployment options, self-hosting
• Arize Phoenix: docs, self-hosting, cost tracking, Phoenix and Arize AX, license
• LangSmith: observability, concepts, OpenTelemetry, evaluation, annotation queues, alerts, platform setup, self-hosted, cloud regions, Deployment, Fleet, Fleet oversight, LLM Gateway spend policies
• Galileo and Splunk Agent Observability: what is Galileo, logging, metrics, model pricing, Splunk Agent Observability overview, Agent Control (Splunk docs), Agent Control concepts, Agent Control repository, Splunk acquisition announcement
• AgentCenter: product, how agents work, tasks, agents, review, cloud VMs, pricing, cost dashboard (blog), spending limits (blog), self-hosting
• Paperclip: what is Paperclip, tasks, agent adapters, execution policy, costs and budgets, activity log, Tool Gateway, budget reservations source, observability, repository