Guide · running agents
Monitoring AI agents in production
Most agent monitoring measures the cost of thinking. What you need are the few signals that predict harm — attempted actions at the boundary, policy denials, decisions waiting on a human, and runs that finished without proving anything.
Activity metrics describe; risk signals predict
The default agent dashboard shows runs, tokens, latency and a success rate. Every number describes activity, and none of them would have moved before any incident worth having a dashboard for.
Four signals actually earn their place:
Attempted actions at the boundary. Not what the agent did — what it tried to do that policy had to rule on. A rising count of attempts at the edge of the permitted set is the earliest warning that a plan has drifted from the goal.
Policy denials, clustered. One denial is noise. The same denial four times in a week is a system telling you something: either the vocabulary is wrong and legitimate work is being blocked, or an agent keeps walking into the same wall.
Decisions waiting on a human, and for how long. The only metric that measures you rather than the agent.
Runs that ended without a verified result. Success reported by the thing being measured is not evidence. A run that “succeeded” with an empty diff has done nothing, and it will report that cheerfully.
Success is not the same as a result
This is the trap specific to agents. Traditional monitoring assumes a process either completes or errors. An agent has a third state that looks identical to the first: it completes, reports success, and produced nothing — or produced something plausible and wrong.
A run that finishes with an empty patch is not a success, and a plan whose summary is confident tells you nothing about whether its diff is correct. So the pipeline should verify rather than believe: if the plan claimed edits and none applied, the run failed, and the reason each edit failed is the useful output.
Alert on what needs a decision
The fastest way to make monitoring useless is one notification per run. People learn to dismiss, dismissal becomes reflex, and the one that mattered is dismissed with the rest.
Push only two categories: something needs a human decision, or something deviated from what policy expected. Everything else — successful, in-policy, verified runs — should be visible when someone looks, and silent when they are not. An interface that only speaks when it needs you is the one people keep installed.
Watching is not controlling
Worth saying plainly, because the tooling market blurs it: a trace is a record, not a constraint. Instrumenting an agent thoroughly tells you precisely how it did the thing you did not want it to do.
Monitoring earns its keep when it feeds control — when a denial cluster changes a policy, a stalled approval changes who gets paged, and an unverified run stops the pipeline instead of decorating a chart.
FAQ
- What should you monitor for AI agents in production?
- Four things that predict harm: actions attempted at the boundary of what is allowed, policy denials and what they cluster around, approvals waiting on a human and how long they have waited, and runs that ended without a verified result. Token counts and latency are operational metrics — useful for cost, silent about risk.
- Is agent observability the same as agent monitoring?
- Observability is the ability to reconstruct what happened from traces and logs. Monitoring is noticing while it matters. Agent tooling is heavily weighted toward the former, which is why teams often have a beautiful trace of an incident and no signal that fired before it. Both are worth having; only one wakes you up.
- Why are token dashboards misleading for agents?
- They measure the cost of thinking, not the risk of acting. An agent can burn a trivial number of tokens and open a pull request that breaks production, or spend heavily and change nothing. Cost per run is worth tracking as a budget line — it just answers a different question from whether the agent is behaving.
- What is the most important agent metric?
- Time-to-decision on parked approvals. Everything else describes the agent; this one measures the system that includes you. An agent blocked for nine hours because nobody looked is not a safe agent — it is a stalled one, and the pressure that builds is what causes someone to approve a batch without reading it.
- Should agent alerts be per-run or per-anomaly?
- Per-anomaly. One notification per agent run trains people to dismiss notifications, and the dismissal habit is what makes the important one invisible. Surface what needs a decision or has genuinely deviated; let successful, in-policy runs be visible on demand rather than pushed.
see the whole flow, live
Everything on this page — the queue, the evidence, the hold-to-approve, the audit trail — is working in the VIVUU terminal as an interactive preview on realistic mock data. No sign-up.