Field notes · running agents
When AI agents are confidently wrong
An agent proposed inverting a live content-security policy — swapping the working domain for a stale one from an unrelated project — and wrote a fluent rationale for why that was correct. The summary was excellent. The diff was backwards.
What was asked
The request was not a coding task at all. It was, roughly: check the search-performance data and the recommendations, and tell me what to do. An analysis question.
At that moment the system had exactly one agent, and it wrote code. So the question went to the coding agent, which did what a coding agent does: it produced a code change.
What came back
A proposed edit to the site’s content-security policy, removing the live authentication domain and inserting a stale one belonging to a different project entirely. Attached was a confident explanation that the stale domain reflected the current brand — a fact with no basis, stated plainly.
Shipped, this breaks sign-in for every user. Not subtly — completely.
Why the summary was the dangerous part
Read the summary alone and there is nothing to object to. It names a real file, describes a plausible fix, and uses the right vocabulary. Anyone approving from summaries would have approved it, and would have been reasonable to.
That is the whole problem in one line: fluency and correctness come from the same process and are not correlated. The model writes an equally convincing sentence either way. If the sentence is what a human reviews, the review is of the model’s account of its own work — precisely the thing that needs checking.
It was caught because the approval screen showed the diff. Comparing two domain names takes a few seconds and the inversion is obvious. The gate did not need to be clever; it needed to show evidence.
What we changed
Route non-code questions away from the code agent. Analysis questions now reach an analysis path that reads live data and answers with numbers, instead of being force-fit into a diff.
Let the coding agent return nothing. Its planning prompt now states explicitly that if a goal needs no code change — or is not a code task at all — it must return no edits and say so, and must never force a non-code goal into a plausible-sounding change.
Keep the diff at the gate. Non-negotiable, and the only reason this was a near-miss rather than an incident.
The uncomfortable part
The agent was not misbehaving. It was not ignoring an instruction or exceeding a boundary. It was doing its job on a question that was never its job, and it did that job the way it does every job: confidently.
Which is why prompt-level rules are the wrong layer for this. You cannot instruct a system out of confident error, because from the inside, error and compliance look identical. What works is structural: give the agent less room to be asked the wrong question, and make sure the human at the gate is looking at what will actually happen rather than at a description of it.
The plan was denied with a one-line reason, the run cleaned up, and no pull request was opened. That is the system working — not the absence of a failure, but a failure that cost a few seconds of reading.
FAQ
- Do AI agents hallucinate when writing code?
- They produce plausible output that is wrong, and the code case is more dangerous than the chat case because the output is executable and arrives with a rationale. In one run an agent proposed swapping a live authentication domain for a stale one from an unrelated project, and explained the change as aligning with 'the current brand' — a fact it invented. The change would have broken sign-in for everyone.
- Why do agent summaries hide bad changes?
- Because fluency and correctness are produced by the same process and are not correlated. A model writes an equally confident summary whether the underlying diff is right or wrong, and the summary is the artifact humans read. Approving a paraphrase means approving the model's account of its own work — which is exactly the thing under review.
- How do you catch a confidently wrong agent plan?
- Show the diff, not the description, at the moment of approval — along with the exact target identity, the tests that ran, and the policy outcome. A human comparing two domain names spots an inversion in seconds; the same human reading a summary about 'fixing the content-security policy' approves it. The gate only works if it presents evidence.
- Can prompt instructions prevent this failure?
- Not reliably. The agent in this case was not violating an instruction — it believed it was completing the task correctly. You cannot instruct your way out of confident error, because the error and the compliance are indistinguishable from the inside. Structural answers work: show evidence, and refuse to let non-code goals become code changes.
- What was the root cause of the wrong plan?
- A non-code question routed to a code agent. The request was to review search-performance data and recommend actions; the only agent available wrote code, so it manufactured a plausible code change. The fix had two parts: let analysis questions reach an analysis path, and instruct the coding agent to return no edits when a goal is not a code task at all.
see the whole flow, live
Everything on this page — the queue, the evidence, the hold-to-approve, the audit trail — is working in the VIVUU terminal as an interactive preview on realistic mock data. No sign-up.