Writing

How to verify AI agent output

By Scott Wilder · November 9, 2026 · 8 min read · All writing

Verify AI agent output by turning every request into a check you can run, then running it against evidence, not against the agent's own opinion. Tencent Cloud's validation guide says each requirement should map to a check, and that fact checking "must return to evidence rather than asking the same model to grade its own response." In practice that means four layers: facts, files, format and completeness.

This page is the hub for keeping an AI agent on a short leash. It answers the main question first, then covers spending controls and coding agent sessions, and links to the deeper pages.

Quick version

Verify AI agent output: the four layers

An agent can sound sure and be wrong. The fix is boring: a short list of checks, written down, run the same way every time.

Layer Question to ask Best checked by
Facts Does a primary source support each claim that matters? A person reading the source
Files Do the files exist, with the right names and required fields? Rules and scripts
Format Does the output parse as the format it claims to be? A parser
Completeness Is anything missing, and is anything there that should not be? A list comparison

The table follows the structure in Tencent Cloud's guide to validating agent outputs. The company is not selling anything related in that article.

Facts: go back to the source

The first rule is simple. Do not ask the same model to grade its own work. The guide says consequential claims "should be traced to official documentation, original data, or another authoritative primary source."

Count of citations is not the test either. The same guide puts it this way: "A large number of citations is not proof: ten pages repeating the same unsupported statement still provide one weak chain of evidence." Open the links. Confirm the page says what the agent says it says.

Watch for invented filler too. The guide notes that generative systems "can add plausible details to make an answer flow smoothly," and tells reviewers to flag anything that was absent from the inputs and unsupported by cited sources but presented as fact.

Files and format: let a machine check

Some checks do not need judgment. Tencent Cloud lists file existence, naming, required fields, schema validity, date formats, broken links and exposed secrets as things that "can often be checked with rules." Run those first. They are cheap and they do not get tired.

For structured output, use the right tool. The same guide says to "parse structured output with the appropriate parser rather than deciding that it visually resembles valid JSON, XML, YAML, or CSV." The reason is practical: "One trailing comma, duplicate key, or escaping error may break the next system in the workflow."

Completeness: expected set against actual set

An agent that quietly skips three of ten items looks fine until someone needs item seven. The guide recommends an expected-set versus actual-set comparison: list the files, records, dates, sections or entities that should be processed, then reconcile that list with the output. This catches omissions and unexpected additions.

It also says that when some data is unavailable, a valid result describes the gap and its effect instead of silently omitting it. Make that a written requirement. An agent that reports "I could not reach this source" is more useful than one that fills the hole.

Judgment checks and the pass line

Some criteria are subjective: tone, fit, whether the answer is actually useful. Keep those separate from the rule checks, and keep them short. Tencent Cloud advises setting the acceptance threshold before seeing the result. Otherwise, reviewers may unconsciously relax requirements for a polished output or add new expectations after a rough one.

Our view: this is the check most people skip. A clean looking result invites a quick approval. Write the pass line first, and the result has to meet it.

AI agent spending controls

Verification covers whether the output is right. Spending controls cover what it cost to get it. Our full write-up is on AI agent spending limits. Here is the short version.

The cost problem is widely recognized. A governance article from nOps says Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. nOps sells AI cost governance and spend management software, so read its framing with that in mind. The same page reports that 98% of respondents to The State of FinOps 2026 report now manage AI spend, up from 63% the year before. That is a rise of 35 percentage points, from 98 minus 63.

Know how the meter runs

Pricing details matter once an agent loops. Per the nOps page, citing Anthropic's pricing, output on Claude's Sonnet 5.5 and Opus 5.5 models is priced at five times input. The same page says cache reads on those models cost 5% of the base input price, and on most other current models 10%. A five minute cache write costs 1.25 times base input. Prices change, so check the provider's own pricing page before you plan around any of these figures.

The plain takeaway: long outputs cost more per token than long inputs on those models, and an agent that loops without a cap can run up a bill while you sleep.

Put a limit and an owner on every agent

Microsoft's Cloud Adoption Framework is written for organizations, but the ideas scale down. It says a governance baseline "sets the minimum requirements that every agent must meet before it is allowed to operate." It also says to record every agent in a single inventory that tracks ownership, purpose, platform and access scope, and to give each agent a distinct identity. Microsoft's page names Microsoft Entra Agent ID for that job.

For one person running a few agents, the version is a short list: which agents exist, what each can touch, and who answers for it. Our view: if you cannot say what an agent can access, it has too much access.

Tooling can help. nOps describes hourly anomaly detection that flags anything running 3x over the same hour in the prior week, which is an example of the kind of alert worth having. Microsoft's Agent Framework documentation lists bounded looping among the features Go supports, which points at the same idea: cap the loop.

Coding agent session management

Coding agents add a second problem. Work happens across long conversations, and the output is changes to real files. Verification and session hygiene go together.

What a session is

In VS Code, a session groups the workspace, code changes and conversations for a task. Finished sessions can hide. The docs say that if a session is completed, it may not appear in the active list, and you change the filter to Archived in the Chat view or Done in the Agents window.

Two features change what the agent knows. Compaction summarizes earlier history to free context space and reduce tokens sent with subsequent requests. That saves tokens, but a summary can lose detail, so recheck anything important afterward. And a nested session starts blank, without the other chats' history, while a fork keeps the source history so you can try an alternative without losing the original discussion.

On archiving, the same page says that for a worktree session, archiving commits uncommitted changes to the session branch before removing the worktree folder. Know that before you archive.

One session, one checkout

Arcadia Mission Control's guide on coding agent session management gives a clear rule: use one live session per checkout. Sharing a checkout, it says, risks mixed commits and overwritten edits. Its fix is to use one worktree per concurrent editing session, since Git worktrees provide separate working directories and branches.

Worktrees are not enough alone. The guide says they isolate working files but do not preserve decisions or isolate shared Git storage, services and databases. So keep intent, decisions and handoff notes in project files.

Handoffs are verification records

A good handoff is a small proof of work. Arcadia's guide says it should identify the branch and candidate revision, the passing check and the outstanding criterion. That is the same idea as the checks above: say what passed, say what is left, and make the next session confirm the previous one stopped before reusing its workspace.

This is a practical match for a day job and one spare hour. You stop, write the state down, and start the next night from the file instead of from memory.

What not to hand to an agent at all

Checks reduce risk. They do not remove it. Some work should stay with you. We cover that in what to never automate with an AI agent. Our view: if a mistake would be expensive, hard to reverse, or would reach other people, a human makes the final call.

Microsoft's guidance points the same way for companies. It says to run adversarial tests on all agents before production release and after significant updates. For a solo builder, that translates to trying to break the agent with bad input before you trust it near anything that matters.

A starting checklist

  1. Write the requirements as checks. Each one gets a pass or fail.
  2. Set the acceptance threshold before the run.
  3. Run the rule checks: files, names, fields, links, secrets.
  4. Parse every structured output.
  5. Compare expected against actual for completeness.
  6. Trace each consequential claim to a primary source.
  7. Record the agent, its access and its spending cap.
  8. End each coding session with a written handoff.

The site's author, a former law enforcement officer, writes about this kind of discipline for people with limited time. You can read more on the About page.

Keep the leash short

The newsletter, Sunday Night Truths, is one email every Sunday at 7 PM Central with one hard truth about work, money or AI agents, readable in two minutes. Signing up also sends The Agent Leash, a one-page checklist of 15 checks before your AI agent touches your money. It is free. Get Sunday Night Truths.

Questions

Can I ask the AI to check its own work?

It is a weak check. Tencent Cloud's guide says fact checking must return to evidence rather than asking the same model to grade its own response. Use a primary source, a parser or a rule instead.

Which agent output checks can be automated?

Rule based ones. Tencent Cloud lists file existence, naming, required fields, schema validity, date formats, broken links and exposed secrets as checks that can often be done with rules. Facts and judgment calls still need a person.

Why do many citations not prove a claim?

Because they may all repeat one source. The guide says ten pages repeating the same unsupported statement still provide one weak chain of evidence. Open the sources and confirm they support the statement.

Why use one worktree per coding agent session?

It avoids collisions. Arcadia Mission Control says sharing a checkout risks mixed commits and overwritten edits. It also notes worktrees alone do not preserve decisions, so keep notes in project files.

Does context compaction affect output quality?

It can. VS Code's docs say compaction summarizes earlier history to free context space and reduce tokens sent with subsequent requests. Our view: after a compaction, recheck any detail that matters.