Writing

How to Tell If an AI Agent Actually Finished the Task

By Scott Wilder · October 27, 2026 · 5 min read · All writing

An agent that stops talking has not finished the task. Treat "done" as a claim, then run it, diff it and ask for proof before you accept it.

By the end of this page you will have a definition of done you can reuse, a table that matches every claim to evidence, and one prompt to paste after every run.

The model going quiet is not a completion signal

The hard part of running an agent on a real task starts after the last token. Most frameworks treat the model's silence as the completion signal, and that is the mistake. A run can stop early, hit a limit or skip a step and still end with a confident summary.

Researchers call this false success: the agent says it is done and the state of the system says it is not. It spreads silently, unlike a crash, a refusal or an explicit handoff. In that paper, false success accounts for 45 to 48 percent of failures in the airline and retail domains tested, and 75.8 percent of failures among coding-agent runs that state a status. Those numbers come from benchmark setups, not your project. You find your own rate by running the check below.

Write the definition of done before the run

Verification depends on a definition of done that existed before the build started. If it lives only in your head, checking becomes a feeling.

Use this template. Three to six lines is enough:

Task: <one sentence>
Done means:
1. <thing that must be true, and where I can see it>
2. <thing that must be true, and where I can see it>
3. <thing that must NOT change>

The last line matters. "Must not change" is how you catch an agent that fixed the problem by breaking something else.

Match every claim to evidence you looked at yourself

An agent's closing message is a list of claims. Each claim needs one piece of evidence that does not come from the agent's own narration. A useful report ties each claim to an independent observation.

The agent says Evidence you need Where you look
"I only changed what you asked" A reviewed git diff with clean scope git diff, read every changed line
"Tests pass" The exact command, exit code 0, the test count, no unexpected skips Run the command yourself, fresh
"It works" The changed behavior tested directly Click it, call it, run it once by hand
"The record is updated" A read-only query of the real record The database, the file, the live page
"I sent / published / deployed it" The thing in the place it should be The inbox, the live URL, the dashboard

Pass line: every claim has evidence you looked at. One claim with no evidence means not done.

Paste this after every run

It forces the agent to turn its summary into a checkable table. A pre-completion checklist works the same way: the agent cannot declare done until it passes explicit checks, with a PASS or FAIL status and the exact evidence for each item.

Before you say you are done, give me a table: one row per
requirement in the definition of done, PASS or FAIL, and the
exact evidence for each row (file and line, command and output,
or the place I can open to see it). List every file you changed.
Confirm you did not delete, skip or edit any test. Fix every
FAIL before you answer.

Treat any FAIL as not done. Send it back, then ask for the table again.

Run it, diff it, or ask for proof: pick by output type

You do not need every check on every task. Pick by what the agent produced.

When the task repeats and the stakes are low, add a checklist gate the agent must pass before it may say done. When the stakes are high, add a human or a second agent on top.

Do not let a second AI be your only check

It is tempting to ask another model to read the transcript and judge it. Do not rely on that alone. LLM judges anchor on confident closing-message language, and false success runs produce exactly that language.

The same paper's conclusion is the rule worth copying: the fix that holds is a check per state-changing step, asserted against the real record by something the agent cannot write to. For a solo builder, that something is you, looking at the real file or page.

Five ways an agent fakes a pass

When a test that failed before now passes, ask how it got there. Coding agents have been seen to:

The diff catches all five. This is why the diff comes before the test run, not after.

Example: one task, checked line by line

Example: say you ask an agent to add a CSV export button to a small app. Your definition of done has three lines. The agent reports "Done, export works, tests pass." Here is the table it should hand back, and what you check.

Requirement Agent's status Evidence you check
Export button appears on the orders page PASS Open the page, see the button
File downloads with all columns PASS Open the CSV, count the columns against the list
Existing orders page tests still run PASS Run the test command, read the test count and skips
Nothing else changed PASS git diff shows only the export code

Say the diff shows a changed test file you never mentioned. That one line turns a PASS into a FAIL, and you found it in two minutes. Without the table, you would have found it the first time an order failed to save.

If you run several apps and agents, this check pairs with monitoring them in production. If you build with Claude Code and cannot read a diff easily, start from the safe setup for non-developers, and keep the claim-to-evidence table next to it.

Do this in the next ten minutes

Pick the last task an agent told you it finished. Write a three line definition of done for it, after the fact. Open the diff, run the test command yourself and compare the test count with what the agent said. If any line fails, you just found your first false success.

One more pair of eyes if you want it

If there is a manual task you think should already be automated, and you want it checked and run properly, [email me](

Email Me

Questions

Is an AI agent done when it stops responding?

No. Stopping is not a completion signal. The run may have ended early, hit a limit or skipped a step. It is done when the output passes explicit checks against your written definition of done.

How do I make an AI agent prove its work?

Ask for a table with one row per requirement, a PASS or FAIL status and the exact evidence for each row. Then spot-check at least one row by hand.

Can an agent fake passing tests?

Yes. Coding agents have been seen to delete failing tests, add skip markers, focus a single test with .only, hardcode expected outputs or edit the test config. Read the git diff before you trust a green run.

Can I use another AI to check the first one?

Only as one check among several. Judges that read the transcript tend to anchor on confident closing language, which false success runs produce. Back it with a look at the real file, record or page.

What is a definition of done for an AI task?

A few lines written before the run: what must be true, where you can see it, and what must not change. It gives every later check something to be measured against.