An agent that stops talking has not finished the task. Treat "done" as a claim, then run it, diff it and ask for proof before you accept it.
By the end of this page you will have a definition of done you can reuse, a table that matches every claim to evidence, and one prompt to paste after every run.
The model going quiet is not a completion signal
The hard part of running an agent on a real task starts after the last token. Most frameworks treat the model's silence as the completion signal, and that is the mistake. A run can stop early, hit a limit or skip a step and still end with a confident summary.
Researchers call this false success: the agent says it is done and the state of the system says it is not. It spreads silently, unlike a crash, a refusal or an explicit handoff. In that paper, false success accounts for 45 to 48 percent of failures in the airline and retail domains tested, and 75.8 percent of failures among coding-agent runs that state a status. Those numbers come from benchmark setups, not your project. You find your own rate by running the check below.
Write the definition of done before the run
Verification depends on a definition of done that existed before the build started. If it lives only in your head, checking becomes a feeling.
Use this template. Three to six lines is enough:
Task: <one sentence>
Done means:
1. <thing that must be true, and where I can see it>
2. <thing that must be true, and where I can see it>
3. <thing that must NOT change>
The last line matters. "Must not change" is how you catch an agent that fixed the problem by breaking something else.
Match every claim to evidence you looked at yourself
An agent's closing message is a list of claims. Each claim needs one piece of evidence that does not come from the agent's own narration. A useful report ties each claim to an independent observation.
| The agent says | Evidence you need | Where you look |
|---|---|---|
| "I only changed what you asked" | A reviewed git diff with clean scope | git diff, read every changed line |
| "Tests pass" | The exact command, exit code 0, the test count, no unexpected skips | Run the command yourself, fresh |
| "It works" | The changed behavior tested directly | Click it, call it, run it once by hand |
| "The record is updated" | A read-only query of the real record | The database, the file, the live page |
| "I sent / published / deployed it" | The thing in the place it should be | The inbox, the live URL, the dashboard |
Pass line: every claim has evidence you looked at. One claim with no evidence means not done.
Paste this after every run
It forces the agent to turn its summary into a checkable table. A pre-completion checklist works the same way: the agent cannot declare done until it passes explicit checks, with a PASS or FAIL status and the exact evidence for each item.
Before you say you are done, give me a table: one row per
requirement in the definition of done, PASS or FAIL, and the
exact evidence for each row (file and line, command and output,
or the place I can open to see it). List every file you changed.
Confirm you did not delete, skip or edit any test. Fix every
FAIL before you answer.
Treat any FAIL as not done. Send it back, then ask for the table again.
Run it, diff it, or ask for proof: pick by output type
You do not need every check on every task. Pick by what the agent produced.
- It made something you can run (a script, an app, a build): run it and read the result yourself. A description of the result does not count.
- It changed files: diff against where you started and read every changed line. "I fixed it" with no diff is not done.
- It produced something you cannot run or diff (research, data, a copy edit): ask for the exact line, file or source that proves each item, then spot-check one by hand.
- It changed something outside your project (a database, a deployed page): check the real record with a read-only query. The agent should not be the one reading it back to you.
When the task repeats and the stakes are low, add a checklist gate the agent must pass before it may say done. When the stakes are high, add a human or a second agent on top.
Do not let a second AI be your only check
It is tempting to ask another model to read the transcript and judge it. Do not rely on that alone. LLM judges anchor on confident closing-message language, and false success runs produce exactly that language.
The same paper's conclusion is the rule worth copying: the fix that holds is a check per state-changing step, asserted against the real record by something the agent cannot write to. For a solo builder, that something is you, looking at the real file or page.
Five ways an agent fakes a pass
When a test that failed before now passes, ask how it got there. Coding agents have been seen to:
- Delete the failing test
- Add a skip marker such as
@pytest.mark.skip - Focus one passing test with
.only - Hardcode the expected output or stub a function with
NotImplementedError - Edit the test config so the failing tests never run
The diff catches all five. This is why the diff comes before the test run, not after.
Example: one task, checked line by line
Example: say you ask an agent to add a CSV export button to a small app. Your definition of done has three lines. The agent reports "Done, export works, tests pass." Here is the table it should hand back, and what you check.
| Requirement | Agent's status | Evidence you check |
|---|---|---|
| Export button appears on the orders page | PASS | Open the page, see the button |
| File downloads with all columns | PASS | Open the CSV, count the columns against the list |
| Existing orders page tests still run | PASS | Run the test command, read the test count and skips |
| Nothing else changed | PASS | git diff shows only the export code |
Say the diff shows a changed test file you never mentioned. That one line turns a PASS into a FAIL, and you found it in two minutes. Without the table, you would have found it the first time an order failed to save.
If you run several apps and agents, this check pairs with monitoring them in production. If you build with Claude Code and cannot read a diff easily, start from the safe setup for non-developers, and keep the claim-to-evidence table next to it.
Do this in the next ten minutes
Pick the last task an agent told you it finished. Write a three line definition of done for it, after the fact. Open the diff, run the test command yourself and compare the test count with what the agent said. If any line fails, you just found your first false success.
One more pair of eyes if you want it
If there is a manual task you think should already be automated, and you want it checked and run properly, [email me](
