Five alerts and one two-minute daily check will cover most of what goes wrong when you run several AI agents alone: the agent that stopped, the one stuck in a loop burning money, and the one that quietly got worse. You can set all of it up in an evening, and you can keep the daily check under two minutes.
The usual advice is written for teams with an on-call rotation. You have a day job, a few apps and no one to hand a pager to. So the list below is short on purpose.
Why a normal uptime check will not catch an agent failing
An agent can be up and still be wrong. Latitude's monitoring guide says agents can hallucinate subtly, get stuck in soft loops, make expensive tool calls, or degrade in quality without ever returning an error. An uptime ping sees a healthy server. It does not see an agent that has made 400 tool calls on one task.
OneUptime's write-up has the right title for the cost of having nothing in place: Your AI Agent Crashed at 3am and Nobody Noticed. When you run several apps alone, that is the normal case, not the rare one.
So you need to cover three different failures, and each needs its own tool:
| Failure | What it looks like | What catches it |
|---|---|---|
| The agent stopped | A scheduled job never ran | A check-in ping that goes red when it is late |
| The agent ran away | Calls and spend climb, the task never finishes | A spend baseline, a per-run counter and a hard cap |
| The agent got worse | Tool errors, more steps per run, bad output | A few agent metrics and your daily look |
Start with a check-in ping on every scheduled agent job
This is the cheapest fix and it catches the most embarrassing failure. Each scheduled job pings a unique URL when it finishes. If the ping does not arrive on time, you get an alert. This is the dead man's switch pattern, and Healthchecks.io works this way: it listens for HTTP requests and emails from cron jobs and scheduled tasks and alerts when one does not arrive on time.
The free Hobbyist plan is listed at 20 jobs with 100 log entries per job. Check the pricing page before you count on that, since plans change.
Set it up like this:
- List every scheduled agent job across all your apps.
- Create one check per job. Do not share one check across apps, or you will not know which one died.
- Set the period to how often the job should run. Set the grace time to how late is still fine.
- Add one HTTP request at the very end of the job, after it succeeds. Not at the start. A job that starts and then hangs would look healthy if you ping on start.
- Send alerts to the place you actually read.
Tight grace time gives you false alarms, and you will start ignoring them. Loose grace time means you find out late. Start loose and tighten it after a week.
This part I'm describing from how the service documents itself. Treat the agent-specific use as untested until you have run it for a week.
Compare each app to its own spend, never to a shared number
The expensive silent failure is a loop that keeps calling the model. A spend baseline approach fixes it: learn what normal looks like, then alert the same day when something passes its own normal.
One shared threshold fails because your quiet app can run away without ever crossing a number set for your busy one.
Example: say you run three apps and, after a week, their normal days cost $1.20, $0.40 and $3.00 in model calls. If you pick 2x as your alert line, that is $2.40, $0.80 and $6.00. The $0.40 app doubling to $0.80 is a loop you want to hear about the same day, and a single $5 line would have missed it completely.
The 2x is a placeholder. Nothing I found gives a correct multiplier. Run one week, look at your own worst normal day, and set the line a bit above it.
Put a hard cap on every key so a loop cannot reach the invoice
An alert tells you. A cap stops the calls. You want both, because you will sometimes be asleep or at your day job.
Airia's guide to stopping a runaway agent says catching a loop means watching each active run: how many calls it has made, how much it has spent, and whether those numbers are heading toward a finished task or just climbing. Developer Toolkit's cost monitoring page covers alerts, hard caps and cheap-model routing together.
Two small tools exist for the cap. Agent-Guard is an open-source Node.js library that wraps OpenAI and Anthropic SDK calls, blocks calls once the budget is hit and keeps a local audit log with no telemetry. I have not tested it, so read its page before you trust it with a production key.
Pick the cap as the most you would be willing to lose on a bad day for that app, and set it where the agent cannot change it, as laid out in spending limits on AI agents. Then log calls and dollars per run, so a loop is visible while it is happening, not on next month's bill.
The five-alert list and the two-minute daily check
This is the part to copy. Every alert needs a threshold, a destination, an owner and a first action, which is the structure Latitude's alert playbook recommends. For you the owner is always you. The first action is the part that saves you at 11pm, so write it down now.
| # | Alert | Fires when (starting placeholder) | Goes to | First action |
|---|---|---|---|---|
| 1 | Job did not check in | Ping is late past grace time | Phone or email you read | Open the job's logs, rerun it once |
| 2 | Spend over baseline | App's spend today passes 2x its own normal | Same | Check the last 3 runs for a loop |
| 3 | One run keeps climbing | A single run passes 3x the call count of your longest normal run without finishing | Same | Kill the run, lower the cap |
| 4 | Tool keeps failing | One tool fails more than 1 in 5 calls for an hour | Same | Check that tool's key, quota and API status |
| 5 | Guardrail tripped | Any run hits a safety flag or hands off to a human | Same | Read that run before the next one starts |
All numbers in the third column are placeholders. Replace them after one week of your own data.
Alert 5 matters more if your agents read outside text. See Prompt Injection: What Every Builder Running AI Agents Should Know.
Latitude also suggests severity tiers you can shrink to one line each. P0 is a critical workflow broken: act now. P1 is sustained quality loss on a core path: act today. P2 is a low-impact oddity: put it in a backlog and look weekly. For this list, treat alerts 1 and 3 as P0, alerts 2 and 4 as P1 and alert 5 as whatever you decide after reading the run.
The two-minute daily check
Run it once a day, same time. A daily cadence is suggested for medium traffic of about 1,000 to 10,000 requests a day. Your apps are probably well under that, so daily is more than enough.
- Any check-in red?
- Did any app pass its own spend baseline or hit its cap?
- Any single run with an odd number of steps or calls?
- Any tool with a pile of failures?
- Open one random run per app and read the output. Does it look right?
The last box is the one alerts cannot do for you. A wrong answer produces no error. Reading one run per app per day is how you catch an agent that got worse without any alarm.
When something is red, do not fix everything at once. Use the weekly rule in Build, Fix or Ignore to decide which fire gets your night.
Mistakes that cost you the most
- Watching only the monthly invoice. You find the loop after the money is gone.
- One threshold for every app. Your quiet app runs away unnoticed.
- An alert with no first action. At 11pm you will stare at it and do nothing.
- Noisy alerts. If most alerts are false alarms, you will ignore the one that is real. Tighten the line after any false alarm.
- Pinging at the start of a job. A hung job still looks healthy.
- Skipping the fix check. After you fix something, confirm the next run is clean, and note what kind of failure it was so you can see repeats.
If you are juggling many agents and losing the thread, How to Manage Multiple AI Agents Without Losing Track covers the organizing side.
Do this in the next ten minutes
Pick your busiest scheduled agent job. Create one check for it in Healthchecks.io, set the period and a loose grace time, and add the ping at the end of the job. That one step covers failure number one, and you can build the rest of the list around it this week.
Want a second pair of eyes on your setup?
If there is a manual task you want automated, or an agent setup you want looked at, send me the details. I read every email myself.
