Writing

How to Monitor AI Agents in Production When You Run Several Apps Alone

By Scott Wilder · October 21, 2026 · 7 min read · All writing

Five alerts and one two-minute daily check will cover most of what goes wrong when you run several AI agents alone: the agent that stopped, the one stuck in a loop burning money, and the one that quietly got worse. You can set all of it up in an evening, and you can keep the daily check under two minutes.

The usual advice is written for teams with an on-call rotation. You have a day job, a few apps and no one to hand a pager to. So the list below is short on purpose.

Why a normal uptime check will not catch an agent failing

An agent can be up and still be wrong. Latitude's monitoring guide says agents can hallucinate subtly, get stuck in soft loops, make expensive tool calls, or degrade in quality without ever returning an error. An uptime ping sees a healthy server. It does not see an agent that has made 400 tool calls on one task.

OneUptime's write-up has the right title for the cost of having nothing in place: Your AI Agent Crashed at 3am and Nobody Noticed. When you run several apps alone, that is the normal case, not the rare one.

So you need to cover three different failures, and each needs its own tool:

Failure What it looks like What catches it
The agent stopped A scheduled job never ran A check-in ping that goes red when it is late
The agent ran away Calls and spend climb, the task never finishes A spend baseline, a per-run counter and a hard cap
The agent got worse Tool errors, more steps per run, bad output A few agent metrics and your daily look

Start with a check-in ping on every scheduled agent job

This is the cheapest fix and it catches the most embarrassing failure. Each scheduled job pings a unique URL when it finishes. If the ping does not arrive on time, you get an alert. This is the dead man's switch pattern, and Healthchecks.io works this way: it listens for HTTP requests and emails from cron jobs and scheduled tasks and alerts when one does not arrive on time.

The free Hobbyist plan is listed at 20 jobs with 100 log entries per job. Check the pricing page before you count on that, since plans change.

Set it up like this:

  1. List every scheduled agent job across all your apps.
  2. Create one check per job. Do not share one check across apps, or you will not know which one died.
  3. Set the period to how often the job should run. Set the grace time to how late is still fine.
  4. Add one HTTP request at the very end of the job, after it succeeds. Not at the start. A job that starts and then hangs would look healthy if you ping on start.
  5. Send alerts to the place you actually read.

Tight grace time gives you false alarms, and you will start ignoring them. Loose grace time means you find out late. Start loose and tighten it after a week.

This part I'm describing from how the service documents itself. Treat the agent-specific use as untested until you have run it for a week.

Compare each app to its own spend, never to a shared number

The expensive silent failure is a loop that keeps calling the model. A spend baseline approach fixes it: learn what normal looks like, then alert the same day when something passes its own normal.

One shared threshold fails because your quiet app can run away without ever crossing a number set for your busy one.

Example: say you run three apps and, after a week, their normal days cost $1.20, $0.40 and $3.00 in model calls. If you pick 2x as your alert line, that is $2.40, $0.80 and $6.00. The $0.40 app doubling to $0.80 is a loop you want to hear about the same day, and a single $5 line would have missed it completely.

The 2x is a placeholder. Nothing I found gives a correct multiplier. Run one week, look at your own worst normal day, and set the line a bit above it.

Put a hard cap on every key so a loop cannot reach the invoice

An alert tells you. A cap stops the calls. You want both, because you will sometimes be asleep or at your day job.

Airia's guide to stopping a runaway agent says catching a loop means watching each active run: how many calls it has made, how much it has spent, and whether those numbers are heading toward a finished task or just climbing. Developer Toolkit's cost monitoring page covers alerts, hard caps and cheap-model routing together.

Two small tools exist for the cap. Agent-Guard is an open-source Node.js library that wraps OpenAI and Anthropic SDK calls, blocks calls once the budget is hit and keeps a local audit log with no telemetry. I have not tested it, so read its page before you trust it with a production key.

Pick the cap as the most you would be willing to lose on a bad day for that app, and set it where the agent cannot change it, as laid out in spending limits on AI agents. Then log calls and dollars per run, so a loop is visible while it is happening, not on next month's bill.

The five-alert list and the two-minute daily check

This is the part to copy. Every alert needs a threshold, a destination, an owner and a first action, which is the structure Latitude's alert playbook recommends. For you the owner is always you. The first action is the part that saves you at 11pm, so write it down now.

# Alert Fires when (starting placeholder) Goes to First action
1 Job did not check in Ping is late past grace time Phone or email you read Open the job's logs, rerun it once
2 Spend over baseline App's spend today passes 2x its own normal Same Check the last 3 runs for a loop
3 One run keeps climbing A single run passes 3x the call count of your longest normal run without finishing Same Kill the run, lower the cap
4 Tool keeps failing One tool fails more than 1 in 5 calls for an hour Same Check that tool's key, quota and API status
5 Guardrail tripped Any run hits a safety flag or hands off to a human Same Read that run before the next one starts

All numbers in the third column are placeholders. Replace them after one week of your own data.

Alert 5 matters more if your agents read outside text. See Prompt Injection: What Every Builder Running AI Agents Should Know.

Latitude also suggests severity tiers you can shrink to one line each. P0 is a critical workflow broken: act now. P1 is sustained quality loss on a core path: act today. P2 is a low-impact oddity: put it in a backlog and look weekly. For this list, treat alerts 1 and 3 as P0, alerts 2 and 4 as P1 and alert 5 as whatever you decide after reading the run.

The two-minute daily check

Run it once a day, same time. A daily cadence is suggested for medium traffic of about 1,000 to 10,000 requests a day. Your apps are probably well under that, so daily is more than enough.

The last box is the one alerts cannot do for you. A wrong answer produces no error. Reading one run per app per day is how you catch an agent that got worse without any alarm.

When something is red, do not fix everything at once. Use the weekly rule in Build, Fix or Ignore to decide which fire gets your night.

Mistakes that cost you the most

If you are juggling many agents and losing the thread, How to Manage Multiple AI Agents Without Losing Track covers the organizing side.

Do this in the next ten minutes

Pick your busiest scheduled agent job. Create one check for it in Healthchecks.io, set the period and a loose grace time, and add the ping at the end of the job. That one step covers failure number one, and you can build the rest of the list around it this week.

Want a second pair of eyes on your setup?

If there is a manual task you want automated, or an agent setup you want looked at, send me the details. I read every email myself.

Email Me

Questions

How do I know my AI agent is failing if it does not throw an error?

Watch behavior, not just uptime. Track steps per run, tool failure rate and spend against that app's own normal, and read one real run output each day. An agent that is wrong or looping can look perfectly healthy to an uptime check.

How often should I check AI agents in production?

Once a day is enough for small apps. A daily cadence is suggested for medium traffic of roughly 1,000 to 10,000 requests a day, and most solo apps sit below that. Let alerts cover the time between checks.

How do I stop an AI agent from running up a big bill?

Use two layers. Set a same-day alert when an app passes its own spend baseline, and set a hard cap so calls stop at your budget. Log calls and dollars per run so you can see a loop while it is happening.

Is there a free way to get alerted when a scheduled agent stops running?

Yes. A check-in service such as Healthchecks.io expects a ping when your job finishes and alerts you if it is late. Its free Hobbyist plan is listed at 20 jobs, but confirm the current limits on the pricing page.

What should an alert for an AI agent say?

Each alert should name the signal, the threshold, where it goes and the first action to take. As a solo founder you are the owner, so the written first action is what matters, such as which logs to open or which key to check.