Automations do not fail randomly. They fail at 3am because 3am is when the batch jobs run, the rate limits reset, the certificates expire, and there is nobody watching a screen. Every one of those conditions is absent from the environment you tested in.

I have debugged a lot of workflows built by capable people that worked perfectly for months and then quietly stopped. The failures are remarkably consistent, and none of them are exotic. Here are the six that account for most of what I see, and what to do about each.

1. The rate limit you never hit in testing

You tested with three records. Production processed eleven hundred, hit a 429, and moved on. Many HTTP nodes treat a non-2xx response as a branch to handle rather than an error to retry, so the run completes successfully with most of the work missing.

The fix: exponential backoff with jitter on every external call, a concurrency cap on batch loops, and — this is the part people skip — a hard failure if retries are exhausted. A workflow that silently completes with partial results is worse than one that crashes, because it never gets investigated.

2. Partial writes with no idempotency

A workflow that writes to three systems and dies after the second has left your data in a state no code path anticipates. Re-running it then creates duplicates in the first two systems.

The fix: an idempotency key derived from the input — a record ID, a hash of the payload, an external reference — passed to every downstream write. Make re-running a workflow the safe, boring default. If you can safely re-run any failed execution without thinking about it, most of your 3am incidents become a morning click.

3. Silent retries that duplicate work

The mirror image of the last one. Platform-level retries are useful and dangerous in the same breath: a step that timed out after the downstream system had already committed will be retried, and now the invoice is sent twice. Timeouts are not failures, they are unknowns.

The fix: the same idempotency keys, plus generous timeouts on operations that are expensive to repeat. Where a system offers a deduplication window or a client-supplied request ID, use it.

4. Schema drift

Someone renamed a field in the CRM. Nobody told you, because from their side it was a one-line change in a settings screen. Your workflow now writes undefined into a field that accepts anything, and does it without complaint for six weeks.

The fix: validate the shape of external data on entry, not deep in the flow. A small schema check that fails loudly beats a large workflow that fails quietly. Treat every external payload as untrusted, both for safety and for your own sanity.

5. Credentials that expire on a schedule nobody wrote down

OAuth tokens expire. API keys get rotated by a security policy you were not copied on. Certificates lapse. These are the most predictable outages in software and they still take people by surprise, because the expiry date lives in a system that does not talk to the system that will break.

The fix: alert on authentication failures as a distinct class, separate from general errors, and put renewal dates in the same calendar as everything else you plan around.

6. Nobody is watching

The most common failure of all. The workflow stopped running three weeks ago. It is not throwing errors, because it is not doing anything. Error alerting cannot detect absence — only presence.

The fix: a heartbeat. Every scheduled workflow should report a successful completion to something that will complain when the report stops arriving. Dead-man’s-switch monitoring is a ten-minute addition and it catches an entire category of failure that error handling structurally cannot.

Error alerting tells you when something went wrong. Only a heartbeat tells you when nothing went at all.

The pre-flight checklist

Before any workflow goes into production, I want to be able to answer yes to all of these:

  • Can I safely re-run any failed execution without creating duplicates?
  • Does every external call retry with backoff, and fail loudly when retries run out?
  • Is there a dead-letter path for records that cannot be processed, and does a human ever see it?
  • Does a failure reach a person through a channel they actually read?
  • Is there a heartbeat that fires if the workflow stops running entirely?
  • Is the incoming data validated on entry, before any of it is used?
  • Can somebody other than me understand this flow from the runbook alone?

Nothing on that list is difficult. Collectively it is the difference between an automation that saves your team time and an automation that costs them a morning every few weeks, plus the slow erosion of trust that follows.

Adding an LLM does not change any of this

If anything, a model in the middle of a workflow raises the stakes: you have added a step that can fail in a new way, by returning something syntactically valid and semantically wrong. Validate structured output against a schema before it is used. Set a budget ceiling per run. And keep a deterministic fallback path for the cases where the model is unavailable, because at some point it will be, and 3am is exactly when you will find out.