What Happens When Agents Break

Armada Works runs eight agents on a single codebase. They commit to main, post daily briefs to a dashboard, and coordinate through state files that live in the repo. Most days, the system works exactly the way it should. This post is about the days it does not.

Not the catastrophic failures people imagine when they hear "autonomous agents." No deleted databases, no rogue deployments, no embarrassing emails sent to prospects. The real failures are quieter, stranger, and more instructive than any of those scenarios. They teach you things about your system that you would never learn from a test suite or a thought experiment.

Here is what actually happens when an agent in a production fleet breaks, drawn from incidents we have logged, diagnosed, and built defenses against over the past four months.

How failures surface

The first thing to understand is that agent failures rarely announce themselves. There is no red alert, no pager going off at 3 a.m. The signal is almost always an absence: a brief that should have appeared on the dashboard and did not. A state file that was not updated. A commit that exists locally but never reached origin.

The CMO agent, which runs each morning and reads every sub-agent's daily brief, is the primary detection layer. When a sub-agent's brief is missing, the CMO flags it in its own synthesis. That is usually how I find out something went wrong. Not a crash report. Not an error log. A gap in the expected output.

This detection model works because the fleet's coordination is pull-based, not push-based. Each agent writes its own state. The CMO reads all of them. If one agent is silent, the CMO notices the silence. The system does not require every agent to phone home to a central coordinator; it requires the coordinator to notice when the room is quieter than it should be.

The taxonomy of agent failures

After logging every incident for four months, the failures fall into four categories. Each one taught us something different about how to build a resilient fleet.

Silent completions

The most disorienting failure mode: the agent did the work, committed it locally, and then died before pushing to origin or posting its brief.

We hit this when a stream idle timeout killed agent runs mid-execution. The agent had already drafted content, updated state files, and run a local commit. But the connection between the agent and the API dropped during a long planning pause (the model goes quiet while thinking through a large context, and the client-side timeout interprets the silence as a dead connection). Everything after that, the push, the deploy, the dashboard POST, never ran.

The result: the dashboard showed nothing. From the CMO's perspective, the agent simply did not run that day. But the work was on disk, one git log away from being visible.

This happened across multiple agents over two weeks before the pattern became clear. Each individual brief looked correct in isolation. It took noticing that three different agents had "no brief today" on different days, combined with finding their unpushed commits locally, to see the systemic cause.

The fix had two layers. First, raise the stream idle timeout from the default to 30 minutes, which reduces how often the connection drops. Second, and this is the structural fix, reorder the finish sequence so the dashboard POST happens immediately after the local commit, before push and deploy. Now if the connection dies after the commit, the dashboard still knows the run happened. The brief becomes the canonical "did this agent finish?" signal, not the push.

Misdiagnosed auth failures

External service authentication is the single most fragile surface in an agent fleet. Not because auth is hard in the abstract, but because the failure mode is subtle: the agent correctly identifies that auth is broken, suggests a plausible fix, and the fix works for exactly one run.

Our SEO agent's connection to Google Search Console failed on fifteen consecutive scheduled runs. Each time, the agent surfaced the error and recommended running gcloud auth application-default login to refresh the credentials. The founder ran the command. The next day's run worked. The run after that failed again.

Fifteen days of ranking data lost, and neither the agent nor I caught the pattern across runs because each individual brief looked like a one-time auth hiccup. The actual cause was a Google Workspace policy that enforces interactive reauthentication every sixteen hours or so. The agent's prompt described the failure as "refresh token expired," which was the wrong diagnosis for this account type. The prescribed fix resolved the symptom temporarily but could never stick because the underlying auth model was fundamentally incompatible with unattended access.

The fix was a service account with domain-wide delegation, which bypasses the interactive reauth requirement entirely. But the deeper lesson was procedural: we added a rule that when an agent flags the same blocker on three consecutive runs without the underlying problem resolving, it should escalate the pattern as a configuration question, not just re-flag the symptom. That rule would have caught this by run four instead of run fifteen.

Missing constraints

Sometimes an agent does exactly what you would expect a reasonable agent to do, and the result is still wrong. Not because the agent made a bad decision, but because the prompt did not include a constraint that only matters in one specific situation.

When our SEO agent hit an auth failure and needed to configure a Google Cloud project for its queries, it ran the equivalent of "list all available projects" and got back exactly one result: a project belonging to a sibling business. With nothing in the prompt saying "do not use that project," the agent treated the only available option as the correct option and started wiring Armada's data pipeline through another company's infrastructure.

I caught it because the project ID in the recovery commands did not match what I expected. The fix was straightforward: pin the expected account, project, and scopes in a configuration table inside the prompt, with an explicit "never use these other accounts" rule. But the incident revealed a class of failure that static analysis or testing would never catch. The agent's reasoning was valid. Its context was incomplete.

This is the kind of failure that makes you rethink how you write agent governance rules. Abstract principles ("use the correct project") are useless when the agent's observable universe contains only one project. Specific constraints ("use project X, never project Y, here is why") are what survive contact with reality.

Environmental stalls

The runtime environment itself can change in ways that block an unattended agent. New tools get added to the platform. New permission gates appear. A tool the agent has never used before triggers a "do you want to allow this?" dialog, and the agent sits there waiting for a human who is not watching.

We hit this when a scheduled SEO run attempted to use a monitoring tool that was not on the pre-approved allow list. Claude Code's permission system popped a modal dialog. The agent stalled. The run produced nothing until I happened to notice the dialog on my screen hours later.

The fix was mechanical: expand the allow list to include the tools agents actually reach for. But the pattern is worth naming because it recurs with any system that has a permission model. Every new capability, every new tool, every platform update is a potential stall point for unattended agents. You need a process for discovering and pre-approving tools before they block a production run.

This is one reason why an agent permissions audit matters even after initial setup. The permission surface is not static. It grows as the platform evolves and as agents discover new tools.

Blast radius

The question people ask most often about agent failures is some version of "what if one goes rogue and takes everything down?" The honest answer, at least for a fleet coordinated through git and state files, is that the blast radius of a single agent failure is surprisingly small.

When the SEO agent's auth broke for fifteen days, the content agent kept shipping blog posts. The outbound agent kept drafting emails. The CMO agent still synthesized briefs from the agents that did report. The fleet degraded gracefully because there is no central runtime connecting the agents. Each one reads the repo, does its work, writes its output, and exits. A broken agent is a missing file, not a crashed server.

This is not an accident. It is a direct consequence of the coordination architecture: state files in git, not a shared database or message queue. When one agent fails to write its state file, the CMO notes the absence. Nothing cascades. Nothing blocks. The system loses that agent's contribution for the day, and everything else proceeds.

Compare this to a monolithic automation platform where a broken auth token or a crashed service takes down the entire pipeline. The fleet model trades some coordination elegance for failure isolation. Every agent is independently deployable, independently debuggable, and independently recoverable. That tradeoff has paid for itself every time something breaks.

Scar tissue over speculation

We keep a running log of every incident: what happened, how we detected it, the root cause, and the defensive rule that came out of it. We call it scar tissue, borrowing the idea that the rules worth keeping are the ones that came from a specific failure, not from a thought experiment.

The stream idle timeout fix came from watching three agents silently fail over two weeks. The service account migration came from fifteen days of misdiagnosed auth. The project-pinning rule came from an agent that did exactly what it was told to do, with the wrong data. None of these rules would have appeared in a speculative security review or a pre-launch checklist.

This is, in a sense, the entire operating philosophy behind a production runbook for an agent fleet. The runbook is not a static document you write once and file away. It is a living artifact that accumulates from contact with reality, and the most important entries are the ones you did not anticipate.

The security guardrails follow the same pattern. The guardrails that matter are not the ones you install on day one. They are the ones you add on day forty-seven after an agent does something technically correct but contextually wrong, and you realize your constraints were not specific enough.

What this means for your fleet

If you are considering running agents in production, expect them to break. Not in dramatic ways. In quiet, instructive ways that teach you things about your system you could not have learned otherwise.

The question is not whether failures will happen. It is whether your architecture makes failures local (one agent, one day, one missing brief) or systemic (one failure cascades into a fleet-wide outage). Git-based coordination, state files the agents own, and a synthesizer agent that reads everything each morning: that is the structure that has made our failures recoverable instead of catastrophic.

The other thing worth knowing: the incidents get less frequent. Not because the system becomes perfect, but because each incident adds a specific, tested defense. Four months in, the fleet runs cleaner than it did in month one, and the runbook is thicker. Both of those things are supposed to happen.

If you want to see how this works in practice, here is how we engage. The Pilot includes standing up the fleet, the runbook, and the scar-tissue log. When we leave, you keep all of it.