What a Production Runbook for an Agent Fleet Looks Like

An AI agent runbook is a structured reference document that tells the operator what each agent does, what credentials it holds, what to check when something looks wrong, and how to shut it down without breaking the rest of the fleet. Armada Works writes one for every agent fleet we deploy, and after 160 days of operating an eight-agent fleet against a production codebase, the runbook is the single most referenced document after the handoff.

Most teams that run AI agents in production skip the runbook entirely. They rely on the prompt file to explain what the agent should do, the state file to show what it did, and their own memory to fill the gap. That works for the first month. It stops working the moment a credential expires on a Friday evening and nobody remembers which token the agent uses, where to rotate it, or what breaks downstream.

This post covers the six sections every production runbook needs, with operational patterns drawn from running a fleet daily since April 2026.

The Six Sections Every Agent Runbook Covers

A production runbook for an agent fleet is not a single massive document. It is one page per agent, each following the same six-section structure. The consistency matters: when something breaks at 7 AM and you are scanning for the relevant section, you should not have to hunt for it.

1. Monitoring. What the agent produces each run, where it writes its output, and what a healthy run looks like. For a Content agent, a healthy run means one new markdown file committed under docs/content/blog/ and a brief posted to the dashboard. For an SEO agent, a healthy run means a state file update with current ranking data and no credential errors. This section should include an auth-health pre-flight: a check that each credential the agent depends on is still valid before the real work starts. More on that below.

2. Permissions. Every credential the agent can reach, classified by blast radius. Account-level tokens (ones that can delete the entire project) should not exist in the agent's environment. Project-level tokens get listed with their renewal cadence and the exact steps to rotate them. This section also documents which PreToolUse hooks are configured and what they block.

3. Rotation. Token expiry dates and the steps to renew each credential before it lapses. API keys, OAuth tokens, and service-role credentials all expire on different schedules. The runbook lists each one with its expected lifetime and a renewal procedure short enough to complete in under five minutes.

4. State-file maintenance. How to compact the agent's state file when it grows past a usable size, and the schedule for doing so. This section did not exist in our original five-section template. After five months of daily operation, it earned its place. Details in the dedicated section below.

5. Escalation. What to do when the agent fails in a way you cannot fix by editing a prompt or rotating a credential. This section defines two paths: self-service fixes (restart the scheduled task, re-run with updated context) and escalation to whoever built the fleet. For Armada engagements, the escalation path during the first 90 days is the optional $1,500/month support tier.

6. Decommissioning. How to pause or permanently remove the agent without breaking the rest of the fleet. Agents coordinate through shared state files. Removing one without updating the synthesizer's read list means the CMO agent will keep looking for a brief that never arrives and will flag a "missing report" anomaly every run. For a full picture of how these coordination dependencies work, see how the fleet actually coordinates.

Auth-Health Monitoring: The Pre-Flight Check Your Runbook Needs

The most insidious failure mode in an agent fleet is not a crash. It is an agent that runs on schedule, produces output, and reports a credential error that nobody fixes. The agent does not stop. It keeps running, keeps filing briefs that say "auth broken," and keeps consuming compute cycles without producing any real work.

Armada's Social Media agent hit this pattern with its Buffer API integration. The OAuth token stopped authorizing around Day 89 of fleet operation. The agent ran every scheduled session after that. Each session's brief noted the auth failure, proposed the fix (re-authorize the Buffer connection), and moved on to the parts of its job it could still do. The CMO agent read each brief, flagged the auth issue in its daily synthesis, and escalated it to the founder. As of this writing, the token has been broken for over 30 consecutive runs.

Nothing caught fire. No data was lost. But 30 scheduled runs produced zero social media output, and the fleet spent compute and context window on an agent that could not do its primary job. That is the cost of a missing auth-health gate.

The fix is a pre-flight check at the top of each agent's monitoring section in the runbook:

AUTH-HEALTH PRE-FLIGHT
  Before each run, verify:
    BUFFER_ACCESS_TOKEN   test: GET /channels (expect 200, not 401/403)
                          if fail: skip social posting, flag in brief,
                                   escalate after 3 consecutive failures
    AGENT_REPORT_KEY      test: POST /api/agent-reports with dry-run flag
                          if fail: agent cannot report; escalate immediately

The pre-flight does not fix the credential. It stops the agent from burning a full run against a broken dependency and makes the failure louder. Three consecutive pre-flight failures should trigger a different escalation path than the standard "flag it in the brief" pattern, because three consecutive failures mean the standard path is not producing a fix.

This is also why the permissions audit is not a one-time setup task. Credentials degrade. OAuth tokens expire. Service policies change. The runbook's monitoring section needs to reflect the actual current state of each dependency, not the state at initial deployment.

State-File Maintenance and Compaction

After 160 days of daily operation, the Armada fleet's state files taught us something we did not anticipate at launch: state files grow without bound, and unbounded growth has real operational consequences.

Each agent appends to its state file every run. A "Recently Shipped" section accumulates entries. A "Queue Snapshot" or "Prospect Tracker" grows as new items arrive. Historical context ("Things To Remember Next Session") accretes useful but never-pruned detail. After five months of daily runs, some state files grew past 1,800 lines. An X Prospecting agent's state file reached 1,100 lines.

The consequences are not hypothetical. Each scheduled-task run starts a fresh session. The agent reads its state file to understand where it left off. When that file is 1,800 lines of historical context, a significant share of the agent's context window is consumed before it does any new work. The agent gets slower, its synthesis gets shallower, and the risk of truncation grows with every run.

The runbook's state-file maintenance section should specify three things:

Compaction schedule. How often to compact, based on the agent's run frequency. Daily agents need compaction every 30 to 60 days. Weekly agents can go 90 days. The trigger is the state file crossing a line-count threshold (we use 500 lines as the warning and 800 as the hard limit).

What to keep. Recent history (last 14 to 21 days of entries), open questions that need resolution, and any "Things To Remember" notes that are still load-bearing. Everything older gets summarized into a one-line entry with a date range: "Sessions 45 through 90: evaluated 300 prospects, 0 replies, Apollo quota exhausted 2026-07-15."

What to archive. The full uncompacted state file goes to a dated archive file (state-archive-2026-09-23.md) in the same directory before compaction. Nothing is deleted. The archive is the audit trail; the active state file is the working context.

A sample runbook entry for state-file maintenance:

STATE-FILE MAINTENANCE
  File:            docs/agents/state/outbound-agent-state.md
  Current size:    ~1,800 lines (OVER threshold)
  Compact when:    >500 lines or every 45 days, whichever comes first
  Keep:            last 14 days of entries + open threads + active prospects
  Archive to:      docs/agents/state/outbound-state-archive-YYYY-MM-DD.md
  Procedure:       1. Copy current state to archive file
                   2. Summarize entries older than 14 days into 1-line digest
                   3. Remove duplicate "Things To Remember" entries
                   4. Verify new state file <500 lines
                   5. Commit archive + compacted state in same commit

This is not glamorous work. It is the kind of maintenance that separates a fleet that runs cleanly at month six from one that accumulates context debt until an agent starts producing garbled output because its state file ate half its context window.

For more on how state files serve as the coordination layer across providers, see multi-agent coordination without vendor lock-in.

A Sample Runbook Page: The Content Agent

Below is what a runbook page looks like for a content agent running on a Mon/Wed/Fri cadence. This is the format Armada Works delivers at the end of a transfer engagement.

Agent:        Content
Cadence:      Mon/Wed/Fri 9:00 AM PT
Prompt file:  docs/agents/content-agent-prompt.md
State file:   docs/agents/state/content-agent-state.md
Brief output: POST /api/agent-reports (agentPrefix: "content")
              + mirror at docs/agents/state/content-brief-YYYY-MM-DD.md

AUTH-HEALTH PRE-FLIGHT
  AGENT_REPORT_KEY    test: POST with dry-run (expect 200)
                      if fail: cannot report; escalate immediately

MONITORING
  Healthy run:  1 new file under docs/content/blog/ OR state-only update
                Brief posted to dashboard before push
                Git commit + push to main
  Drift signal: Brief says "queue empty" for 3+ consecutive runs
                Output word count <800 or >3000 (outside style range)
                Commit touches files outside docs/content/ or docs/agents/state/
                Same blocker flagged 3+ consecutive runs without resolution

PERMISSIONS
  Credentials used:
    AGENT_REPORT_KEY    scope: POST /api/agent-reports only
                        blast radius: row-level (single table insert)
                        rotation: manual, no expiry unless revoked
  Hooks:
    PreToolUse          blocks: rm -rf, git push --force, supabase db push
                        config: .claude/settings.json + ~/.claude/settings.json

ROTATION
  AGENT_REPORT_KEY     no automatic expiry; rotate if compromised
                       steps: generate new key in Supabase dashboard,
                       update .env.local, update Vercel env vars

STATE-FILE MAINTENANCE
  Current size:        ~400 lines (under threshold)
  Compact when:        >500 lines or every 60 days
  Keep:                last 14 days shipped + open questions + voice notes
  Archive to:          docs/agents/state/content-state-archive-YYYY-MM-DD.md

ESCALATION
  Self-service:        re-run scheduled task manually
                       edit prompt file if output drifted
  Escalate to:         founder (or Armada support if active)
  Trigger:             3 consecutive failed runs, credential error,
                       agent writes outside its designated paths,
                       same blocker flagged 3+ runs (escalate as config issue)

DECOMMISSIONING
  1. Remove scheduled task
  2. Remove "content" from CMO agent's read list
  3. Archive prompt file (do not delete; git history is the record)
  4. Final state file entry: "Decommissioned YYYY-MM-DD, reason: [X]"

This page fits on a single screen. That is deliberate. Robert Cowherd, founder of Armada Works, follows a rule for runbooks: if the operator has to scroll to find the section they need, the page is too long. One screen per agent, six sections, no prose.

What Breaks Without a Runbook

The failure modes that hurt most are the ones that accumulate silently. After operating the fleet for over five months, four patterns have recurred in every fleet configuration we have run.

Credential expiration goes unnoticed. The most expensive version of this: our SEO agent's Google Search Console connection failed on fifteen consecutive scheduled runs. Each time, the agent surfaced the error and recommended a fix. The founder ran the fix. The next day it failed again. Fifteen days of ranking data lost because the prompt misdiagnosed the failure as a refresh-token expiry when the actual cause was a Google Workspace policy requiring interactive reauthentication every sixteen hours. A service account with domain-wide delegation fixed the auth model permanently. A runbook entry with the correct token type, its actual expiry behavior, and the right recovery procedure would have reduced fifteen days of stale data to one. For the full incident writeup, see what happens when agents break.

Auth failures persist without escalation. A subtler version of the credential problem: the agent correctly identifies that auth is broken, correctly proposes the fix in every brief, and nobody applies the fix. The Buffer API failure described above is the textbook case. The agent ran 30+ sessions reporting the same auth error. Each brief was technically correct. The escalation path (flag it for the founder) was working as designed. But "working as designed" produced zero social media output for a month. The runbook's escalation section needs a harder trigger: if the same auth failure persists past N runs, the agent should change its behavior (skip the broken dependency, reduce its run frequency, or produce a different kind of output) rather than burning full runs against a wall.

Decommissioning leaves orphaned references. When a team pauses an agent without updating the synthesizer's read list, the CMO agent flags a "missing brief" every run. Over a few weeks, the operator learns to ignore that flag. Once they start ignoring one anomaly, they start ignoring others. The monitoring surface degrades not because it broke, but because it cried wolf. A decommissioning checklist (six steps, under a minute) prevents this entirely.

State files grow past usability. Covered in detail above, but worth naming alongside the other failure modes. The fix is not a code change. It is a maintenance procedure documented in the runbook and executed on a schedule. The Outbound agent's state file reaching 1,800 lines was not a bug. It was the absence of a compaction procedure that should have been in the runbook from day one.

Where the Runbook Lives

The runbook is a markdown file in the repo, committed to main, versioned like everything else. Not a Notion page, not a Google Doc, not a wiki behind a separate login. The same repository that holds the agent prompts and state files holds the runbook. This matters for provider portability: when coordination state lives in the repo, switching providers or adding agents from a different vendor does not require migrating documentation out of a proprietary tool.

Armada delivers it as docs/agents/runbook.md in every engagement. The file is editable with any text editor. When you add a new agent, you copy the six-section template, fill in the specifics, and commit. When you rotate a credential, you update the rotation section in the same commit. When you compact a state file, you note the compaction date in the maintenance section. The runbook stays current because updating it is the same workflow as maintaining the fleet.

The permissions audit covers the credential scoping that feeds the runbook's permissions section. The monitoring post covers the daily brief pattern that feeds the monitoring section. The handoff checklist covers the transfer process that the decommissioning section supports. The agent governance post covers the security framework that informs the permissions and hooks documentation. Together, those posts form the operational baseline for running an agent fleet you actually own.

If you want help building a runbook for your specific stack, book a 30-minute discovery call. Armada Works includes a production runbook in the deliverables of every Pilot and Transfer engagement.

Frequently Asked Questions

What is an AI agent runbook?

An AI agent runbook is a structured reference document that covers monitoring, permissions, credential rotation, state-file maintenance, escalation paths, and decommissioning steps for each agent in a fleet. The prompt tells the agent what to do. The runbook tells the human operator what to do when something goes wrong or needs maintenance.

How often should you update an agent runbook?

Update the runbook whenever you change something it documents: a new credential, a changed cadence, a modified escalation path, a paused or added agent, or a compacted state file. The easiest way to keep it current is to update it in the same git commit as the change it describes. Quarterly reviews catch anything that drifted between updates.

What happens when an agent fails without a runbook?

The operator has to reconstruct the agent's credential set, expected behavior, and recovery steps from memory or by reading the prompt file. For simple failures (a missed run, a brief that looks wrong), this costs ten minutes of investigation. For complex failures (expired credentials, cross-agent coordination issues, misdiagnosed auth models), it can cost days. The 15-day GSC auth failure described above is a real example of what happens when the recovery documentation is wrong.

Can you automate runbook checks?

Partially. Credential expiry dates can be monitored with a scheduled script that checks token validity and alerts before expiration. Auth-health pre-flights can verify that each dependency responds before the agent starts its real work. State-file line counts can be checked automatically and flagged when they cross a threshold. But the highest-value runbook checks (is the agent's output still aligned with the prompt? is the escalation path still correct? has a state file accumulated enough context debt to need compaction?) require a human reading the document against the current state of the fleet.

How is a runbook different from an agent prompt?

The prompt is instructions for the agent. The runbook is instructions for the human operator. The prompt tells the Content agent to draft blog posts from a queue, write briefs, and commit to main. The runbook tells the operator what a healthy run looks like, which credential the agent uses, how to rotate that credential, when to compact its state file, and how to shut it down without breaking the rest of the fleet. Both are necessary. Neither replaces the other.

Why did the runbook expand from five sections to six?

The original five-section template (monitoring, permissions, rotation, escalation, decommissioning) did not include state-file maintenance because the problem had not surfaced yet. After five months of daily operation, state files in the Armada fleet grew past 1,000 lines, consuming significant shares of agent context windows and degrading synthesis quality. The sixth section, state-file maintenance, covers compaction schedules, retention rules, and archival procedures. It earned its place through operational experience, not theoretical planning.