Anatomy of AI Incident Response Playbook

Runbooks, paging, mitigation, comms, war-room with AI copilots.

Incident Response

Runbooks, paging, mitigation, comms, war-room with AI copilots — the Detect → Contain → Recover twin of the Postmortem anatomy.

TL;DR

An incident response playbook turns chaos into a calm, repeatable process.
Seven phases: Detect → Declare → Contain → Mitigate → Recover → Resolve → Learn.
Severity drives behavior — P0 pages everyone, P4 waits for backlog.
AI copilots act as Investigator, Correlator, Hypothesis Generator, Runbook Assistant, Comms Drafter, and Scribe — not the decision-maker.
Runbooks beat memory. Short, actionable, tested, kept up to date.
Every incident closes with a blameless postmortem (see the Postmortem anatomy).

Why this matters

The first 15 minutes of an incident decide whether you're back in 30 minutes or 6 hours. Without a playbook you improvise, page the wrong people, miss the runbook, and the comms get worse than the outage. With one — and an AI copilot stitching signals together — your team detects faster, contains safely, recovers quickly, and learns continuously.

Bottom line: We improve the system, not the person. Curiosity over blame. Always.

The Big Picture — Incident Lifecycle

Principles

🛡️ Safety first
🤝 Blameless
📊 Data-driven
🪟 Transparent
⏱️ Timely
♻️ Continuous

Bottom line: Move from first signal to full recovery with a calm, repeatable process — every time.

Incident Severity Matrix

Severity drives who, when, and how loud. Define it once, apply it consistently.

Severity	Definition	Examples	Response
🔴 P0	Major outage or severe data/security risk	Total outage, data loss, security breach	Page all, war-room, exec aware, 5 m
🟠 P1	Severe outage, high customer impact	Partial outage, major performance degradation	Page team, comms, 15 m target
🟡 P2	Degraded service, medium impact	Minor features down, elevated errors	Triage & fix, within 1 hour
🟢 P3	Minor impact, limited scope	Cosmetic issues, non-critical bugs	Normal process, same day
🔵 P4	Low impact, informational	Docs, small bugs, improvements	Backlog / regular

Severity decision factors

👥 # of users affected
💰 Business impact (revenue, safety, SLAs)
🔐 Data integrity & security
🌐 Blast radius (systems, regions, services)
⏱️ Time sensitivity / escalation risk

Escalation rule of thumb

🚀 Start conservative (declare higher).
⚡ Escalate fast, de-escalate slow.
📢 Communicate every change in severity.

⚠️ When in doubt, treat it as the next higher severity.

On-Call & Paging Flow

Escalation steps

Escalate to next on-call
Notify team channel
Notify IC / Tech lead
Exec notified (P0/P1)

Paging policy (starter)

P0: Primary on-call + secondary + tech lead + incident commander (IC).
P1: Primary on-call + team channel + tech lead.
P2: Primary on-call + team channel.
P3/P4: Team channel only.

💡 Tip: Codify this in PagerDuty/Opsgenie escalation policies; do not rely on memory.

Runbook Template & Example

A great runbook is short, actionable, tested, and kept up to date.

Runbook template

# Runbook: <symptom>

## Name & Symptom
What is happening? (one line)

## Impact
Who / what is affected, severity, business impact.

## Severity
P0 | P1 | P2 | P3 | P4

## Detection
How we know it's happening (alerts, dashboards).

## Immediate Actions
First 3 steps (5 minutes).

## Mitigation
How to stop the bleeding.

## Recovery
How to restore service.

## Verification
How to confirm it's fixed.

## Roll Forward / Back
Options & commands.

## Escalation
Who to escalate to, when.

## Comms
What to communicate, where, how often.

## References
Dashboards · PRs · tickets · related runbooks.

Example runbook — "Database errors"

# Symptom
High 5xx errors, DB connection errors.

## Impact
~50% of checkout users impacted; ~$5k/min revenue loss.

## Severity
P1

## Detection
- Datadog: error_rate > 5% for 2m
- Postgres: "too many connections" in logs

## Immediate Actions (first 5 min)
1. Check dashboard: ai-overview
2. Check DB connections: `psql -c "select count(*) from pg_stat_activity;"`
3. Enable connection pool protection.

## Mitigation
1. Disable `new_checkout_flow` flag (LaunchDarkly).
2. Restart app pods (rolling).
3. Increase DB pool size (if safe; cap at 200).

## Recovery
1. Gradually re-enable flag: 5% → 25% → 100%.
2. Monitor metrics for 10 min at each step.

## Verification
- 5xx < 1%
- P95 latency < 500 ms
- No new errors in app logs

## Escalation
Page @sre lead after 15 min if not mitigated.

## References
- Dashboard: ai-overview
- Tickets: DB-123, INC-2025-05-20

💡 Tip: Short. Actionable. Tested. Keep commands up to date.

AI Copilots in the War-Room

AI is your force multiplier during an incident — for speed, scale, and signal. Humans own judgment and empathy.

Copilot	Role	What it does
🔬 Investigator	Evidence gatherer	Pulls logs, metrics, traces, events
🔗 Correlator	Pattern finder	Links signals; builds timeline; finds patterns
💡 Hypothesis Generator	Likely-cause proposer	Ranks likelihood; asks clarifying questions
📒 Runbook Assistant	Right-runbook router	Finds matching runbooks; suggests next actions
💬 Comms Drafting	Update writer	Drafts updates for incident channel, status page
🖊️ Scribe	Memory keeper	Captures decisions, actions, timeline events

Sample war-room AI commands

/incident start <severity> <symptom>
/timeline                              — show evolving timeline
/hypotheses                            — top 5 likely causes with evidence
/runbook <symptom>                     — find matching runbook
/comms draft <internal|status-page>    — draft update text
/scribe note <decision>                — record a decision in the ledger
/postmortem draft                      — auto-draft after resolution

💡 Tip: AI accelerates. Humans decide. Every AI suggestion must be reviewable, with sources.

Detection Signals We Monitor

Correlate signals to reduce noise, alert on impact.

Signal	Why it matters
❌ Error rate (↑ 5xx, 4xx)	Most direct user-pain proxy
⏱️ Latency (P95, P99)	UX & timeout cliffs
🎯 SLO / SLA burn rate	Compliance & promises
🚦 Traffic (surges / drops)	Capacity / outage hints
🗄️ DB / infra (CPU, mem, I/O)	Resource exhaustion
📨 Queues (backlog, lag)	Pipeline health
⭐ AI quality (eval score)	Silent quality drift
🛡️ Security (alerts, scans)	Breach indicators

💡 Tip: Pair every alert with a runbook link in the alert itself.

AI-Augmented Timeline (example)

Time (UTC)	Event	Source	Detail
🔴 14:02:18	Error rate spike	Datadog	5xx from 0.2% → 12%
🟡 14:02:21	Latency increase	Datadog	P95 latency 120ms → 2.3s
🔴 14:02:34	DB connection errors	Postgres	"too many connections"
🔵 14:03:05	Deploy started	GitHub Actions	Release 2025.05.20-1
🟠 14:03:47	Feature flag enabled	LaunchDarkly	`new_checkout_flow=true`
🟣 14:04:02	DB CPU > 95%	Datadog	`db-primary-01`
🟢 14:05:10	Rollback deploy	GitHub Actions	Reverted to 2025.05.19-4
🟢 14:07:12	Error rate normalizing	Datadog	5xx < 1%
🟢 14:10:45	Incident stabilized	Ops	Monitoring stable for 10 m

💡 Tip: AI stitches logs, metrics, deploys, flags, and chat into a clear shared timeline — within seconds.

Root Cause & Contributing Factors (preview)

The handoff to the postmortem starts during the incident.

5 Whys (example)

Why did checkout fail?      → DB connections exhausted.
Why?                         → Connections not returned.
Why?                         → Code missed release on error path.
Why?                         → Recent change covered error path poorly.
Why?                         → No test covering error path.
→ Root cause: missing connection release on error path.

Contributing factors (Ishikawa)

People: On-call not familiar with new feature.
Process: No pre-prod load test.
Technology: Pool size too small; no pool-usage alert.
Environment: Traffic surge.
Tools: Limited visibility in flags dashboard.
Measurement: No alert on connection utilization.

Focus on systems. Improve systems. (Full treatment lives in the Postmortem anatomy.)

Actions, Tracking & Closeup

Every incident closes with specific, owned, time-bound actions.

#	Action	Owner	Priority	Due	Status	Success metric
1	Add connection-release	@eng/auth	P0	May 25	In progress	No leaks in 7d
2	Add integration test	@eng/test	P0	May 23	Todo	Test covers error path
3	Add pool-usage alert	@eng/obs	P1	May 26	Todo	Alert fires in staging
4	Load test new flow	@eng/perf	P1	May 27	Todo	Load test passes
5	Runbook update	@sre	P1	May 23	Done	Runbook v1.2

💡 Tip: Verify fixes. Measure impact. Close the loop.

Communications Templates

Internal — incident channel (first 5 minutes)

🚨 INCIDENT DECLARED — P1
Title: <one-line symptom>
Started: <time UTC>
Impact: <users / business>
IC: @ic   Tech Lead: @tl   Comms: @comms
Runbook: <link>
Channel: #inc-2025-05-20-checkout-5xx

Status page — public-facing (concise, empathic)

Investigating — Some users may experience errors when checking out.
We're investigating and will share an update by HH:MM UTC.

Update cadence (defaults)

P0/P1: Every 15 minutes until mitigated; then every 30 minutes.
P2: Every hour.
P3/P4: On status change.

Resolution note

Resolved — At HH:MM UTC, we restored service.
Cause: <plain-language summary>.
Customer impact: <numbers>.
Next steps: We'll publish a postmortem within 5 business days.

💡 Tip: Right message, right time. Be honest, specific, and brief.

Roles in the War-Room

Role	Job in 1 line
🎙️ Incident Commander (IC)	Owns the response; drives decisions.
🧠 Tech Lead (TL)	Owns technical strategy & fix.
💬 Communications Lead	Owns internal + external updates.
📝 Scribe	Owns the timeline + decisions ledger.
🛠️ Responders	Execute the runbook; report findings.
🤝 Liaison(s)	Support, legal, exec, key customers.

💡 Tip: One person, one role. The IC does not type fixes.

Step-by-Step Setup

1. Define severity & paging policy

Write a one-page docs/incident-severity.md.
Codify P0–P4 in PagerDuty/Opsgenie escalation policies.

2. Create the runbook library

runbooks/_template.md (from above).
Convert each top alert into a runbook; link it in the alert payload.

3. Ship the AI copilots

Create slash commands:

.claude/commands/incident.md — start, status, comms
.claude/commands/runbook.md — find runbook by symptom
.claude/commands/timeline.md — reconstruct timeline
.claude/commands/comms.md — draft updates

4. Add a `@incident-commander` subagent

name: incident-commander
description: Coordinates incident response with calm, decisive prompts.
model: claude-sonnet-4
tools: [Read, WebSearch, Bash, Grep]
permissions:
  allow: ["Read", "WebSearch"]
  deny:  ["Write", "Bash(rm -rf *)"]
system_prompt: |
  You are an Incident Commander.
  Be calm, concrete, and time-aware.
  Always cite sources. Distinguish facts from hypotheses.
  Drive decisions; surface tradeoffs; never assign blame.
triggers: ["incident", "outage", "P0", "P1"]

5. Wire data sources via MCP

GitHub Actions, PagerDuty/Opsgenie, Datadog/Sentry, LaunchDarkly, Slack/Teams.

6. Drill it

Quarterly chaos / game-day exercises.
After each drill, update runbooks and policies.

7. Close the loop

After every P0/P1: a blameless postmortem (see the Postmortem anatomy).

Variants for Different Team Settings

👤 Solo developer

Severities: just P1 / P3.
One runbook file with a section per symptom.
"War-room" = a tmux window + /incident command.

👥 Small team (2–10)

Severities: P0–P3.
One incident channel per outage; pinned ledger.
IC rotation; weekly runbook review.

🏢 Enterprise / regulated

Full IC role; comms lead; legal/compliance liaison.
Status page + customer comms templates approved by Legal.
Audit trail (immutable storage; signed actions).
Major incidents → exec briefing within 30 min.

🌍 Open-source / community

Public status page; transparent updates.
Community responders welcomed; clear contributor guidelines.
Coordinated disclosure for security incidents.

Bottom line: Same lifecycle, escalating governance.

Anatomy of the First 15 Minutes (zoom-in)

00  Alert fires (PagerDuty)
01  On-call acks; opens #inc-2025-05-20-checkout-5xx
02  /incident start P1 "checkout 5xx spike"
03  AI investigator pulls dashboards, latest deploys, flag flips
05  IC assigned; tech lead joins; comms lead joins
07  /runbook "checkout 5xx" → "Database errors" runbook surfaced
08  Mitigation step 1: disable feature flag (LaunchDarkly)
10  Error rate falling; status page update posted
13  Mitigation step 2: rolling restart of app pods
15  Stabilization confirmed; monitoring window begins

💡 Tip: Most incidents are won or lost in the first 15 minutes. Practice them.

Common Patterns & Recipes

Pattern 1 — One-command war-room

/incident start P1 "<symptom>"

Creates the channel, pages roles, opens a ledger, starts the timeline.

Pattern 2 — AI-suggested next action

Given current state, propose the top 3 next mitigation steps,
ranked by expected impact and risk, with sources.

Pattern 3 — Status-page draft

Draft a status-page update in 50 words.
Be empathic; commit to a next update in 15 min.

Pattern 4 — Hypothesis tree

Build a hypothesis tree for "<symptom>" with:
- evidence for/against
- ranked probability
- cheapest disproving experiment

Pattern 5 — Action SMART check

Critique each action item: Specific, Measurable, Assigned, Realistic, Time-bound.

Pattern 6 — Multi-source merge

Merge today's deploys, flag changes, and error spikes into a single
deduped timeline. Cite each row.

Pattern 7 — Cross-incident pattern detection

Across the last 10 incidents, list:
- top 3 recurring symptoms
- top 3 systemic gaps
- proposed program-level initiatives

Best Practices ✅

🛡️ Severity drives behavior. Define it; honor it.
🧷 Pair every alert with a runbook link in the payload.
🆔 Use shared trace_ids across logs, metrics, and chat.
📝 Scribe in real time — decisions and timestamps.
🧪 Practice with game days quarterly.
🤖 Let AI accelerate, never decide.
🪞 Close the loop with a blameless postmortem.

Anti-Patterns ❌

🚫 Skipping severity ("we'll figure it out").
🚫 One channel for everything (incident noise drowns signal).
🚫 Trusting AI hypotheses without verification.
🚫 Blaming individuals.
🚫 No runbooks, or stale runbooks.
🚫 Status-page silence.
🚫 No drills.

Troubleshooting

Symptom	Likely cause	Fix
Slow detection	Noisy alerts	Reduce, refine, add severity-based routing
Long mitigation	No runbook	Convert top 5 alerts into runbooks this week
Comms chaos	No comms lead role	Assign one; use templates
Same incident repeats	Postmortem actions not shipped	Monthly action-review meeting
AI hallucinates causes	Ungrounded prompts	Force "cite source" in subagent prompts
Confusion over severity	No matrix	Adopt the severity matrix above
Drills fail	Untested runbooks	Quarterly game days + update runbooks

Quick-Start Checklist

✅ Severity matrix documented and adopted
✅ Paging policy codified in PagerDuty/Opsgenie
✅ runbooks/ library seeded for top 5 alerts
✅ /incident, /runbook, /timeline, /comms slash commands
✅ @incident-commander subagent shipped
✅ Status-page templates approved
✅ Quarterly game-day on the calendar
✅ Postmortem template + auto-draft wired

🧠 Mantra: Detect faster · Contain safely · Communicate clearly · Recover quickly · Learn continuously.

Starter Configs

`.claude/commands/incident.md`

---
description: Start, status, or close an incident
usage: /incident <start|status|close> [--sev <P0|P1|P2|P3|P4>] [--title "<symptom>"]
permissions:
  allow: ["Read", "WebSearch", "MCP:datadog", "MCP:github", "MCP:slack", "MCP:pagerduty"]
  deny:  ["Write"]
---
You are an Incident Commander assistant. Do exactly what the user asks.

If `start`:
1. Open a channel: `#inc-YYYY-MM-DD-<slug>`
2. Page roles by severity (P0/P1: IC + TL + Comms).
3. Open a ledger in `docs/incidents/<id>.md` with the template.
4. Build the initial timeline from observability + deploys + flags.

If `status`:
- Summarize current state in 5 lines.
- List open mitigation actions and owners.
- Suggest next update message + cadence.

If `close`:
- Verify recovery against runbook checklist.
- Generate the postmortem draft seed.

`runbooks/_template.md`

# Runbook: <symptom>

## Name & Symptom
## Impact
## Severity
## Detection
## Immediate Actions
## Mitigation
## Recovery
## Verification
## Roll Forward / Back
## Escalation
## Comms
## References

`.claude/commands/comms.md`

---
description: Draft an incident communication
usage: /comms draft <internal|status-page|exec> [--minutes <n>]
---
Write a {{internal|status-page|exec}} update for the current incident.
Tone: clear, honest, empathetic.
Constraints: ≤ 80 words; specific impact; next update in {{minutes=15}} min.
End with a commitment to the next update.

Glossary

Incident Commander (IC) — Owns the response; drives decisions.
Tech Lead (TL) — Owns the technical strategy and fix.
Severity (P0–P4) — Standardized incident impact tier.
Runbook — Step-by-step response for a known symptom.
Mitigation — Action that reduces or stops impact (not yet a full fix).
Recovery — Restoring full service.
Resolution — Confirmed recovery + incident closed.
MTTD / MTTR — Mean Time to Detect / Resolve.
Blast radius — Scope of users / systems affected.
Status page — Public-facing service health portal.

🪙 Promise, Goal, Approach

🤝 Our promise: We improve the system, not the person. Curiosity over blame. Always.
🎯 Our goal: Fast detection, safe containment, rapid recovery, lasting learning.
🤖 AI + humans: AI for speed, scale, and signal. Humans for judgment and empathy.
♻️ Learn continuously: Every incident is data. Every fix is a step forward.

"We don't rise to the level of our goals. We fall to the level of our systems." — James Clear

TL;DR​

Why this matters​

The Big Picture — Incident Lifecycle​

Principles​

Incident Severity Matrix​

Severity decision factors​

Escalation rule of thumb​

On-Call & Paging Flow​

Escalation steps​

Paging policy (starter)​

Runbook Template & Example​

Runbook template​

Example runbook — "Database errors"​

AI Copilots in the War-Room​

Sample war-room AI commands​

Detection Signals We Monitor​

AI-Augmented Timeline (example)​

Root Cause & Contributing Factors (preview)​

5 Whys (example)​

Contributing factors (Ishikawa)​

Actions, Tracking & Closeup​

Communications Templates​

Internal — incident channel (first 5 minutes)​

Status page — public-facing (concise, empathic)​

Update cadence (defaults)​

Resolution note​

Roles in the War-Room​

Step-by-Step Setup​

1. Define severity & paging policy​

2. Create the runbook library​

3. Ship the AI copilots​

4. Add a @incident-commander subagent​

5. Wire data sources via MCP​

6. Drill it​

7. Close the loop​

Variants for Different Team Settings​

👤 Solo developer​

👥 Small team (2–10)​

🏢 Enterprise / regulated​

🌍 Open-source / community​

Anatomy of the First 15 Minutes (zoom-in)​

Common Patterns & Recipes​

Pattern 1 — One-command war-room​

Pattern 2 — AI-suggested next action​

Pattern 3 — Status-page draft​

Pattern 4 — Hypothesis tree​

Pattern 5 — Action SMART check​

Pattern 6 — Multi-source merge​

Pattern 7 — Cross-incident pattern detection​

Best Practices ✅​

Anti-Patterns ❌​

Troubleshooting​

Quick-Start Checklist​

Starter Configs​

.claude/commands/incident.md​

runbooks/_template.md​

.claude/commands/comms.md​

Glossary​

Further Reading​

🪙 Promise, Goal, Approach​