Anatomy of AI-Powered Postmortem

Blameless retros with AI: timelines, RCA, contributing factors.

Blameless retros with AI-augmented timelines, root-cause analysis, contributing-factor analysis, and learning loops — turning every incident into a net improvement in reliability, safety, and team effectiveness.

---

TL;DR

• A modern postmortem turns an incident into an opportunity — never a witch hunt.
• The flow: Detect → Contain → Investigate → Analyze → Decide → Act → Learn.
• Be blameless, curious, data-driven, clear, actionable, and timely.
• AI is your investigator, analyst, writer, and coach — not the decision-maker.
• Distinguish a single Root Cause from the many Contributing Factors.
• Close the loop with evaluations that prove your fixes actually worked.

---

Why this matters

Systems fail. People are human. The real question after an incident isn't "who screwed up?" — it's "what about our system made this failure possible, and how do we make it impossible (or harmless) next time?" AI shortens the gap between "something broke" and "we've learned and improved" from days to hours — without diluting the human judgment that matters most.

Bottom line: "We don't rise to the level of our goals. We fall to the level of our systems." — James Clear

---

The Big Picture — The Postmortem Lifecycle

Principles

• 🤝 Blameless — investigate the system, not the people.
• 🔍 Curious — assume there's always something deeper.
• 📊 Data-driven — evidence beats opinion.
• 🪟 Clear — anyone in the company can read it.
• ✅ Actionable — every finding maps to an owner + due date.
• ⏱️ Timely — draft within 48 hours; publish within a week.

Bottom line: A postmortem that takes a month is a postmortem nobody reads.

---

AI Superpowers Throughout

Use AI as a force multiplier at every stage of the lifecycle.

Capability	What AI does	Where it helps
🧩 Aggregate	Collect logs, metrics, traces, chats, changes automatically	Investigate
🔗 Correlate	Link signals across systems & time with trace_ids	Investigate / Analyze
✍️ Summarize	Condense noise into readable narratives & key facts	Analyze / Draft
🧠 Analyze	Find patterns, anomalies, likely causes	Analyze
💡 Recommend	Suggest actions, owners, prevention	Decide

💡 Tip: AI is your investigator, analyst, writer, and coach — not the decision-maker. Humans own truth, accountability, and tone.

---

AI-Augmented Incident Timeline (example)

AI builds the timeline in seconds by stitching logs, metrics, deploys, and chat.

Time (UTC)	Event	Source	Detail
🔴 14:02:18	Error rate spike	Datadog	5xx jumped from 0.2% → 12%
🟡 14:02:21	Latency increase	Datadog	P95 latency 120 ms → 2.3 s
🔴 14:02:34	DB connection errors	Postgres	"too many connections"
🔵 14:03:05	Deploy started	GitHub Actions	Release 2025.05.20-1
🟠 14:03:47	Feature flag enabled	LaunchDarkly	new_checkout_flow=true
🟣 14:04:02	DB CPU > 95%	Datadog	db-primary-01
🟢 14:05:10	Rollback deploy	GitHub Actions	Reverted to 2025.05.19-4
🟢 14:07:12	Error rate normalizing	Datadog	5xx < 1%
🟢 14:10:45	Incident stabilized	Ops	Monitoring stable for 10 m

💡 Tip: A single shared trace_id makes this stitch-up trivial. Without it, expect hours of grep.

---

Postmortem Template (AI-Assisted)

AI can draft 80% of this in minutes. Humans decide the truth, actions, and tone.

Section	What it captures
📌 Executive Summary	What happened, impact, duration, status.
💥 Impact	Who was affected, what was impacted, severity.
⏱️ Timeline	Chronological sequence of key events.
🧪 What Happened	Technical narrative of the incident.
🎯 Root Cause	The one thing that, if fixed, prevents recurrence.
🧩 Contributing Factors	The conditions that made it possible.
✅ What Went Well	What worked. Be proud of it.
⚠️ What Didn't Go Well	What didn't work. What we'll improve.
📋 Action Items	Actions, owners, priority, due dates.
🛡️ Preventive Actions	Systemic changes to prevent recurrence.
🔁 Follow-up & Success Criteria	How we'll verify the fix & measure success.
🔗 References	Links to dashboards, PRs, docs, tickets.

---

AI-Assisted Postmortem Workflow

Step	AI does	Outcome
🚨 Trigger	Collects initial signals & context.	Incident channel + ledger started
📊 Data Collection	Aggregates & normalizes all data.	Single artifact for the timeline
⏱️ Timeline	Builds chronological timeline with correlations.	Verified shared narrative
🔬 Analysis	Proposes root cause + contributing factors with evidence.	Hypotheses to validate
📝 Draft Postmortem	Generates first draft. Highlights gaps.	Working document for the team
👥 Review & Validate	Answers questions. Suggests clarifications.	Final, human-owned report
🚀 Publish & Track	Creates tasks. Tracks progress & impact.	Closed-loop learning

💡 Tip: AI accelerates the process. Humans ensure accuracy, empathy, and action.

---

Data Sources AI Pulls From

The more sources you correlate, the more accurate your analysis.

Source	Examples
👁️ Observability	Metrics, logs, APM, traces
🚨 Alerts	PagerDuty, Opsgenie, Datadog, Sentry
🚀 Deployments	GitHub Actions, ArgoCD, Terraform
🎚️ Feature Flags	LaunchDarkly, Unleash
💬 Chat / Comms	Slack, Teams, incident channel
🎫 Tickets / Issues	Jira, Linear, GitHub Issues
📒 Runbooks	Read or write actions taken
🛠️ Infrastructure	CloudWatch, GCP, Azure logs

💡 Tip: More sources = better accuracy. Correlated signals beat single logs.

---

Root Cause & Contributing Factor Analysis

Two ideas, often confused:

Root Cause

The fundamental issue that, if fixed, would have prevented the incident.

Example: "A database connection leak in the new checkout service caused connection pool exhaustion, leading to cascading failures."

Contributing Factors

The conditions and decisions that made the incident possible — or made it worse.

• New feature flag enabled without load test
• Insufficient connection pool sizing
• Missing alert on connection utilization
• No auto-scaling on DB tier
• Incomplete runbook for rollback

Classic techniques (AI accelerates each one)

5 Whys

1. Why did checkout fail?       — DB couldn't accept new connections.
2. Why?                          — Connection pool was exhausted.
3. Why?                          — Connections weren't returned after use.
4. Why?                          — Recent code missed a `client.release()` path.
5. Why?                          — No integration test exercised the error branch.
→ Root cause: missing `release()` on error path.
→ Contributing: no test, no pool-utilization alert, no canary.

Fishbone (Ishikawa) — categories of contributing factors

• People — On-call training, handoff quality.
• Process — Change-management, runbooks.
• Technology — Connection pool, autoscaler, monitoring.
• Environment — Traffic surge, dependent service.
• Tools — Feature-flag visibility, deploy speed.
• Measurement — Missing alert, eval gaps.

💡 Tip: Focus on the system. Improve the system. A culture of asking "why?" five times is worth more than any AI tool.

---

Evaluations for Incident Response

Run evals on every P1/P2 incident. Learn and improve the process.

Eval	Questions	Goal
⏱️ Timeliness	Were we alerted quickly? Time to acknowledge? Detection coverage?	Detect in < X minutes
🛡️ Mitigation effectiveness	Time to mitigate? Blast radius contained? Customer impact reduced?	MTTR < X minutes
💬 Communication	Updates timely & clear? Right channels used? Stakeholders informed?	Clear & empathetic
📝 Postmortem quality	Root cause accurate? Actions specific & measurable? Learning captured?	Actionable & specific

💡 Tip: Score yourselves honestly. Trends matter more than any single number.

---

Drift & Regression Detection

Even after the fix, watch for regressions:

• 📊 Metric drift — error rate (7-day avg)
• ✅ Quality drift — eval score (7-day avg)
• 🤖 Behavior drift — hallucination rate, response shape

💡 Tip: Detect early. Alert fast. Prevent the next incident.

(See the Anatomy of AI Observability poster for the full drift-detection toolkit.)

---

Alerting & Auto-Response

Pair each alert with a runbook and an auto-response where safe.

Alert examples

• 🔴 Critical: Error rate > 3% for 5 m
• 🔴 Critical: Latency P95 > 3 s for 5 m
• 🟡 Warning: Eval score drop > 10%
• 🟡 Warning: Spike in hallucinations
• 🔵 Info: New model deployed
• 🔵 Info: Large config change

Auto-response playbook

1. Page on-call (with runbook link).
2. Triage with AI copilot (/incident).
3. Mitigate (rollback, disable flag, scale).
4. Communicate status.
5. Create incident & timeline.
6. Draft postmortem.
7. Track actions to completion.

---

Step-by-Step Setup

1. Standardize the incident channel

Every incident gets a Slack/Teams channel named inc-YYYY-MM-DD-summary with a pinned ledger.

2. Wire the data sources

Connect Datadog/Sentry/PagerDuty/GitHub/LaunchDarkly to your AI postmortem agent via MCP or webhooks.

3. Drop in the templates

• docs/postmortems/_template.md
• runbooks/_template.md

4. Add a /postmortem slash command

---
description: Generate the first draft of a postmortem
usage: /postmortem <incident_id>
permissions:
  allow: ["Read", "WebSearch", "MCP:datadog", "MCP:github", "MCP:slack"]
  deny:  ["Write"]
---
You are an SRE writing a blameless postmortem.
1. Pull all signals for {{incident_id}} (alerts, logs, traces, deploys, chat).
2. Build an event timeline.
3. Propose Root Cause + Contributing Factors with evidence.
4. Draft the report using docs/postmortems/_template.md.
5. Highlight gaps where humans must verify.
Tone: respectful, curious, system-focused.

5. Ship a /timeline and /runbook command

• /timeline → reconstruct the timeline only.
• /runbook <symptom> → suggest matching runbook.

6. Add a @postmortem-writer subagent

name: postmortem-writer
description: Drafts blameless postmortems with evidence.
model: claude-sonnet-4
tools: [Read, WebSearch, Bash, Grep]
system_prompt: |
  You are an SRE. Be blameless, curious, data-driven, clear.
  Never name people; refer to roles or teams.
  Distinguish Root Cause from Contributing Factors.
  Cite sources for every claim.
permissions:
  allow: ["Read", "WebSearch"]
  deny:  ["Write", "Bash(rm -rf *)"]
triggers: ["postmortem", "incident", "RCA"]

7. Define the eval suite for postmortems

postmortem_quality:
  - clarity_score:        weight: 0.25
  - root_cause_accuracy:  weight: 0.25
  - action_specificity:   weight: 0.25
  - blameless_tone:       weight: 0.15
  - timeliness:           weight: 0.10

Run it on every published postmortem; trend the scores.

8. Close the loop

A monthly review meeting reads the action-item completion rate + eval trend.

---

Variants for Different Team Settings

👤 Solo developer

• Use /postmortem for any outage > 30 min.
• Single Markdown file + Git history is your "ledger."
• 1 action item per incident, completed within a week.

👥 Small team (2–10)

• Incident channel + /postmortem draft + 30-min review meeting.
• Weekly action-item review.
• Shared docs/postmortems/ folder.

🏢 Enterprise / regulated

• Incident commander role.
• Full audit trail (immutable, signed).
• Compliance & legal review gate for severity ≥ P1.
• Quarterly trend analysis across incidents.
• Tie postmortem quality to engineering performance review goals (not individual blame).

🌍 Open-source / community

• Public postmortems for outages affecting users.
• Anonymized contributor names.
• Crowdsourced action items via GitHub Issues.

Bottom line: Same lifecycle, escalating governance.

---

Anatomy of a Single Section — Zoom In on "What Happened"

A great "What Happened" section answers three questions, with citations.

## What Happened

**Trigger.** At 14:02 UTC, link enabled the
`new_checkout_flow` flag for 100% of traffic. The new code path opens
a database connection per request but never releases it on the
404-handler branch.

**Propagation.** Within 90 seconds, the primary checkout DB's
connection pool was exhausted. New requests began failing with
HTTP 5xx (link). Dependent services
(cart, recommendations) backed up because they share the pool.

**Detection & response.** PagerDuty paged the on-call at 14:03:50.
The on-call disabled the feature flag at 14:04:42 and triggered a
rollback at 14:05:10. Error rate normalized by 14:07:12.

**Customer impact.** Approximately 41,000 checkout attempts failed
(8.2% of the window). Revenue impact estimated at ~$28k.

Notice: no names, every claim cited, specific numbers, plain language.

---

Common Patterns & Recipes

Pattern 1 — /postmortem one-liner

/postmortem inc-2025-05-20-checkout-5xx

Pattern 2 — Blameless rewrite

Rewrite the following section in a blameless tone.
Refer to roles, not names. Focus on systems.
<paste section>

Pattern 3 — Contributing-factor brainstorm

Given this timeline + root cause, list 5–7 contributing factors
across people, process, technology, environment, tools, measurement.
For each: evidence, severity, and a candidate preventive action.

Pattern 4 — Action-item SMART check

Critique each action item: is it Specific, Measurable, Assigned,
Realistic, Time-bound? Suggest improvements.

Pattern 5 — Eval-driven quality check

Score this postmortem 0–1 against:
clarity, root-cause accuracy, action specificity, blameless tone, timeliness.
Explain each score with 1 line of evidence.

Pattern 6 — Pattern detection across incidents

Across the last 10 postmortems in docs/postmortems/, list:
- recurring root-cause themes
- top 3 systemic gaps
- proposed program-level initiatives

Pattern 7 — Postmortem-as-runbook seed

Generate a runbook seed from this postmortem:
- Symptom detection
- First 3 mitigation steps
- Verification checks
- Communication template

---

Best Practices ✅

• 🤝 Be blameless and curious — always.
• 📐 Write for learning, not for politics.
• 🎯 Make actions specific & measurable.
• 🔁 Review postmortems for trends (quarterly).
• 🪞 Close the loop: measure action impact.
• 🆔 Instrument everything with trace_ids.
• ✍️ Use consistent templates.
• 🤖 Let AI draft; let humans decide.

Anti-Patterns ❌

• 🚫 Naming individuals as causes.
• 🚫 Postmortems with no action items.
• 🚫 Single root cause forced when many factors contributed.
• 🚫 Publishing weeks late (insights are stale).
• 🚫 Same incident, repeat causes (no learning loop).
• 🚫 AI hallucinations taken as fact without verification.
• 🚫 Closed-door process for non-sensitive incidents.

---

Key Metrics Dashboard

Metric	Target
⏱️ MTTR (mean time to resolve)	42 m or less
🔍 MTTD (mean time to detect)	< 6 m
🔁 Change failure rate	< 2.1%
⭐ Eval score (avg postmortem quality)	> 0.92
🤖 Hallucination rate in drafts	< 1%
📊 Weekly incidents	Trend ↓
✅ Action-item completion rate	> 87%
💰 Cost / incident (eng-hrs + $)	Trend ↓

💡 Tip: Measure what matters. Improve what you measure.

---

Troubleshooting

Symptom	Likely cause	Fix
Postmortems delayed	No template / no SLA	Adopt the template; commit to draft in 48 h
Same root causes repeat	Action items never shipped	Monthly action-review meeting
Tone slips into blame	No coach / no review	Add "blameless tone" eval; pre-review by an SRE lead
Timeline is messy	Sources not correlated	Standardize trace_id across systems
AI invents events	No grounding	Force "cite source" in subagent system prompt
Actions vague	No SMART check	Add the SMART critique pass
No learning	No trends review	Quarterly cross-incident retro

---

Quick-Start Checklist

• ✅ Incident channel naming convention
• ✅ Standard postmortem template
• ✅ /postmortem slash command
• ✅ @postmortem-writer subagent
• ✅ Wired data sources (alerts, deploys, chat, traces)
• ✅ Eval suite for postmortem quality
• ✅ Monthly action-review meeting
• ✅ Quarterly cross-incident retro

🧠 Mantra: Blameless. Curious. Data-driven. Clear. Actionable. Timely.

---

Starter Configs

docs/postmortems/_template.md

# Postmortem: <one-line summary>

**Incident ID:** inc-YYYY-MM-DD-slug
**Severity:** P{1|2|3}
**Status:** Draft | In Review | Published
**Duration:** HH:MM
**Author(s):** @role/@role
**Date Published:**

## Executive Summary
3–5 sentences. What happened, impact, status.

## Impact
- Users:
- Revenue:
- SLO burn:

## Timeline
| Time (UTC) | Event | Source | Detail |
|---|---|---|---|

## What Happened
Trigger → propagation → detection → response → recovery.

## Root Cause
The one thing.

## Contributing Factors
- People:
- Process:
- Technology:
- Environment:
- Tools:
- Measurement:

## What Went Well
-

## What Didn't Go Well
-

## Action Items
| # | Action | Owner | Priority | Due |
|---|---|---|---|---|

## Preventive Actions
Systemic, not patch-level.

## Follow-up & Success Criteria
How will we verify the fix and measure success?

## References
- Dashboard: …
- PR: …
- Runbook: …
- Slack thread: …

Repo layout

observability/
└── otel-config.yaml
dashboards/
├── ai-overview.json
└── incidents.json
alerts/
└── alerts.yaml
evaluations/
├── cases/
├── rubrics/
└── run-evals.sh
docs/
├── runbooks/
└── postmortems/

---

Glossary

• Postmortem — A structured, blameless write-up of an incident.
• Blameless — Focus on systems and learning, not individuals.
• Root Cause — The fundamental issue that, fixed, prevents recurrence.
• Contributing Factors — Conditions that enabled or worsened the incident.
• MTTD / MTTR — Mean Time to Detect / Resolve.
• Action Item — Specific, owned, time-bound corrective work.
• Runbook — Step-by-step response for a known symptom.
• Eval — Scored test of a process or output.
• Drift — Slow regression of metrics, quality, or behavior.

---

Further Reading

• Search: "Google SRE blameless postmortem culture"
• Search: "Etsy 'Debriefing Facilitation Guide'"
• Search: "learning from incidents in software"
• Search: "AI for incident response 2026"

---

🪙 The Goal & Principles

• 🎯 Goal: Turn every incident into a net improvement in reliability, safety, and team effectiveness.
• 🙅 Not about blame. We fix systems. People are human.
• 🧱 Focus on systems. Systems fail; systems can be improved.
• ♻️ Learn continuously. Every incident is data. Every fix is a step forward.

"We don't rise to the level of our goals. We fall to the level of our systems." — James Clear