From the just-culture work of Sidney Dekker (Just Culture, 2007) and John Allspaw (Blameless PostMortems and a Just Culture, 2012); Google's practice is set out by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds. (Site Reliability Engineering, 2016).
Reviews an incident on the premise that people acted reasonably on what they knew, so the account surfaces system and process causes and produces owned action items.
Five questions in the checkout postmortem, and what each one brings out
At each of the facilitator's turns, pick a question: one that looks for a culprit, or one that asks what the engineer saw and knew. Her answer shows, with a verdict, and the count tallies the turns that kept the room on the system and off a culprit. Illustrative.
Facilitator
On-call engineer
Facilitator
On-call engineer
Turns kept on the system: 0 of 5Turns kept on the system: 1 of 5Turns kept on the system: 2 of 5Turns kept on the system: 3 of 5Turns kept on the system: 4 of 5Turns kept on the system: 5 of 5
- Opening (Facilitator). Checkout was down for 47 minutes on Saturday. Six responders are in the room. Pick how the facilitator opens.
- Let's start with who pushed the config change. The engineer who pushed it says almost nothing for the rest of the hour. A hunt for a culprit: the people with the facts stop talking.
- Nobody here is in trouble. Walk me through what each of you saw and knew. The engineer who pushed the change speaks first: the deploy tool never asked for a canary on a config change. The rule is set, and the first system cause comes out.
- The change (On-call engineer). The deploy tool sends config changes straight to every server. Code changes go to a canary first.
- The first alert (Facilitator). At 02:14 errors spiked and she rolled back the other payment service. Pick the question.
- Why did you roll back the wrong service? I thought it was the right one at the time. A why-did-you question gets a defense, and no new fact.
- What did the dashboard show you at 02:14? Both payment services in one panel. The other one had the newer deploy, so it looked like the cause. What she knew: the dashboard made the wrong rollback the reasonable one.
- What she saw (On-call engineer). Given that display, rolling back the newest deploy was what most engineers would have done.
- The long middle (Facilitator). Twenty-six minutes passed before the real cause was found. Pick the question.
- Twenty-six minutes is a long time. What took so long? I was working through it as fast as I could. Pressure, and the answer closes the topic.
- What did you reach for next, and who could you call? The runbook, which still pointed at last year's dashboard. And nobody: the second on-call slot has been empty since spring. Two causes in the system: a stale runbook and an unfilled slot.
- What slowed it (On-call engineer). Both were known to the team lead and neither had an owner.
- The causes (Facilitator). The facilitator writes up the causes. Pick the heading.
- Root cause: human error during a rollback. The room nods, and the four fixable causes drop out of the write-up. Blame in a softer form: nothing gets fixed.
- Contributing causes: the deploy tool, the dashboard, the runbook, the staffing. Each cause is one the team can change, and none of them is a person. Causes in the system, each with a fix.
- Four causes (On-call engineer). The engineer's actions appear in the timeline, not in the causes.
- Action items (Facilitator). Pick what the postmortem asks for.
- Everyone be more careful with config changes. No owner, no date. The same gap breaks search in November. A wish: nothing changes.
- Four items, each with one owner and one date. Canary for config changes, in two weeks. Split the panel, one week. Rewrite the runbook, Friday. Fill the slot, this quarter. Owned action items, tracked until they close.
- The write-up (On-call engineer). Two other teams find the same canary gap in their own deploys the next week.
An example
Checkout at an online furniture retailer went down for 47 minutes early on a Saturday. The first draft of the incident note said the on-call engineer rolled back the wrong service. Her manager was already talking about retraining.
The engineering director called a postmortem for Tuesday, gave the facilitator one rule, no blame, and asked each responder what they saw and knew at each moment. The account changed. The config change that broke checkout went out through a deploy tool that never asked for a canary on config. The dashboard grouped both payment services in one panel, and the other service showed the newer deploy, so rolling that one back was the reasonable move. The runbook pointed at last year's dashboard. She was alone because the second on-call slot had been empty since spring.
The postmortem produced four action items, each with an owner and a date, and nothing about retraining. It was published to all of engineering, and two other teams found the same gap in their own deploys.
Why it works
People tell the full story of an incident only when telling it is safe. A review that hunts for the person who erred teaches everyone to say less next time, and the details that matter most, what was confusing and what was missing, are exactly the ones a nervous engineer leaves out.
The premise that everyone acted reasonably on what they knew changes the question from who failed to what made the failure reasonable. That question leads to the dashboard, the tool, the runbook, and the staffing, which can be fixed, and away from the person, who would be replaced by someone facing the same traps. Publishing the write-up widely turns one team's incident into every team's warning.
The thinking behind it
John Allspaw John Allspaw, then head of operations at Etsy, carried the just culture idea from safety science into software in his post Blameless PostMortems and a Just Culture (2012), on Etsy's engineering blog. He argued that engineers who expect punishment stop giving full accounts, so a postmortem should ask what happened and why, without blaming the person, so mistakes are reported in detail. He later studied human factors and systems safety at Lund University.
Sidney Dekker Sidney Dekker, a safety scientist at Griffith University, set out the case in Just Culture: Balancing Safety and Accountability (2007). He argues that organizations learn more when they treat an error as a symptom of the system it happened in than when they blame the individual who made it. The book's subtitle names the balance a blameless review has to strike, and he later developed restorative just culture.
Betsy Beyer Betsy Beyer edited Google's 2016 book on site reliability engineering, published by O'Reilly, with Chris Jones, Jennifer Petoff, and Niall Richard Murphy. Its chapter on postmortem culture, written by John Lunney and Sue Lueder, describes the blameless postmortem as Google practices it: a written record of an incident, its impact, its causes, and its follow-up actions, written on the assumption that everyone involved acted in good faith on the information they had.
How to run it
- Meet within a few days. Hold it after the responders have slept and while the timeline is still fresh, usually two or three working days after.
- Appoint a facilitator for the rule. Give one person, ideally from outside the team, the job of stopping any question that hunts for a culprit.
- Rebuild the timeline with what each person knew. For each decision, record what the person saw on their screens and what they had been told at that moment.
- Ask what made each action reasonable. Where a choice looks wrong in hindsight, find the display, tool, or instruction that made it look right at the time.
- Find causes across tools, process, and staffing. List every contributing cause, since incidents rarely have only one.
- Assign action items with owners and dates. Each item names one person and one date, and is tracked until it closes.
- Publish the write-up. Send it to every team that runs similar systems, with the timeline and the action items intact.
Where it goes wrong
Blame comes back in a softer form. Phrases such as human error, lack of attention, or should have known put the cause back on the person, and the room hears it. The facilitator rewrites each one as a question about what the person saw and what made the action reasonable.
The action items are wishes. Be more careful and improve communication have no owner and no end, and nothing changes. Every item needs one name, one date, and a test for done.
The write-up stays inside the team. A postmortem filed in one team's folder teaches nobody else, and the same gap breaks another service a month later. Publish it widely, and track the items in the open until they close.
When to use something else
- After-Action Review. When the event was planned, such as a launch or an exercise, and the team wants a quick comparison of the plan with what happened.
- Five Whys. When one failure needs its cause chased down in a short session and the causes are likely to form a single chain.
- Chain of Custody. When the incident may end in discipline, a lawsuit, or a regulator's review, and the evidence has to be proved.
Where it sits in SOLVE eX
Go to the source
- Just Culture: Balancing Safety and Accountability, 2007
- Blameless PostMortems and a Just Culture, Etsy Code as Craft, 2012
- Site Reliability Engineering: How Google Runs Production Systems, O'Reilly, 2016