From clinical medicine's bedside method, made a teachable discipline by Richard C. Cabot (Differential Diagnosis, 1911) and refined by Lawrence L. Weed (1969) and Pat Croskerry (2003).
Lists every plausible cause of a symptom, flags the common and the dangerous ones, then gathers the evidence that separates them until one explanation is strong enough to act on.
Four causes of a retention drop, narrowed by tests that tell them apart
Seven-day retention fell 8 percent in a week. At each test, pick the check to run: the evidence answers, and the verdict says whether that check separates the causes or fits all of them. The meter counts the causes still in play; a good test removes at least one. Illustrative.
Analyst
The evidence
Analyst
The evidence
Separating tests picked: 0 of 2Separating tests picked: 1 of 2Separating tests picked: 2 of 2
Causes still in play4of 4
- List the causes (Analyst). Four candidates, with rough starting odds: a bug in last week's release (35 percent), the usual seasonal dip (30), a rival's launch (20), a tracking error that only looks like a drop (15). Two are flagged as must not miss: the bug, because users leave every day it stays live, and the tracking error, because if the drop is not real every other analysis is wasted.
- First test (Analyst). Start with the cause that would make every other test pointless if it were true, as long as it is cheap to check.
- Ask the release team whether the update could have broken anything. They are confident it did not. Testing passed and nothing is in the error queue. An opinion, and it leaves all four causes standing.
- Compare the drop in the analytics tool with the server's own login records. Server logins fell by the same 8 percent over the same days. Separating: the drop is real, so the tracking error is out.
- Pull the same weeks from last year. Last year these weeks dipped 2 percent. Useful, but the cause that would void the rest is still unchecked.
- Three remain (The evidence). With tracking ruled out, three causes remain, and the bug is now the likeliest. The next test should split the bug from the two causes that would hit every user alike.
- Second test (Analyst). Look for a comparison each remaining cause predicts differently.
- Split retention by app version: users on the new release against those still on the old. Users on the new version retain 11 points worse; users on the old version are flat. Separating: season and the rival would hit both versions, so only the release fits.
- Survey users who left and ask why. The answers scatter: price, boredom, a rival app, nothing in particular. Slow and noisy, and every cause predicts some of these answers.
- Read the press coverage of the rival's launch. The launch was big news in the category. It shows the rival could matter, and says nothing about whether it did.
- One cause dominates (The evidence). Only the bug fits the version split. Its odds are now high enough to act on, while the other causes stay on the list in case the fix does not bring retention back.
- Act and confirm (Analyst). Roll back the release for users not yet affected, ship the fix, and recheck retention by version in a week. If the gap closes, the diagnosis holds; if not, return to the list.
An example
A meal-planning app's seven-day retention fell 8 percent in one week, and by Tuesday's meeting three people had three answers: the product lead blamed a rival's launch, marketing blamed the season, and engineering said the release was clean. The head of analytics wrote all three on the board, added a fourth (the tracking was broken and nothing had happened), and put a rough chance beside each.
She then marked two as must not miss. A live bug would cost users every day it stayed, and a tracking error would make every other analysis worthless. The first test was the cheapest that could clear the tracking question: server logins fell by the same 8 percent, so the drop was real. The second was chosen because the remaining causes predicted different things: season and the rival would hit every user, a bug only those who had updated. Users on the new version were retaining 11 points worse.
The team rolled the release back that afternoon and confirmed the gap had closed a week later.
Why it works
A team that grabs the first plausible cause stops looking once it has one, and every later fact gets read in its light. Writing down every plausible cause first keeps the alternatives alive long enough to be tested. Two rankings then set the order of work: how likely each cause is, and how costly it would be to miss. A rare cause that is cheap to check and ruinous to miss gets checked early, even though it is unlikely.
The second insight is in choosing tests. A test is worth running only if the causes predict different results for it. Evidence that every cause would produce, such as complaints or a dip in a chart, moves nothing. Evidence that one cause predicts and the others forbid moves the odds a long way, so the list shrinks fast.
The thinking behind it
Richard C. Cabot Richard C. Cabot, a Harvard physician who taught diagnosis through cases, made the method teachable in Differential Diagnosis: Presented through an Analysis of 383 Cases (1911), with a second volume in 1915. The book is organized around the complaint a patient brings, such as pain, fever, or cough, and works outward to the diseases that could produce it, the same order as a team starting from a symptom. He also began the Massachusetts General Hospital case conferences that became the Case Records of the New England Journal of Medicine.
Lawrence L. Weed Lawrence L. Weed, a physician, created the problem-oriented medical record in Medical Records, Medical Education, and Patient Care (1969). Care is organized around a numbered list of the patient's problems, each with notes in the SOAP form: subjective, objective, assessment, plan. The structure keeps every open problem on the page and forces each note to state the reasoning behind the next step, which is how a differential stays honest over days. The form is now close to universal in medicine.
Pat Croskerry Pat Croskerry, a physician and researcher on diagnostic error, cataloged in The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them (2003), in Academic Medicine, the habits of mind that lead clinicians to wrong answers: anchoring on a first impression, closing too early, and overconfidence among them. He argued that clinicians can learn to correct their own reasoning, with the step back to examine one's own thinking as the central strategy.
How to run it
- State the symptom precisely. Say what changed, by how much, since when, and where it shows (all users, one region, one version).
- List every plausible cause. Write them all down before testing any, including the one that says the symptom is not real.
- Rank by likelihood. Put a rough chance beside each, using base rates from past incidents where you have them.
- Flag the must-not-miss causes. Mark the ones that are costly to miss or would void every other analysis, whatever their odds.
- Choose tests that separate. For each candidate test, ask whether the causes predict different results; drop the ones they all predict.
- Update after each result. Strike causes the evidence rules out and revise the odds of the rest before choosing the next test.
- Act when one cause dominates. Treat the leading cause once its odds justify it, and keep the list until the fix confirms it.
Where it goes wrong
The list closes too early. The loudest person's cause becomes the working answer before the alternatives are written down; spot it when the meeting debates one cause's details and nobody has named a second. Write the full list first.
Tests confirm instead of separating. A team runs the checks its favorite cause predicts and reads every hit as proof, though the other causes predicted the same. For each test, write what each cause predicts before running it.
Nothing gets ruled out. Every cause stays at some odds forever, and the team never acts. Set the threshold for acting at the start, and when a cheap reversible fix is available, use it as a test.
When to use something else
- Five Whys. When the cause is already known and the job is tracing why it happened, down to what can be fixed.
- Bayesian Updating. When there is one hypothesis and the work is revising its probability as signals arrive.
- Abductive Reasoning. When the explanations need scoring on fit and simplicity before any test is possible.
Where it sits in SOLVE eX
Go to the source
- Differential Diagnosis: Presented through an Analysis of 383 Cases, W. B. Saunders; volume two 1915, 1911
- Medical Records, Medical Education, and Patient Care: The Problem-Oriented Record as a Basic Tool, Case Western Reserve University Press, 1969
- The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them, Academic Medicine 78(8), 2003