Set out by Austin Bradford Hill (The Environment and Disease: Association or Causation?, 1965).
Weighs an observed link between an exposure and an outcome against nine tests, from strength and consistency to timing and dose response, to judge whether it is causal.
Three explanations for a drop in churn, weighed on Hill's nine considerations
Churn among new customers fell from 9 to 5 percent. Each rival explanation is scored 0 (no support) to 3 (strong support) on eight of the considerations. The ninth, timing, is a must: a cause that came after the effect is out, whatever it scores, as the support hours are here. Change a score, or fail the onboarding flow on timing, and watch the ranking move. Illustrative.
Timing: the cause began before churn started falling
MustNew onboarding flowAnnual plan discountLonger support hoursStrength: how large the link is
Weight1New onboarding flowAnnual plan discountLonger support hoursConsistency: seen across regions and customer types
Weight1New onboarding flowAnnual plan discountLonger support hoursSpecificity: the drop is where this cause acts, and only there
Weight1New onboarding flowAnnual plan discountLonger support hoursDose response: more exposure, less churn
Weight1New onboarding flowAnnual plan discountLonger support hoursPlausibility: a believable mechanism
Weight1New onboarding flowAnnual plan discountLonger support hoursCoherence: fits everything else known
Weight1New onboarding flowAnnual plan discountLonger support hoursExperiment: a holdout or test agrees
Weight1New onboarding flowAnnual plan discountLonger support hoursAnalogy: similar causes have done this elsewhere
Weight1New onboarding flowAnnual plan discountLonger support hoursWeighted total
New onboarding flow19OutAnnual plan discount10OutLonger support hours20OutStrongest causal caseNo explanation passes on timing: look for another cause.New onboarding flowAnnual plan discountLonger support hours
In order
- 19New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).
- 10New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).
- 20New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).
An example
A project management software company saw churn among new customers in their first ninety days fall from 9 percent to 5 percent between February and June. Three teams claimed the credit: product pointed to the new onboarding flow launched in early March, finance to the discount on annual plans begun in February, and support to the longer hours it started in May. The head of analytics weighed all three on Hill's considerations.
Support's case fell first. Churn was already falling in March, two months before the hours changed, and a cause cannot follow its effect. The annual plan discount had a plausible mechanism, but churn fell as much among customers on monthly plans, which the discount could not explain. The onboarding flow held up on most counts: the drop was confined to customers who went through it, those who finished more of its steps churned less, and a ten percent holdout kept on the old flow churned at 8 percent. The company funded the next round of onboarding work.
Why it works
No single test proves that an association in observational data is causal, and Hill never claimed one did. What the considerations do is ask the questions that a confounded, reversed, or chance association usually fails. A large effect is harder to explain away; one seen across settings is less likely to be a quirk of one sample; a dose response is hard to produce by accident; a mechanism, coherence with other facts, and a supporting experiment each close a different gap. Timing alone can rule a cause out, since a cause must come first.
Scoring rival explanations side by side matters as much as the tests. The question becomes which account the evidence supports best, and a favored story has to beat its rivals on the same terms.
The thinking behind it
Austin Bradford Hill Austin Bradford Hill, a medical statistician, set out his nine considerations in The Environment and Disease: Association or Causation? (1965), an address published in the Proceedings of the Royal Society of Medicine. They are ways of judging whether an observed association is causal, among them strength, consistency, and temporality, the cause coming before its effect. He spoke from long practice: he helped establish the randomized clinical trial and, with Richard Doll, the link between smoking and lung cancer.
How to run it
- State the association precisely. Write what moved with what, by how much, and over what period, such as churn among new customers falling from 9 to 5 percent between February and June.
- List every rival explanation. Include the dull ones, such as a seasonal pattern, a shift in the mix of customers, or a change in how churn is counted.
- Check timing first. Date each candidate cause against the start of the effect, and drop any that came after it.
- Score the rest on the other eight. Use a short scale for strength, consistency, specificity, dose response, plausibility, coherence, experiment, and analogy, with a note of the evidence behind each score.
- Look hardest at dose and experiment. More exposure should mean more effect, and a holdout or a staged rollout should show the gap; both are hard to fake.
- Judge the whole, in words. Compare the explanations side by side and say how strong the leading case is before anyone acts on it.
Where it goes wrong
The total gets treated as a verdict. Eight scores added up look precise, yet one clean experiment can outweigh several weak considerations. Read the total as a summary and the notes as the case, and when two explanations land close, design a test that separates them.
The second is scoring only the favored explanation. With no rival on the same sheet, any plausible story passes; spot it when the page has one column, and add the cheapest rival anyone can name, such as seasonality.
The third is skipping timing because the dates are fuzzy. A launch rolled out over six weeks still has a first day in each region. Date the cause where it first reached customers and check the effect against that date.
When to use something else
- Causal Diagram. When the explanations share causes and the team needs to see which variables to control for before comparing them.
- Difference-in-Differences. When a change reached some groups and not others, and their earlier trends can serve as the comparison.
- Randomized Controlled Trial. When a randomized test is possible, since it settles the question more firmly than any review of observational evidence.
Where it sits in SOLVE eX
Go to the source
- The Environment and Disease: Association or Causation?, Proceedings of the Royal Society of Medicine, 1965