Bradford Hill Criteria

Thinking Tools4.1 Root Causes and the Case for Going BackAlso known as Hill's Criteria of Causation

Set out by Austin Bradford Hill (The Environment and Disease: Association or Causation?, 1965).

Weighs an observed link between an exposure and an outcome against nine tests, from strength and consistency to timing and dose response, to judge whether it is causal.

Three explanations for a drop in churn, weighed on Hill's nine considerations

Churn among new customers fell from 9 to 5 percent. Each rival explanation is scored 0 (no support) to 3 (strong support) on eight of the considerations. The ninth, timing, is a must: a cause that came after the effect is out, whatever it scores, as the support hours are here. Change a score, or fail the onboarding flow on timing, and watch the ranking move. Illustrative.

Timing: the cause began before churn started falling

MustNew onboarding flowAnnual plan discountLonger support hours

Strength: how large the link is

Weight1New onboarding flow3Annual plan discount2Longer support hours3

Consistency: seen across regions and customer types

Weight1New onboarding flow2Annual plan discount1Longer support hours3

Specificity: the drop is where this cause acts, and only there

Weight1New onboarding flow3Annual plan discount0Longer support hours2

Dose response: more exposure, less churn

Weight1New onboarding flow3Annual plan discount1Longer support hours3

Plausibility: a believable mechanism

Weight1New onboarding flow3Annual plan discount3Longer support hours3

Coherence: fits everything else known

Weight1New onboarding flow2Annual plan discount1Longer support hours2

Experiment: a holdout or test agrees

Weight1New onboarding flow2Annual plan discount0Longer support hours1

Analogy: similar causes have done this elsewhere

Weight1New onboarding flow1Annual plan discount2Longer support hours3

Weighted total

New onboarding flow19OutAnnual plan discount10OutLonger support hours20Out

Strongest causal caseNo explanation passes on timing: look for another cause.New onboarding flowAnnual plan discountLonger support hours

In order

  1. 19New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).
  2. 10New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).
  3. 20New onboarding flowAnnual plan discountLonger support hoursOut: fails a must (Timing: the cause began before churn started falling).

An example

A project management software company saw churn among new customers in their first ninety days fall from 9 percent to 5 percent between February and June. Three teams claimed the credit: product pointed to the new onboarding flow launched in early March, finance to the discount on annual plans begun in February, and support to the longer hours it started in May. The head of analytics weighed all three on Hill's considerations.

Support's case fell first. Churn was already falling in March, two months before the hours changed, and a cause cannot follow its effect. The annual plan discount had a plausible mechanism, but churn fell as much among customers on monthly plans, which the discount could not explain. The onboarding flow held up on most counts: the drop was confined to customers who went through it, those who finished more of its steps churned less, and a ten percent holdout kept on the old flow churned at 8 percent. The company funded the next round of onboarding work.

Why it works

No single test proves that an association in observational data is causal, and Hill never claimed one did. What the considerations do is ask the questions that a confounded, reversed, or chance association usually fails. A large effect is harder to explain away; one seen across settings is less likely to be a quirk of one sample; a dose response is hard to produce by accident; a mechanism, coherence with other facts, and a supporting experiment each close a different gap. Timing alone can rule a cause out, since a cause must come first.

Scoring rival explanations side by side matters as much as the tests. The question becomes which account the evidence supports best, and a favored story has to beat its rivals on the same terms.

The thinking behind it

Austin Bradford Hill Austin Bradford Hill, a medical statistician, set out his nine considerations in The Environment and Disease: Association or Causation? (1965), an address published in the Proceedings of the Royal Society of Medicine. They are ways of judging whether an observed association is causal, among them strength, consistency, and temporality, the cause coming before its effect. He spoke from long practice: he helped establish the randomized clinical trial and, with Richard Doll, the link between smoking and lung cancer.

How to run it

  1. State the association precisely. Write what moved with what, by how much, and over what period, such as churn among new customers falling from 9 to 5 percent between February and June.
  2. List every rival explanation. Include the dull ones, such as a seasonal pattern, a shift in the mix of customers, or a change in how churn is counted.
  3. Check timing first. Date each candidate cause against the start of the effect, and drop any that came after it.
  4. Score the rest on the other eight. Use a short scale for strength, consistency, specificity, dose response, plausibility, coherence, experiment, and analogy, with a note of the evidence behind each score.
  5. Look hardest at dose and experiment. More exposure should mean more effect, and a holdout or a staged rollout should show the gap; both are hard to fake.
  6. Judge the whole, in words. Compare the explanations side by side and say how strong the leading case is before anyone acts on it.

Where it goes wrong

The total gets treated as a verdict. Eight scores added up look precise, yet one clean experiment can outweigh several weak considerations. Read the total as a summary and the notes as the case, and when two explanations land close, design a test that separates them.

The second is scoring only the favored explanation. With no rival on the same sheet, any plausible story passes; spot it when the page has one column, and add the cheapest rival anyone can name, such as seasonality.

The third is skipping timing because the dates are fuzzy. A launch rolled out over six weeks still has a first day in each region. Date the cause where it first reached customers and check the effect against that date.

When to use something else

  • Causal Diagram. When the explanations share causes and the team needs to see which variables to control for before comparing them.
  • Difference-in-Differences. When a change reached some groups and not others, and their earlier trends can serve as the comparison.
  • Randomized Controlled Trial. When a randomized test is possible, since it settles the question more firmly than any review of observational evidence.

Where it sits in SOLVE eX

Go to the source

  • The Environment and Disease: Association or Causation?, Proceedings of the Royal Society of Medicine, 1965

Back to the catalog