Propensity Score Matching

Thinking Tools6.4 Conducting Follow-Up and Taking Feedback

Developed by Paul R. Rosenbaum and Donald B. Rubin (The Central Role of the Propensity Score in Observational Studies for Causal Effects, 1983).

Pairs each unit that received a treatment with a similar untreated one, matched on its estimated chance of being treated, so the groups compare as if assigned at random.

Matching each store that joined the loyalty program to the most similar store that did not

Each store has an adoption score: the chance, estimated from its size, region, and past growth, that a store like it would join. Pick a match for each of the four stores that joined; the black line is their scores and the colored line is the scores of the matches you picked. Poor matches give the same five points as the naive comparison; close the gap between the lines and the estimate settles near three. Store A starts matched as an example. Illustrative.

candidate 1
candidate 1
candidate 1
candidate 2
Average gap in adoption score across the four pairs
18.0 points
Naive estimate of extra growth: all 4 joiners against all 40 other stores
5.0 points
Matched estimate of extra growth: each joiner minus its match, averaged
5.0 points

Matches you pickedStores that joined

0255075100Store AStore BStore CStore D0255075100Store AStore BStore CStore D

An example

A regional grocery chain had let store managers decide whether to join a new loyalty program, and the four stores that joined grew 7 percent over the year against 2 percent for the other forty. Marketing's summary said the program added five points of growth and proposed rolling it out everywhere.

The analytics lead pointed out that the joiners were not a random draw. Larger, faster-growing stores in growing suburbs had signed up first, and they would have grown faster anyway. She modeled which stores joined from their size, region, and growth in the two prior years, giving every store a score: its estimated chance of joining. Then she paired each joiner with the nonjoining store whose score was closest.

The pairs looked alike on everything the model used, and pair by pair the joiners grew about three points faster. The program still paid for itself at three points, but the rollout was limited to stores resembling the four joiners, and the rest were held out as next year's comparison.

Why it works

When units choose a treatment for themselves, the ones that choose it usually differ from the ones that do not, and those differences drive the outcome too. A raw comparison credits the treatment with all of it. Matching on every characteristic at once is impossible with many characteristics and few units. The propensity score solves that by collapsing them into one number, the estimated chance of being treated given the characteristics. Units with the same score are, on average, balanced on everything the score was built from, so comparing them is close to comparing randomly assigned groups on those characteristics.

The limit is in that last phrase. Matching balances only what was measured. A difference no one recorded, such as a manager's energy, can still bias the result, so the estimate is only as good as the list of characteristics behind the score.

The thinking behind it

Paul R. Rosenbaum Paul R. Rosenbaum, a Wharton statistician of causal inference in observational studies, introduced the propensity score with Donald B. Rubin in The Central Role of the Propensity Score in Observational Studies for Causal Effects (1983), in Biometrika. They defined it as the probability of receiving a treatment given observed covariates, and proposed it as the basis for matching treated and untreated units when no experiment is possible, the situation of most business decisions made after the fact.

Donald B. Rubin Donald B. Rubin, a Harvard statistician, is known for the potential outcomes framework for causal inference, often called the Rubin causal model, which defines a treatment's effect as the difference between what happens to a unit with the treatment and what would have happened without it. In the 1983 Biometrika paper with Paul Rosenbaum, he showed that comparing treated and untreated units with the same propensity score removes the bias that comes from the covariates the score was built on.

How to run it

  1. List what drove the choice. Write down every characteristic that plausibly affected both who got the treatment and the outcome, measured before the treatment.
  2. Fit the score. Model the chance of treatment from those characteristics, usually with a logistic regression.
  3. Check the overlap. Confirm that treated and untreated units share a range of scores; units outside it have no fair match.
  4. Match on the score. Pair each treated unit with the untreated unit whose score is closest, within a set maximum gap.
  5. Check the balance. Compare the matched groups on each characteristic; if they still differ, refit the score and match again.
  6. Estimate the effect. Average the outcome differences across the matched pairs and set it beside the raw comparison.
  7. Test what was not measured. Ask how strong an unrecorded difference would have to be to erase the effect, and whether one that strong is plausible.

Where it goes wrong

The score leaves out what mattered. A model of store size and region misses the manager's ambition, which drove both joining and growth; spot it when people who know the units can name a reason for the choice that is not in the data. Add the measurable proxies, and treat the result as an upper bound.

The characteristics are measured after the treatment. Using growth during the program year to build the score folds part of the effect into the match. Use only data from before the choice.

Poor matches are kept. When no close match exists, the pair adds bias. Set a maximum gap, drop the unmatched, and say how many were dropped.

When to use something else

Where it sits in SOLVE eX

Go to the source

  • The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika, 1983

Back to the catalog