First used by John Snow on London's cholera (On the Mode of Communication of Cholera, 1855), given its modern form by Orley Ashenfelter and David Card (1985), made famous by Card and Alan B. Krueger (1994), and set out in standard form by Joshua D. Angrist and Jörn-Steffen Pischke (Mostly Harmless Econometrics, 2009).
Estimates the effect of a change by comparing the before and after shift in a group that got it with the shift in a similar group that did not.
The new layout's effect is what the test stores gained beyond what the other stores gained anyway
The square plots the test stores' change in sales against the change in similar stores that kept the old layout. The line is where the test stores would sit if the layout did nothing: the same change as the others, plus any gap in trend the two groups already had before. The effect is the square's height above the line. Raise the other stores' change and watch a strong-looking result shrink. Illustrative.
The test saysThe layout added sales beyond the change every store saw.No lift beyond the market: the other stores' change explains it all.
An example
A home goods chain refit twelve stores with a new floor layout in March, and by June their weekly sales were up 9 percent on the winter. The merchandising team took the 9 percent to the board as the case for refitting forty more stores at $180,000 each.
The finance analyst asked what the other stores had done over the same months. Thirty stores of similar size and region, still on the old layout, were up 6 percent: spring and a rival's closure had lifted everyone. She also pulled the year before the refit and found the two groups had grown in step, within half a point of each other each quarter, so the comparison stores were a fair stand-in for what the test stores would have done.
The layout's effect was the difference between the two changes, 3 points. At 3 points the refit still paid back, in about four years instead of sixteen months. The board approved twenty stores, with the next twenty to follow if the gap held through the holidays.
Why it works
A before and after comparison mixes the change being judged with everything else that happened over the same weeks: the season, the economy, a competitor's move. A comparison between stores that got the change and stores that did not mixes it with whatever already made those stores different. Taking the difference of the two before and after changes removes both. The comparison group's change stands in for the time trend, and subtracting each group's own before period removes the fixed differences between them.
The whole method rests on one assumption: without the change, the two groups would have moved together. It cannot be proved, but it can be tested on the periods before the change. Groups that tracked each other for a year are a credible comparison; groups that were already drifting apart are not.
The thinking behind it
John Snow John Snow, physician and epidemiologist, argued in the second edition of On the Mode of Communication of Cholera (1855) that cholera spreads through contaminated water. He compared London households supplied by the Southwark and Vauxhall company, which drew its water from the sewage-polluted Thames, with households supplied by the Lambeth company, which had moved its intake upriver, and found far higher death rates among the first. Calling that comparison a difference in differences is a later economists' reading; Snow did not set out the estimator.
David Card David Card, a labor economist at Berkeley, gave the comparison its modern use. With Orley Ashenfelter he compared the earnings of trainees and nontrainees before and after job training in Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs (1985), and showed how sensitive such estimates are to how people come to enter a program. With Alan B. Krueger he compared fast-food employment in New Jersey, which raised its minimum wage, with Pennsylvania, which did not (1994). That study's finding was disputed; the design is widely taught. Card shared the 2021 Nobel Memorial Prize in Economic Sciences.
Joshua D. Angrist Joshua D. Angrist, an MIT labor economist, wrote with Jörn-Steffen Pischke of the London School of Economics Mostly Harmless Econometrics (2009), the standard practitioner's guide to research designs that draw causal conclusions without a randomized experiment: instrumental variables, regression discontinuity, and difference in differences. Angrist shared the 2021 Nobel Memorial Prize in Economic Sciences for methods of drawing causal conclusions from natural experiments.
How to run it
- Name the change and its date. Fix which units got it and exactly when, so before and after are unambiguous.
- Pick a comparison group. Choose units that did not get the change and resemble the treated ones in size, region, and customers.
- Check the trends before. Plot both groups over several periods before the change and confirm they moved together.
- Measure both changes. Take each group's after period minus its own before period.
- Subtract. The treated group's change minus the comparison group's change is the estimated effect.
- Test the result's footing. Rerun with a different comparison group or a fake change date and confirm the effect appears only where it should.
- Decide on the estimate. Rebuild the business case on the difference in differences and note the assumption it rests on.
Where it goes wrong
The groups were never parallel. Test stores chosen because they were growing fastest will keep outgrowing the rest, and the method credits the change with their momentum; spot it when the pre-period lines diverge. Pick the comparison group before looking at results, or adjust for the pre-existing gap.
Something else hit one group. A local road closure or a new manager at the test stores lands in the estimate. List every event that touched only one group during the window.
The change leaked into the comparison. Staff moved between stores, or customers shifted from comparison stores to the refitted ones, so the comparison group's change was not untouched. Check for spillover, and choose comparison units far enough away that it cannot happen.
When to use something else
- Randomized Controlled Trial. When the change has not happened yet and the units can be assigned to it at random.
- Propensity Score Matching. When no before period exists and the treated units must be paired with similar untreated ones.
- Causal Diagram. When the first need is to map which variables confound the comparison before choosing a design.
Where it sits in SOLVE eX
Go to the source
- On the Mode of Communication of Cholera, second edition, 1855
- Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs, Review of Economics and Statistics 67, no. 4, 1985
- Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, American Economic Review, 1994
- Mostly Harmless Econometrics, 2009