Discovered by Francis Galton (Regression Towards Mediocrity in Hereditary Stature, 1886) and set out for decision makers by Daniel Kahneman (Thinking, Fast and Slow, 2011).
Expects extreme results to be followed by less extreme ones, because part of any extreme is luck, and luck does not repeat on schedule.
The top ten sales reps of the quarter, and what to expect from them next quarter
The top ten of 200 reps averaged 140 percent of quota this quarter. Set how closely one quarter's results predict the next, measured from past quarters; the expected result for the next three quarters falls back by the share that was luck, and then stays there. Illustrative.
Top ten, expected % of quotaIf the lead were all skillCompany average
An example
A software company's sales leadership reviewed the quarter: the top ten of its 200 account executives averaged 140 percent of quota, the bottom ten 60 percent, and the whole team 100 percent. The top ten won a trip, and the bottom ten were put on performance plans. Next quarter the top ten averaged 117 percent and the bottom ten 85, and the sales VP concluded that the plans worked and the trip had made the winners complacent.
The sales operations lead checked the history. Across the last eight quarters, a rep's result predicted the next quarter's with a correlation of about 0.4, so about 40 percent of a standout quarter's distance from the average repeats and the rest was luck in which deals closed when. On that number alone the top ten should have been expected at 116 percent and the bottom ten at 84, almost exactly what happened. The company kept the plans but judged them against the 84 percent expected without them, and based the trip on a two-quarter average, which carries less luck.
Why it works
Any result that is part skill and part luck will, at its extremes, be largely luck. Of 200 reps, the top ten this quarter are the ones whose skill and luck both ran high, and luck does not carry over. Next quarter their skill stays and their luck is average, so their results fall back toward the middle without anyone doing anything. The bottom ten rise for the same reason.
The size of the fall is set by how much the measure repeats. When one quarter predicts the next with a correlation of 0.4, the best estimate keeps 40 percent of any lead and gives back the rest. Shrinking forecasts by that share stops a company from crediting a program, or blaming a person, for a change chance alone would have produced.
The thinking behind it
Francis Galton Francis Galton found the effect in the heights of parents and children, reported in 1886 in the Journal of the Anthropological Institute as Regression Towards Mediocrity in Hereditary Stature. Children of unusually tall or short parents were, on average, closer to the mean than their parents, which he called regression towards mediocrity and which is now called regression to the mean. The same arithmetic holds for any measure that is part stable and part chance, a sales quarter among them.
Daniel Kahneman Daniel Kahneman, with Amos Tversky, showed in On the Psychology of Prediction (Psychological Review, 1973) that intuitive predictions follow representativeness and neglect base rates and regression to the mean. In Intuitive Prediction: Biases and Corrective Procedures (1979) they proposed ways to correct such forecasts, starting from the record of similar cases, and Thinking, Fast and Slow (2011) set out regression to the mean for decision makers.
How to run it
- Name the extreme and its measure. A record quarter, a slump, a top-rated hire, or a store that suddenly improved.
- Measure how much the measure repeats. Correlate each period's result with the next across past periods; that number is the share that carries over.
- Shrink the forecast toward the average. Expected next result is the average plus the correlation times the distance from it.
- Give no credit for the return. An intervention made after an extreme counts only if the result beats the shrunk forecast.
- Reward on longer windows. An average over two or more periods carries less luck than a single one.
- Compare against a control. Where possible, track similar extremes that received no intervention.
- Discount personal highs and lows too. A brilliant month or an awful one says less about the next month than it feels.
Where it goes wrong
The classic failure is crediting the intervention. The worst performers get a plan and improve, and the plan gets the credit; the best get praise and slip, and the praise gets the blame. Spot it when every program aimed at the bottom seems to work and every reward at the top seems to backfire, and recover by judging each one against the shrunk forecast.
The second is shrinking without measuring. A correlation guessed at 0.2 when it is really 0.8 writes off real skill, so compute it from past periods. The third is treating regression as fate. The top performers really are better on average, by the share that repeats, and a company that ignores that loses its best people to firms that noticed.
When to use something else
- Reference Class Forecasting. When the forecast should start from how similar cases usually turn out, before any individual record.
- Goodhart-Aware Metric Selection. When the extreme results come from a metric people are paid on, and gaming may be part of the lead.
- Control Chart. When the question is whether a process has really changed or is showing ordinary variation.
Where it sits in SOLVE eX
Go to the source
- Regression Towards Mediocrity in Hereditary Stature, Journal of the Anthropological Institute 15, 1886
- Thinking, Fast and Slow, chapter 17, 2011