Developed by Sarah Lichtenstein and Baruch Fischhoff (Training for Calibration, 1980) with Lawrence D. Phillips (Calibration of Probabilities, 1982), brought to business by Douglas W. Hubbard (How to Measure Anything, 2007), and carried into forecasting by Philip E. Tetlock and Dan Gardner (Superforecasting, 2015).
Drills people on questions with known answers, scoring how often their 90 percent ranges contain the truth, until stated confidence matches the actual hit rate.
Six questions with known answers: pick a range that is 90 percent sure, then count how many it caught
Play the estimator: for each question, choose the range that would hold the answer nine times in ten. The true answer and whether the range caught it appear at once, and the count runs under the panel. Calibrated ranges catch about nine in ten; most people's first ranges catch fewer, because they are narrower than the confidence they claim. The questions and answers are real.
The coach
The estimator
The coach
The estimator
Ranges that caught the answer: 0 of 6Ranges that caught the answer: 1 of 6Ranges that caught the answer: 2 of 6Ranges that caught the answer: 3 of 6Ranges that caught the answer: 4 of 6Ranges that caught the answer: 5 of 6Ranges that caught the answer: 6 of 6
- The rule (The coach). For each question, the estimator picks the range that is 90 percent likely to hold the answer. Before choosing, compare the range with a bet: win $1,000 if the answer is in the range, or win it on a wheel that pays nine times in ten. Preferring the wheel means the range is too narrow.
- Piano keys (The estimator). A standard modern piano, black and white keys together.
- 90 percent sure: 80 to 86 The answer is 88. Missed: the answer is outside the range.
- 90 percent sure: 76 to 92 The answer is 88. Caught: the answer is inside the range.
- 90 percent sure: 60 to 120 The answer is 88. Caught, but a range this wide tells a planner almost nothing.
- Bones (The estimator). The usual count for an adult skeleton.
- 90 percent sure: 180 to 200 The answer is 206. Missed: the answer is outside the range.
- 90 percent sure: 150 to 250 The answer is 206. Caught: the answer is inside the range.
- 90 percent sure: 100 to 400 The answer is 206. Caught, but a range this wide tells a planner almost nothing.
- Panama Canal (The estimator). The year it opened to shipping.
- 90 percent sure: 1910 to 1920 The answer is 1914. Caught: the answer is inside the range.
- 90 percent sure: 1900 to 1925 The answer is 1914. Caught: the answer is inside the range.
- 90 percent sure: 1880 to 1950 The answer is 1914. Caught, but a range this wide tells a planner almost nothing.
- Earth to Moon (The estimator). The average over the Moon's orbit.
- 90 percent sure: 190,000 to 210,000 The answer is about 238,900 miles. Missed: the answer is outside the range.
- 90 percent sure: 150,000 to 230,000 The answer is about 238,900 miles. Missed: the answer is outside the range.
- 90 percent sure: 100,000 to 500,000 The answer is about 238,900 miles. Caught: only the wide range held it, a sign of how far off a first guess can be.
- Golden Gate Bridge (The estimator). The year it opened to traffic.
- 90 percent sure: 1930 to 1940 The answer is 1937. Caught: the answer is inside the range.
- 90 percent sure: 1925 to 1945 The answer is 1937. Caught: the answer is inside the range.
- 90 percent sure: 1900 to 1960 The answer is 1937. Caught, but a range this wide tells a planner almost nothing.
- UN members (The estimator). Full members, as of 2024.
- 90 percent sure: 150 to 180 The answer is 193. Missed: the answer is outside the range.
- 90 percent sure: 170 to 210 The answer is 193. Caught: the answer is inside the range.
- 90 percent sure: 120 to 250 The answer is 193. Caught, but a range this wide tells a planner almost nothing.
- The score (The coach). Six ranges at 90 percent should catch five or six answers. Fewer means the ranges were narrower than the confidence they claimed; the fix is to widen them on the next round of fresh questions until the hit rate matches.
An example
A pharmaceutical company's portfolio team asked each project lead for a 90 percent range on launch timing and peak sales, and fed the ranges into its investment model. A review of five years of past ranges found that the true outcome had landed inside them about half the time. The model was treating guesses as if they were twice as reliable as they were.
Before the next planning cycle, the head of portfolio strategy ran a two-hour calibration session for the twenty project leads. Each answered thirty questions with known numeric answers by giving a 90 percent range, then scored themselves. The average hit rate was 52 percent. The group learned the equivalent bet: picture winning $1,000 if the answer is in the range, or on a wheel that pays nine times in ten, and widen the range until the two feel the same. A second round of fresh questions came in at 71 percent, a third at 84. The leads then redid their project ranges, which came back noticeably wider, and the model's risk numbers rose.
Why it works
Almost everyone is overconfident in the same direction: ranges meant to hold the answer nine times in ten hold it far less often. The error is invisible in everyday work because the true answers come back slowly, one at a time, and are rarely compared with the range that was given. Without feedback, nobody learns.
The drill supplies the missing feedback in an hour. Questions with known answers resolve instantly, a batch of them gives a hit rate, and the gap between the confidence claimed and the hit rate achieved is undeniable. Repeated with fresh questions, the gap narrows, because people learn what 90 percent sure actually feels like. The equivalent bet helps by turning an abstract confidence into a comparison people can feel. The payoff is estimates whose stated uncertainty means what it says, which every model built on them needs.
The thinking behind it
Sarah Lichtenstein Sarah Lichtenstein and Baruch Fischhoff showed in Training for Calibration (1980) that people given scored practice and feedback on the confidence of their answers became better calibrated, so that their stated probabilities came closer to how often they were right. With Lawrence D. Phillips they surveyed the research to date in Calibration of Probabilities (1982), on how closely people's stated probabilities match how often they are right. Later reviews find that training improves calibration only to a limited extent.
Douglas W. Hubbard Douglas W. Hubbard, a consultant in decision analysis, brought calibration drills to business in How to Measure Anything: Finding the Value of Intangibles in Business (2007). He argues that anything that matters can be measured in the sense of reducing uncertainty about it, and that the first step is training the estimators, so their intervals contain the truth as often as they claim.
Philip E. Tetlock Philip E. Tetlock, with Dan Gardner, described in Superforecasting (2015) the habits of the most accurate forecasters in the Good Judgment Project, which competed from 2011 in a forecasting tournament run by the US intelligence research agency. Calibration, saying 70 percent only about things that happen 70 percent of the time, is among the habits the book describes in the best of them; the book carried the idea from the laboratory into working forecasting.
How to run it
- Assemble questions with known answers. Use twenty to forty numeric questions the group cannot look up, mixing general knowledge and facts from their own field.
- Ask for 90 percent ranges. Each person writes a low and a high value they are 90 percent sure contains the answer.
- Score the hit rate. Reveal the answers and have each person count how many ranges caught the truth.
- Teach the equivalent bet. Compare each range with a wheel that pays nine times in ten, and widen the range until the two bets feel equal.
- Repeat with fresh questions. Run new rounds until each person's hit rate is close to 90 percent.
- Apply it to real estimates at once. Have people redo their working ranges for plans and models while the feeling of 90 percent is fresh.
Where it goes wrong
The common failure is a one-off session. Hit rates improve in the room and drift back within months, because real estimates still get no feedback. It shows up as new ranges as narrow as the old ones by the next planning cycle. Score real forecasts as they resolve and repeat short drills each cycle.
A second failure is overcorrecting into useless width. Some people learn to give ranges so wide they always hit, which says nothing. Check that ranges hit about nine in ten, not ten in ten, and score both. A third is drills that do not transfer: trivia trains the feeling of confidence, but people also need practice on questions from their own work.
When to use something else
- Brier Score. When the forecasts are yes-or-no probabilities that resolve over time, and the job is scoring a record.
- Words of Estimative Probability. When the problem is that vague words such as likely mean different numbers to different people.
- Fermi Estimation. When the estimate has no reference point and must be built from rough parts before any range is set.
Where it sits in SOLVE eX
- Any step: Tools that serve every step: habits of thought and checks on the thinking itself.
Go to the source
- Training for Calibration, Organizational Behavior and Human Performance 26, 1980
- Calibration of Probabilities: The State of the Art to 1980, Judgment under Uncertainty: Heuristics and Biases, 1982
- How to Measure Anything: Finding the Value of Intangibles in Business, 2007
- Superforecasting: The Art and Science of Prediction, 2015