Reference Class Forecasting

Thinking Tools3.4 Facts About the FutureAlso known as Base-Rate Reasoning, Outside View

From Daniel Kahneman and Amos Tversky (On the Psychology of Prediction, 1973; Intuitive Prediction: Biases and Corrective Procedures, 1979), developed as the outside view by Kahneman and Dan Lovallo (Timid Choices and Bold Forecasts, 1993) and as reference class forecasting by Bent Flyvbjerg (2006).

Starts any forecast from how a class of similar past cases actually turned out, then adjusts only for differences that have evidence behind them.

An ERP rollout's schedule, anchored on how twenty similar rollouts actually went

The bars are the record of twenty comparable rollouts: the middle half took 11 to 19 months, the whole set 8 to 30. The forecast starts at the class median of 14 and moves only by the adjustment the evidence supports; here, minus 10 percent because the team has done one rollout before. The spread then sets the reserve: enough to reach 19 months, when three rollouts in four had finished. Push the adjustment until the forecast meets the team's plan of 9 and see how far below the middle half it sits. Illustrative.

9
14
−10%
Forecast, starting from the class
12.6 months
Contingency to cover three rollouts in four
6.4 months
Forecast minus the team's plan
3.6 months
Middle half of the twentyAll twenty past rollouts0 mo10 mo20 mo30 moTeam's planForecastMiddle half of the twentyAll twenty past rollouts0 mo10 mo20 mo30 moTeam's planForecast

An example

A specialty food distributor was replacing its order and finance systems with a new ERP. The implementation team, the vendor, and the CFO agreed on nine months, built from a detailed task plan, and the board was ready to approve the date.

A board member asked the CFO to try one more estimate first. The finance team pulled the record of twenty comparable rollouts at distributors of similar size, from vendor references and an industry group. The median had taken 14 months; the middle half took 11 to 19; the slowest took 30. Only one had finished in nine months or less. The team then asked what made this project different, with evidence: the company had done one ERP rollout before, with several of the same people, which justified about 10 percent off. Nothing else on the list of differences held up; most were reasons every team gives.

The forecast came to about 12.6 months. The board approved a 13-month plan with a contingency for the 19 months the record said one project in four exceeded.

Why it works

A team planning its own project builds the estimate from the inside: the tasks, the people, the plan. Every step looks doable, and the things that actually sink projects, the unplanned delays, rework, departures, and vendor slips, are invisible from there because nobody can list them in advance. The inside view is reliably too optimistic.

The outside view sidesteps the list. A class of similar past projects already contains every kind of trouble, in roughly the proportions it happens, so its record is a better starting point than any plan. Starting from the class median and adjusting only for differences that have evidence behind them keeps the forecast anchored to what actually happens. The rule that adjustments need evidence matters as much as the anchor: without it, teams argue their way back to the plan one special reason at a time.

The thinking behind it

Daniel Kahneman Daniel Kahneman and Amos Tversky showed in On the Psychology of Prediction (1973) that intuitive predictions follow how well a case resembles a category and neglect base rates. In Intuitive Prediction: Biases and Corrective Procedures (1979) they separated singular information about the case at hand from distributional information about similar cases, and proposed starting forecasts from the distribution. That proposal is the outside view in its first form.

Dan Lovallo Dan Lovallo, a scholar of strategy and decisions, named the inside view and the outside view of forecasting with Daniel Kahneman in Timid Choices and Bold Forecasts (1993), in Management Science. They argued that planners should anchor estimates on the outcomes of a reference class of similar past projects. In Delusions of Success (2003), in the Harvard Business Review, the two carried the argument to executives.

Bent Flyvbjerg Bent Flyvbjerg, a scholar of megaprojects at Oxford and Copenhagen, turned the outside view into reference class forecasting for large projects in From Nobel Prize to Project Management: Getting Risks Right (2006). He traced cost and schedule overruns to optimism bias and strategic misrepresentation, and corrected forecasts with the record of similar past projects in place of the planners' own estimates.

How to run it

  1. Define exactly what is forecast. Name the measure and its end point, such as months from contract to go-live on the core modules.
  2. Find a reference class. Gather at least ten past cases similar enough to compare, from the company's history, vendors, peers, or industry data.
  3. Record the class outcomes. Write down the median, the middle half, and the worst cases, without trimming the ones that look unusual.
  4. Anchor at the class median. Start the forecast there, whatever the plan says.
  5. Adjust only for evidenced differences. Move the forecast for each real difference that has proof behind it, by a justified amount, and reject the rest.
  6. Set the plan and the contingency from the spread. Plan near the forecast and hold reserves for the outcomes the class says happen one time in four or five.
  7. Write down the class and each adjustment. Keep the record so the forecast can be checked later and the next project starts from a better class.

Where it goes wrong

The common failure is a class built to flatter. The team picks only the past projects that went well, or defines the class so narrowly that its own plan looks typical. It shows up as a reference class that agrees with the inside view. Choose the class before looking at its results, and include the failures.

A second failure is death by adjustment: every difference is claimed as a reason to move the forecast back toward the plan. Demand evidence for each and cap the total. A third is ignoring the tail. The median sets the plan, but the spread sets the contingency, and a plan with no reserve for the slow quarter of outcomes will be late in the way the class predicted.

When to use something else

  • Outside View. When the team's own estimate rests on evidence with a track record, and the question is how far to move toward it: blend the two views by how well such estimates have predicted outcomes.
  • Monte Carlo Simulation. When the forecast must be built from many uncertain parts and their ranges, and no clean class of whole projects exists.
  • Fermi Estimation. When no record of similar cases exists and the number must be built from rough parts multiplied together.

Where it sits in SOLVE eX

Go to the source

  • On the Psychology of Prediction, Psychological Review, 1973
  • Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking, Management Science, 1993
  • From Nobel Prize to Project Management: Getting Risks Right, Project Management Journal, 2006
  • Intuitive Prediction: Biases and Corrective Procedures, TIMS Studies in Management Science 12, 1979
  • Delusions of Success: How Optimism Undermines Executives' Decisions, Harvard Business Review, 2003

Back to the catalog