Superforecasting

Thinking Tools3.4 Facts About the FutureAlso known as Good Judgment Methods

Developed by Philip E. Tetlock and written up with Dan Gardner (Superforecasting: The Art and Science of Prediction, 2015).

A set of forecasting habits (base rates first, questions broken into parts, many viewpoints, numeric probabilities, and frequent small updates) that measurably improves forecasts.

One deal, four pieces of news, and how far each should move the number

The question is whether a rival's purchase of a regional carrier closes by June 30. At each of the forecaster's turns pick a move: the bar is the forecast, and the tag says whether the move fits the evidence. Illustrative.

The news

Forecaster

The news

Forecaster

Select a stage to read it, or play the sequence.The question. Both boards approved the deal in January. The question is written so it can be scored: yes if the purchase closes on or before June 30, no otherwise.First number. The bar starts at 50. Pick the forecaster's opening number.Pick a move to see the reply.The forecast sits at 50.A shrug: it uses nothing anyone knows about deals like this one.The forecast moves to 70.The outside view first: a base rate from a class of similar deals.The forecast moves to 95.The inside view alone: every deal in the class had board approval, so it says little about timing.March news. The competition regulator issues a second request for information, which usually adds months to the review.First update. A second request for information has arrived. Pick the forecaster's update.Pick a move to see the reply.The forecast drops 40 points.An overreaction: most deals that get a second request still close, only later.The forecast drops 15 points.Sized to the evidence: it hurts the date more than the deal.The forecast stays where it was.An underreaction: the question is about the date, and this news is about the date.April news. The buyer offers to sell the one depot where the two networks overlap, the remedy regulators usually ask for in a case like this.Second update. The buyer has offered the usual remedy. Pick the forecaster's update.Pick a move to see the reply.The forecast rises 10 points.A step up of the right size: the remedy helps, and the review clock still runs.The forecast rises 35 points.An overreaction: an offer is not an approval, and the date is still tight.The forecast stays where it was.An underreaction: news that cuts the other way counts too.May news. A trade blog cites one unnamed source saying the deal is in trouble. Neither company comments.Third update. One unnamed source says the deal is in trouble. Pick the forecaster's update.Pick a move to see the reply.The forecast drops 30 points.A big move on one anonymous source.The forecast drops 5 points.Small, because the evidence is thin; a second source would earn a bigger move.The answer. The deal closed two weeks after the deadline, so the question resolves no. The careful path ends at 60, which scores far better than 95 would have; scored over many questions, the habit shows whether a forecaster's 60s come true six times in ten.

Moves sized to the evidence: 0 of 4Moves sized to the evidence: 1 of 4Moves sized to the evidence: 2 of 4Moves sized to the evidence: 3 of 4Moves sized to the evidence: 4 of 4

Forecast, in percent50of 100

  1. The question (The news). Both boards approved the deal in January. The question is written so it can be scored: yes if the purchase closes on or before June 30, no otherwise.
  2. First number (Forecaster). The bar starts at 50. Pick the forecaster's opening number.
    • Start at 50: it could go either way. The forecast sits at 50. A shrug: it uses nothing anyone knows about deals like this one.
    • Start at 70: about seven in ten deals this size close within six months of signing. The forecast moves to 70. The outside view first: a base rate from a class of similar deals.
    • Start at 95: both boards have already approved it. The forecast moves to 95. The inside view alone: every deal in the class had board approval, so it says little about timing.
  3. March news (The news). The competition regulator issues a second request for information, which usually adds months to the review.
  4. First update (Forecaster). A second request for information has arrived. Pick the forecaster's update.
    • Cut by 40 points: the regulator means to block it. The forecast drops 40 points. An overreaction: most deals that get a second request still close, only later.
    • Cut by 15 points: a second request usually adds three to six months. The forecast drops 15 points. Sized to the evidence: it hurts the date more than the deal.
    • Hold: deals like this usually close. The forecast stays where it was. An underreaction: the question is about the date, and this news is about the date.
  5. April news (The news). The buyer offers to sell the one depot where the two networks overlap, the remedy regulators usually ask for in a case like this.
  6. Second update (Forecaster). The buyer has offered the usual remedy. Pick the forecaster's update.
    • Raise by 10 points: the main objection now has an answer. The forecast rises 10 points. A step up of the right size: the remedy helps, and the review clock still runs.
    • Raise by 35 points: it's as good as done. The forecast rises 35 points. An overreaction: an offer is not an approval, and the date is still tight.
    • Hold until the regulator says yes. The forecast stays where it was. An underreaction: news that cuts the other way counts too.
  7. May news (The news). A trade blog cites one unnamed source saying the deal is in trouble. Neither company comments.
  8. Third update (Forecaster). One unnamed source says the deal is in trouble. Pick the forecaster's update.
    • Cut by 30 points: the deal is in trouble. The forecast drops 30 points. A big move on one anonymous source.
    • Cut by 5 points, and look for a second source. The forecast drops 5 points. Small, because the evidence is thin; a second source would earn a bigger move.
  9. The answer (The news). The deal closed two weeks after the deadline, so the question resolves no. The careful path ends at 60, which scores far better than 95 would have; scored over many questions, the habit shows whether a forecaster's 60s come true six times in ten.

An example

A freight company's strategy team had to decide when to cut prices on its northern routes, and the answer turned on whether a rival's purchase of a regional carrier would close by June 30. The first draft of the memo said the deal was certain, since both boards had approved it.

The team lead asked for a number instead, built in order. Of the deals of similar size the team could find, about seven in ten had closed within six months of signing, so the forecast opened at 70 percent. When the regulator asked for more information in March, the team cut it to 55; when the buyer offered to sell the one overlapping depot, they raised it to 65; a blog rumor in May moved it to 60. The memo went out with 60 percent and a plan for both cases. The deal closed in mid-July, and the price cut, timed for the later case, landed the week the rival integrated.

Why it works

Most forecasts fail in one of two ways: they ignore how often things like this usually happen, or they swing hard on every headline. Starting from a base rate anchors the number in a record of similar cases, and adjusting for what makes this case different brings in the specifics without letting them take over. Breaking the question into its parts (will the regulator approve, and by when) turns one vague guess into several smaller ones that can each be checked.

Small, frequent updates keep the number close to the evidence as it arrives, and stating it as a percentage makes it possible to score. A forecaster who is scored across many questions learns whether their 70s happen seven times in ten, and that feedback is what makes the habits improve with practice.

The thinking behind it

Philip E. Tetlock Philip E. Tetlock is a political psychologist who studies forecasting and judgment. His Good Judgment Project entered the forecasting tournament that IARPA, the research agency of the US intelligence community, ran from 2011, in which teams of volunteers forecast world events and were scored against what happened. Superforecasting: The Art and Science of Prediction (2015), written with Dan Gardner, sets out the habits that marked the project's most accurate forecasters.

Dan Gardner Dan Gardner, a Canadian journalist and author, wrote Superforecasting with Tetlock (Crown, 2015), the popular account of the Good Judgment Project's findings: who the volunteers that forecast world events most accurately were, and the habits that set them apart from the rest. The research is Tetlock's; Gardner is the coauthor of its popular telling, written for readers outside the field of forecasting research.

How to run it

  1. Write a question that can be scored. Give it a date and a rule for yes and no that two people would read the same way.
  2. Start from a base rate. Find a class of similar cases and ask how often the event happened in it.
  3. Break the question into its parts. Name the two or three things that must happen for a yes, and put a number on each.
  4. Adjust for what makes this case different. Move off the base rate in modest steps, and write down the reason for each one.
  5. State a percentage. Use the whole scale; 60 and 70 mean different things, and the difference is what gets scored.
  6. Update in small steps as news arrives. Ask how much more likely the news would be in a yes world than in a no world, and move by about that much.
  7. Score the forecast when it resolves. Keep a record across many questions so you can see whether your numbers come true as often as they claim.

Where it goes wrong

The base rate comes from the wrong class. A class picked to fit the case, or too small to mean anything, just restates the inside view with a number on it. Try two or three classes and see how far apart they are.

The number never moves. A forecaster who anchors on the first figure treats every piece of news as noise. Write down in advance what news would move you, and by how much.

The questions are never scored. Without a record, a forecaster cannot tell good judgment from a lucky year. Score every resolved question, and look hardest at the confident misses.

When to use something else

  • Reference Class Forecasting. When the decision needs only an outside-view number from a class of similar cases, and no tracking over time.
  • Bayesian Updating. When the evidence can be put in numbers and the update should be computed from them.
  • Brier Score. When the job is to score a record of past forecasts and see where they ran too confident.

Where it sits in SOLVE eX

Go to the source

  • Superforecasting: The Art and Science of Prediction, 2015

Back to the catalog