Brier Score

Thinking Tools6.4 Conducting Follow-Up and Taking Feedback

Developed by Glenn W. Brier (Verification of Forecasts Expressed in Terms of Probability, 1950), with the decomposition by Allan H. Murphy (A New Vector Partition of the Probability Score, 1973).

Scores probability forecasts by the squared gap between the stated chance and what happened, rewarding honest calibration and splitting skill into reliability and resolution.

Five forecasts that have resolved, each scored by its squared gap from what happened

A sales operations lead gave each question a chance last year; all five have now resolved, as the slider labels say. Each bar is that forecast's penalty: the gap between the chance and the outcome (1 if it happened, 0 if not), squared. The average is the Brier score: 0 is perfect, and the line at 0.25 is what saying 50 percent to everything earns. Move a forecast toward the truth and its bar shrinks; note what one confident miss does to the average. Illustrative.

90%
70%
80%
60%
30%
Brier score, the average penalty
0.20
Score for always saying 50%
0.25
Largest single penalty
0.64

Penalty on each forecastAlways saying 50%

0.000.250.500.751.000.010.090.640.160.09Harbor dealPrice riseWest regionNew directorRival's launch0.000.250.500.751.000.010.090.640.160.09HarbordealPriceriseWestregionNewdirectorRival'slaunch

An example

A software company's revenue team made dozens of probability calls each quarter: will this deal close by the end of the month, will this renewal churn, will the new region hit target. Leaders argued about who was a good forecaster, and the argument always came back to whoever had made the most memorable call.

The head of sales operations started logging every forecast with a date, a chance, and a rule for how the question would resolve. After two quarters she scored each person's resolved forecasts with the Brier score. The team average was 0.19, against 0.25 for someone who said 50 percent to everything. Two account directors who sounded confident in every meeting scored 0.27, worse than the coin. One quiet regional manager scored 0.11. Splitting the scores showed the directors' problem was calibration: their 90 percent calls came true about 60 percent of the time.

The company now weights each forecaster's deal probabilities by their past score when it builds the quarter's number, and shares each person's score with them every month.

Why it works

Judging a single probability forecast by its outcome is close to meaningless: a 70 percent call that misses may have been right. Over many forecasts, though, the record can be scored, and the Brier score does it with one simple rule. Each forecast's penalty is the squared gap between the stated chance and what happened, 1 or 0, and the score is the average.

Squaring makes the score proper: a forecaster minimizes expected penalty only by stating the probability they actually believe, so hedging and exaggerating both cost points. It also punishes confident misses heavily, so a single 90 percent call that fails costs as much as several small errors. Comparing with the always-50-percent score of 0.25, or with the base rate, shows whether a forecaster adds anything, and the decomposition says whether the gap comes from poor calibration or from failing to tell likely events from unlikely ones.

The thinking behind it

Glenn W. Brier Glenn W. Brier, a meteorologist and statistician, proposed the score in a 1950 paper in the Monthly Weather Review on how to verify weather forecasts stated as chances. His measure is the mean squared difference between the probabilities a forecaster gave and what happened, each outcome counted as 1 or 0. It rewards forecasters who are both confident and right, and the measure still carries his name: the Brier score.

Allan H. Murphy Allan H. Murphy, a scholar of weather forecast verification, split the Brier score into three parts in A New Vector Partition of the Probability Score (1973): reliability, how closely stated probabilities match observed frequencies; resolution, how well the forecasts separate events that happen from events that do not; and the uncertainty of the events themselves. The partition tells a forecaster what to fix as well as how they scored.

How to run it

  1. Write questions that resolve cleanly. Each needs a yes-or-no outcome, a resolution date, and a rule anyone could apply to settle it.
  2. Record the probability at the time. Log the chance, the forecaster, and the date before the outcome is known, in a place nobody edits later.
  3. Record the outcome as 1 or 0. At the resolution date, mark 1 if it happened and 0 if it did not.
  4. Score each forecast. Square the difference between the probability and the outcome; 90 percent on an event that happened scores 0.01.
  5. Average and compare. Average across each person's forecasts and compare with 0.25 for always saying 50 percent, and with a forecaster who always gives the base rate.
  6. Decompose the score. Check calibration by grouping forecasts by stated chance and comparing with how often each group came true.
  7. Feed the scores back. Share each person's score regularly and use it to weight their forecasts in combined numbers.

Where it goes wrong

The common failure is scoring too few forecasts. A handful of resolved questions produces scores dominated by luck, and people get labeled good or bad on noise. It shows up as rankings that reshuffle every month. Wait for dozens of forecasts per person before drawing conclusions.

A second failure is scoring easy questions. A forecaster who answers only near-certain questions can post an excellent score while adding nothing. Compare against the base rate, and score everyone on the same questions where possible. A third is vague resolution: questions with fuzzy outcomes get settled in the forecaster's favor. Write the resolution rule before anyone forecasts.

When to use something else

  • Calibration Training. When the scores show overconfidence and the work is training people to state ranges and probabilities that match their hit rate.
  • Superforecasting. When the aim is building a forecasting team and its habits, with scoring as one part of the practice.
  • Thinking in Bets. When decisions are few and large, and the work is judging each one by its reasoning apart from its result.

Where it sits in SOLVE eX

Go to the source

  • Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review, 1950
  • A New Vector Partition of the Probability Score, 1973

Back to the catalog