From clinical trial registration (ClinicalTrials.gov, 2000), and in forecasting from Philip E. Tetlock's Good Judgment Project (2011).
Writes down the hypothesis, prediction, data, and decision rule before results arrive, and time-stamps the record so hindsight cannot rewrite what was expected.
A price test with its prediction written first, and four chances to rewrite history
A meal-kit company tests a rise from $60 to $66 a box in two regions. At each of the team's turns, pick a move; the data or the room answers, with a verdict on whether the move kept the record honest. Illustrative.
Pricing team
The data and the room
Pricing team
The data and the room
Moves that kept the record honest: 0 of 5Moves that kept the record honest: 1 of 5Moves that kept the record honest: 2 of 5Moves that kept the record honest: 3 of 5Moves that kept the record honest: 4 of 5Moves that kept the record honest: 5 of 5
- Write the prediction (Pricing team). The test starts Monday. Pick what the team writes down first.
- The price rise should not hurt sign-ups much. Any result will fit this sentence. Too vague to fail, so the test cannot teach anything.
- New sign-ups fall 4 percent, range 1 to 7; above 7 percent, we roll back. A number, a range, and a rule decided before any data. A prediction that can be wrong, with the decision fixed in advance.
- Lock it (Pricing team). Pick where the prediction lives during the test.
- Keep it in the team's slide deck. The deck is edited four times during the test. Nothing proves what the page said before the result.
- Email it to the finance director and save it, dated, in the shared log. Finance has the page with Monday's date on it. Time-stamped outside the team: it provably came first.
- Six weeks later (The data and the room). New sign-ups fell nine percent in the test regions. In the same data, annual-plan sign-ups rose three percent.
- Read the result (Pricing team). Pick how the team reports the test.
- Lead with the annual-plan rise: the price works for loyal customers. A slide with that headline is ready within a day. A forking path: a comparison chosen after seeing the data.
- Report the 9 percent fall against the written rule, and roll the price back. The decision takes ten minutes, because it was made six weeks ago. The result read the way everyone agreed to read it.
- The unplanned finding (Pricing team). Pick what happens to the annual-plan rise.
- Treat it as proof and raise the price on annual plans now. Nothing on the page predicted it. A chance pattern acted on as if it were tested.
- Log it as a lead and register a new test of annual plans, prediction first. A second page goes to finance, dated, before the new test starts. Kept, labeled, and tested properly.
- The quarterly review (The data and the room). A vice president says the drop was always expected and the test went roughly as planned.
- Answer the room (Pricing team). Pick how the head of pricing answers.
- Agree: a drop was always likely. The minutes record that the test went as expected. Hindsight rewrites the record, and the miss is forgotten.
- Read the dated page aloud: we predicted 4 percent and got 9. The room moves on to why the estimate was off. The miss stays a miss, and the next estimate can learn from it.
- The follow-up test (The data and the room). The annual-plan test, registered in advance, finds a rise of two percent, inside the range of chance. The roll-back stands.
An example
A meal-kit company planned to raise its price from $60 to $66 a box, tested first in two regions. Before the test began, the head of pricing had the team write one page: the question, the prediction that new sign-ups would fall by no more than four percent, a range of one to seven, the data source, and a rule decided in advance: above seven percent, roll the price back. She emailed it to the finance director and saved it, dated, in a shared log.
Sign-ups fell nine percent. In the same data, sign-ups for annual plans rose three percent, and a slide headlined the price works for loyal customers appeared within a day. The page made the call simple. The test had missed the rule, so the price rolled back, and the annual-plan rise went into the log as a lead for a separate test, registered in advance. When a vice president said at the quarterly review that some drop had always been expected, she read the page aloud. The annual-plan test, run properly, found nothing beyond noise.
Why it works
Once a result is known, people remember having expected something close to it, and they find the story in the data that fits what they wanted. Both happen without dishonesty. A prediction written down and dated before the result arrives is immune to both: it says what the team actually expected, and the decision rule says in advance what result would change the plan.
The rule also closes the forking paths. Any rich dataset contains some subgroup that moved the right way by chance, and a team that chooses its comparison after seeing the data can always find a win. Fixing the measure, the comparison, and the threshold beforehand means the result is read one way, the way everyone agreed to. Findings outside the plan are kept, labeled as leads, and tested next, with their own prediction written first.
The thinking behind it
Philip E. Tetlock Philip E. Tetlock, a political psychologist who studies forecasting and judgment, led the Good Judgment Project, which competed from 2011 in a forecasting tournament run by IARPA, the US intelligence research agency. In a tournament, forecasts are made before events resolve and scored against what happens. Superforecasting: The Art and Science of Prediction (2015), written with Dan Gardner, describes the habits of the most accurate of them. A dated prediction with a number on it is what made that scoring, and that learning, possible.
How to run it
- Write the question and the prediction. State what you expect as a number with a range, such as sign-ups fall four percent, between one and seven.
- Fix the measure and the comparison. Name the data, the period, and the groups you will compare, before any data arrives.
- Write the decision rule. Say what you will do for each result, such as roll back above seven percent.
- Date it and send it. Email it to someone outside the team or save it in a dated log, so it provably existed first.
- Run the work as planned. Change nothing about the measure or the groups once the test starts.
- Read the result against the page. Compare the outcome with the written prediction and apply the rule as written.
- Label the rest as leads. Record findings outside the plan as hypotheses, and register a new test for each one worth pursuing.
Where it goes wrong
The prediction is too vague to fail. The price rise will not hurt much survives any result, so the test teaches nothing. Spot it when no outcome would count as wrong; insist on a number and a range.
The page is quietly revised. Someone updates the prediction after an early look at the data, and the record now proves nothing. Keep the original unchanged and date every addition separately.
An unplanned subgroup is reported as the finding. The annual-plan rise looks like a win, and the headline moves to it. Ask whether the comparison was on the page before the test; if it was not, it is a lead, and it gets its own test before anyone acts on it.
When to use something else
- Randomized Controlled Trial. When the change can be randomized across many customers, and the effect must be separated from chance with a proper test.
- Brier Score. When many probability forecasts need scoring over time, to measure how well calibrated a person or team is.
- Falsifiability Test. When the first question is whether a claim could be shown wrong by any result at all.
Where it sits in SOLVE eX
Go to the source
- ClinicalTrials.gov public trial registry, 2000
- Good Judgment Project, the IARPA ACE forecasting tournament, 2011