From the law of Charles Goodhart (1975), in the wording Marilyn Strathern gave it (1997), with the failure types of David Manheim and Scott Garrabrant (2018).
Before a number is tied to pay, promotion, or budget, predicts how people will game it and adds counterweights so the metric keeps tracking the result that matters.
A bonus on call time shortens calls and stops solving problems, until a second number pulls the other way
Agents are paid for short calls. At rest the bonus is 10 percent of pay with nothing paired against it: Play, and average call time falls while calls solved on first contact fall with it, as rushed calls come back as repeats (R1). Give repeat calls weight in the same bonus and Play again. Illustrative.
Average call time, minutes
Solved on first call
Month 12 of 12
- Average call time, minutes
- 7.6
- Solved on first call
- 69%
- R1, Rush and return
- Rushed calls leave problems unsolved, unsolved problems call back, and the fuller queue pushes agents to rush again. Two negative links: the loop reinforces. Two negative (−) links: an even count, so the loop reinforces.
- Bonus on call time
- Share of an agent's pay tied to average call time. Some of the push goes into real improvements: better scripts, faster lookups.
- Rushed calls
- The gaming: share of calls cut short, transferred, or closed before the problem is fixed, because that is the fastest way to hit the number.
- Average call time
- The metric. It improves both from real gains and from rushing, and the dashboard cannot tell the two apart.
- Weight on repeat calls
- The counterweight: part of the same bonus paid for few repeat calls, so a rushed call costs the agent who rushed it.
- Solved on first call
- The result the company wants: customers whose problem is fixed in one call. 75 percent before the bonus.
- Repeat calls
- Customers calling back about the same problem. They swell the queue, and a longer queue pushes agents to rush more.
An example
A home insurer's claims call center tied 10 percent of agents' pay to average call time, which had crept up to nine minutes. A year later the dashboard showed 7.6 minutes, and the operations director wanted to raise the bonus. A quality analyst pulled a second number first. Callers whose problem was fixed on the first call had fallen from 75 percent to 69, and the repeat calls had eaten most of the time saved. Agents had learned to transfer hard calls, promise callbacks, and close claims questions that were still open.
Before the next bonus cycle the team ran the metric through a Goodhart check: what result do we want, how could an agent hit this number without moving it, and which number would move against the shortcut? They kept call time in the bonus, paid 40 percent of it on low repeat calls by the same agent, and audited twenty recorded calls per agent each quarter. Call time settled near 8.6 minutes, and first-call resolution rose to about 77 percent.
Why it works
A number is useful because it moves with something that matters. Paying people to move the number changes what moves it. Some of the push goes into the real thing, and some goes into whatever moves the number most cheaply, often a shortcut that leaves the real thing worse. The cheaper the shortcut, the faster the number and the result come apart, and the dashboard shows only the number.
Asking how the metric will be gamed before it carries money makes the shortcut visible while it is still hypothetical. The usual answer is a counterweight: a second number that the shortcut pushes the wrong way, paid from the same bonus, so gaming one costs the agent on the other. Checking the pair against the result after the incentive starts tells whether the design held.
The thinking behind it
Charles Goodhart Charles Goodhart, a monetary economist at the Bank of England and later the London School of Economics, stated the law in Problems of Monetary Management: The U.K. Experience (1975), published in the Reserve Bank of Australia's Papers in Monetary Economics. His point was that a statistical regularity observed in the past tends to break down once someone leans on it to control the outcome. He was writing about steering an economy by its monetary numbers, and the point carries to any figure an organization starts to steer by.
Marilyn Strathern Marilyn Strathern, a Cambridge social anthropologist known for her work in Papua New Guinea and on audit culture, gave the law the general form most people now quote, in 'Improving Ratings': Audit in the British University System (European Review, 1997): a measure turned into a target stops being a good measure. Her setting, the rating of universities, puts the pattern inside organizations, where a score that decides funding draws effort toward itself.
David Manheim David Manheim, with Scott Garrabrant, who had first named the variants, set out in Categorizing Variants of Goodhart's Law (2018) four ways that optimizing a metric breaks its link to the goal it was chosen for: regressional, extremal, causal, and adversarial. In plain terms they are noise in the top scores, relationships that fail at the extremes, a metric that moves with the goal without driving it, and deliberate gaming. Naming the likely one points to the defense it needs.
How to run it
- Name the result that matters. Write it in words before naming any number, such as customers whose problem is fixed.
- List candidate metrics. Check how well each tracked that result in data from before any incentive.
- Ask how each could be hit cheaply. Ask the people who will be paid on it; they know the shortcuts.
- Name the kind of failure. Noise, extremes, a false cause, or deliberate gaming; each calls for a different defense.
- Add counterweights. Pair the metric with one the shortcut pushes the wrong way, set guardrails, and audit a sample.
- Keep the stakes proportionate. The larger the share of pay on one number, the harder people will look for the cheap route to it.
- Recheck after the incentive starts. Compare metric and result each quarter; a widening gap means the metric is being worked.
Where it goes wrong
The first failure is testing the metric only before the money arrives. A number that tracked the result well last year proves little about how it will behave once people are paid on it. Spot it when the case for a metric rests on historical correlation alone, and recover by scheduling a recheck after the first incentive period.
The second is piling on metrics until nobody knows what is wanted. Five counterweights dilute the signal and invite gaming of the weakest one, while one or two well-chosen pairs do more. The third is treating gaming as a character flaw. People respond to what they are paid for, and a design that relies on nobody responding has already failed.
When to use something else
- KPI Design. When the work is building a full set of measures for a team, beyond guarding one metric against gaming.
- Premortem. When the worry is the whole incentive plan, and a story of how it failed, told in advance, will surface more than one metric's shortcuts.
- Regression to the Mean. When the top scorers on a metric are about to be rewarded and the question is how much of their lead is luck.
Where it sits in SOLVE eX
Go to the source
- Problems of Monetary Management: The U.K. Experience, Papers in Monetary Economics, Reserve Bank of Australia, 1975
- Improving Ratings: Audit in the British University System, European Review, 1997
- Categorizing Variants of Goodhart's Law, 2018