Poker Decision Review

Thinking Tools6.4 Conducting Follow-Up and Taking FeedbackAlso known as Hand-History Review

From professional poker's hand-history review, carried into business by Annie Duke (Thinking in Bets, 2018).

Rebuilds each significant decision from a session, with what was known, the options, and the reasoning, and grades the decision apart from how it turned out.

Eight decisions from a sales quarter, graded apart from how they turned out

Each decision is placed twice over: across by its result, up by how good the call was given what was known then. The first is placed as an example; read each note, grade the call before you look at its result, and place it. Illustrative.

Items (select one, then a box)
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss
  • Not placedPlacedEarnedBad luckDumb luckDeserved loss

The decision, given what was known

SoundPoor
Bad luck
Earned
Deserved loss
Dumb luck
BadlyWell

How it turned out

Unchecked referral hire. Known: a friend vouched; no references called. Result: the rep is the top closer this quarter.Not placed yet.Unchecked referral hire. Known: a friend vouched; no references called. Result: the rep is the top closer this quarter.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Unchecked referral hire. Known: a friend vouched; no references called. Result: the rep is the top closer this quarter.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Unchecked referral hire. Known: a friend vouched; no references called. Result: the rep is the top closer this quarter.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Skipping the checks was a poor call that happened to pay off. The next unchecked hire is the risk.Unchecked referral hire. Known: a friend vouched; no references called. Result: the rep is the top closer this quarter.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Refused a 30% discount. Known: the margin floor is 20%, and deeply discounted accounts churn twice as often. Result: the buyer signed with a rival.Not placed yet.Refused a 30% discount. Known: the margin floor is 20%, and deeply discounted accounts churn twice as often. Result: the buyer signed with a rival.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Refused a 30% discount. Known: the margin floor is 20%, and deeply discounted accounts churn twice as often. Result: the buyer signed with a rival.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.The call followed the margin rule and the churn data. Losing one deal does not make it wrong.Refused a 30% discount. Known: the margin floor is 20%, and deeply discounted accounts churn twice as often. Result: the buyer signed with a rival.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Refused a 30% discount. Known: the margin floor is 20%, and deeply discounted accounts churn twice as often. Result: the buyer signed with a rival.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Forecast unsigned deal. Known: no signed paper, and the buyer's champion had just left. Result: it slipped, and the quarter missed.Not placed yet.Forecast unsigned deal. Known: no signed paper, and the buyer's champion had just left. Result: it slipped, and the quarter missed.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Forecast unsigned deal. Known: no signed paper, and the buyer's champion had just left. Result: it slipped, and the quarter missed.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Forecast unsigned deal. Known: no signed paper, and the buyer's champion had just left. Result: it slipped, and the quarter missed.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Forecast unsigned deal. Known: no signed paper, and the buyer's champion had just left. Result: it slipped, and the quarter missed.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Every warning sign was visible when the forecast went in. The process failed, not the luck.Piloted before rollout. Known: a new pricing page, untested. Result: the pilot caught a checkout bug; the rollout went clean.Not placed yet.Piloted before rollout. Known: a new pricing page, untested. Result: the pilot caught a checkout bug; the rollout went clean.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.A cheap test before a big move, and it paid for itself.Piloted before rollout. Known: a new pricing page, untested. Result: the pilot caught a checkout bug; the rollout went clean.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Piloted before rollout. Known: a new pricing page, untested. Result: the pilot caught a checkout bug; the rollout went clean.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Piloted before rollout. Known: a new pricing page, untested. Result: the pilot caught a checkout bug; the rollout went clean.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Raised renewal prices 8%. Known: usage up 40%, ninety days' notice given. Result: one large account left over budget cuts.Not placed yet.Raised renewal prices 8%. Known: usage up 40%, ninety days' notice given. Result: one large account left over budget cuts.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Raised renewal prices 8%. Known: usage up 40%, ninety days' notice given. Result: one large account left over budget cuts.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.The account left for its own reasons. The pricing call was sound on everything known.Raised renewal prices 8%. Known: usage up 40%, ninety days' notice given. Result: one large account left over budget cuts.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Raised renewal prices 8%. Known: usage up 40%, ninety days' notice given. Result: one large account left over budget cuts.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Won a logo at a loss. Known: priced below cost, with no plan to raise it. Result: the customer tripled its seats.Not placed yet.Won a logo at a loss. Known: priced below cost, with no plan to raise it. Result: the customer tripled its seats.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Won a logo at a loss. Known: priced below cost, with no plan to raise it. Result: the customer tripled its seats.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Won a logo at a loss. Known: priced below cost, with no plan to raise it. Result: the customer tripled its seats.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.The expansion was not foreseen or planned for. A poor bet that came in.Won a logo at a loss. Known: priced below cost, with no plan to raise it. Result: the customer tripled its seats.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.Skipped security review. Known: the review was required for regulated buyers. Result: a bank's audit failed the deal.Not placed yet.Skipped security review. Known: the review was required for regulated buyers. Result: a bank's audit failed the deal.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Skipped security review. Known: the review was required for regulated buyers. Result: a bank's audit failed the deal.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Skipped security review. Known: the review was required for regulated buyers. Result: a bank's audit failed the deal.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Skipped security review. Known: the review was required for regulated buyers. Result: a bank's audit failed the deal.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.The rule existed for exactly this buyer. Skipping it was the error.Kept the proven rep. Known: a slower but proven rep against a fast new one, for a complex renewal. Result: renewed on time.Not placed yet.Kept the proven rep. Known: a slower but proven rep against a fast new one, for a complex renewal. Result: renewed on time.Earned: A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.Matching the rep to the complexity of the deal was the right call, and it worked.Kept the proven rep. Known: a slower but proven rep against a fast new one, for a complex renewal. Result: renewed on time.Bad luck: A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.Kept the proven rep. Known: a slower but proven rep against a fast new one, for a complex renewal. Result: renewed on time.Dumb luck: A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.Kept the proven rep. Known: a slower but proven rep against a fast new one, for a complex renewal. Result: renewed on time.Deserved loss: A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.

Your sort matched 0 of 8.Your sort matched 1 of 8.Your sort matched 2 of 8.Your sort matched 3 of 8.Your sort matched 4 of 8.Your sort matched 5 of 8.Your sort matched 6 of 8.Your sort matched 7 of 8.Your sort matched 8 of 8.

Earned
A sound call that worked. Keep doing it, and check that the reasoning, not luck, is what worked.
Bad luck
A sound call that lost. Change nothing about the method; the same call, made again, wins more often than it loses.
Dumb luck
A poor call that worked. The most dangerous box: the win teaches the wrong lesson unless someone names it.
Deserved loss
A poor call that lost. Find the step that failed (missing facts, skipped checks, a bad read) and fix that step.

The sort

  • Unchecked referral hire: Dumb luck. Skipping the checks was a poor call that happened to pay off. The next unchecked hire is the risk.
  • Refused a 30% discount: Bad luck. The call followed the margin rule and the churn data. Losing one deal does not make it wrong.
  • Forecast unsigned deal: Deserved loss. Every warning sign was visible when the forecast went in. The process failed, not the luck.
  • Piloted before rollout: Earned. A cheap test before a big move, and it paid for itself.
  • Raised renewal prices 8%: Bad luck. The account left for its own reasons. The pricing call was sound on everything known.
  • Won a logo at a loss: Dumb luck. The expansion was not foreseen or planned for. A poor bet that came in.
  • Skipped security review: Deserved loss. The rule existed for exactly this buyer. Skipping it was the error.
  • Kept the proven rep: Earned. Matching the rep to the complexity of the deal was the right call, and it worked.

An example

A software company's head of sales ran her quarterly review on decisions instead of deals. She picked the twelve calls that had moved the most money and wrote each one up on a card: what the team knew, the options it had, the choice, and the reasoning. Results went on the back.

The team graded each card face up, then turned it over. Refusing a buyer's demand for a 30% discount had lost the deal to a rival, and the room's first instinct was to call it a mistake; graded before the result, it followed the margin floor and the churn data, so it went in bad luck. A hire made on a friend's word, with no references called, had become the top closer; graded first, it was a poor call, and it went in dumb luck. The quarter's real lesson was in the deserved-loss box: two forecasts built on deals with no signed paper. The fix was a rule that unsigned deals stay out of the commit.

Why it works

An outcome mixes two things: how good the decision was, and luck. Over one result there is no telling them apart, so people grade the decision by the result, which Annie Duke calls resulting. That teaches the wrong lessons twice: sound calls that lost get abandoned, and poor calls that won get repeated until the luck runs out.

The review separates the two by rebuilding each decision from what was known at the time and grading it before the result is in view. The four boxes then say what to do. Earned wins and bad luck call for no change in method; deserved losses point to the step that failed; dumb luck is the box that needs the most attention, because nothing else will flag it. Reviewing a batch of similar decisions shows patterns no single result can.

The thinking behind it

Annie Duke Annie Duke, a former professional poker player who writes on decision making under uncertainty, brought poker's habit of reviewing hand histories into business in Thinking in Bets (2018). The book argues for judging decisions by their quality and not their outcomes, and names the opposite habit resulting. It also asks for beliefs to be held as bets, each with a degree of confidence, so that reviewing a decision means asking what the decision maker knew and how sure they had a right to be.

How to run it

  1. Pick the period and the batch. A quarter of sales calls, a year of hires, a fund's investments: decisions of one kind, enough of them to compare.
  2. Choose the decisions that mattered. Ten to twenty, chosen by the money or risk at stake, not by how they turned out.
  3. Rebuild each from what was known then. The facts in hand, the options, the choice, and the reasoning, written without hindsight.
  4. Hide the results. Put each outcome where the group cannot see it until the grade is set.
  5. Grade each decision. Sound or poor, given the information and the time available, with one sentence of reason.
  6. Reveal results and place each in its box. Earned, bad luck, dumb luck, or deserved loss.
  7. Act on two boxes. Fix the failed step behind each deserved loss, and name each piece of dumb luck aloud so nobody copies it.

Where it goes wrong

The results leak. Everyone in the room already knows how the deal ended, and the grading bends toward it. Have someone outside the deal write the case, or grade from notes made at the time, and ask each grader to state the reason before discussion.

Bad luck becomes an excuse. Every loss gets filed as unlucky, and nothing is ever fixed. Ask what a better decision maker would have known or checked; if there is an answer, it was not luck.

The batch is too small. Three decisions in a review can only produce anecdotes. Wait until there are enough similar decisions for a pattern, or review the same kind of decision across several people.

When to use something else

  • Thinking in Bets. When the decision has not been made yet and the job is stating it as a bet with a probability.
  • After-Action Review. When the review is of one team action just finished, and what matters is what happened and what to change.
  • Blameless Postmortem. When one incident did real damage and the causes sit in systems and process, not a person's judgment.

Where it sits in SOLVE eX

Go to the source

  • Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts, Portfolio, 2018

Back to the catalog