Mill's Methods

Thinking Tools4.1 Root Causes and the Case for Going BackAlso known as Methods of Agreement and Difference

Set out by John Stuart Mill (A System of Logic, 1843).

Compares cases with and without an effect using five set patterns, agreement, difference, the two combined, residues, and concomitant variation, to narrow down its likely cause.

Five ways to read six product launches for a cause

Six kitchenware launches: A, C, and E sold out; B, D, and F did not. At each of the analyst's turns the panel gives the cases a method compares; pick what that method points to. The launches' turns say what it settles and what it leaves open. Illustrative.

Analyst

The six launches

Analyst

The six launches

Select a stage to read it, or play the sequence.Agreement. A had a free trial and an influencer push. C had a free trial, a holiday launch, and a price under $50. E had a free trial and a marketplace listing.Pick a move to see the reply.Only C had it.Missed: a cause has to be present in every case of the effect.All three hits had one, and they share nothing else.Agreement: the one circumstance common to every success.Only C launched in the holidays.Missed: two of the three hits had no holiday launch.What agreement gives. Agreement narrows the field to the trial, but a shared factor nobody recorded, such as one product manager, would pass the same test.Difference. C and F were alike in every recorded way, a holiday launch and a price under $50, except that C had a free trial. C sold out; F sold 2,000 units.Pick a move to see the reply.Both had it.Missed: what both cases share cannot explain why they differ.It is the only recorded way the two differ.Difference: the one circumstance present in the success and absent in the failure.What difference gives. With everything else held the same, the trial is the gap between a hit and a miss. Difference is the logic of a controlled experiment.Joint method. Every success had a free trial. B had an influencer push and a holiday launch; D had a price under $50 and an influencer push; F had a holiday launch and a price under $50. None of the three had a trial.Pick a move to see the reply.B had one and failed.Missed: it shows up among the failures too.Present in all three hits and absent from all three misses.Joint method: agreement in both groups at once, the strongest pattern of the five.What joint gives. Two groups agree on it from opposite sides. The case for the trial is now strong enough to test on purpose.Residues. E sold 12,000 units. The other launches put a free trial at about 5,000 units on its own. E's only other feature was the marketplace listing.Pick a move to see the reply.It explains about 5,000 of them.Missed: 7,000 units are still unexplained.Subtract the known effect, and 7,000 units remain for the one feature left.Residues: what is left after known causes are removed points at the remaining one.What residues give. The listing becomes a candidate no other method surfaced. It is only as good as the 5,000-unit estimate it rests on.Concomitant variation. Prices across the six launches ran from $35 to $65. Every $10 cut came with about 1,500 more units sold, in hits and misses alike.Pick a move to see the reply.C and F were both priced at $45, and one sold out.Missed: price moves units, and it did not decide hit or miss.Sales rise steadily as price falls, across all six.Concomitant variation: two things that vary together are likely linked.What variation gives. Price is a lever on volume. It is not the switch between a hit and a miss; the trial is the candidate for that.

Methods read correctly: 0 of 5Methods read correctly: 1 of 5Methods read correctly: 2 of 5Methods read correctly: 3 of 5Methods read correctly: 4 of 5Methods read correctly: 5 of 5

  1. Agreement (Analyst). A had a free trial and an influencer push. C had a free trial, a holiday launch, and a price under $50. E had a free trial and a marketplace listing.
    • The price under $50 Only C had it. Missed: a cause has to be present in every case of the effect.
    • The free trial All three hits had one, and they share nothing else. Agreement: the one circumstance common to every success.
    • The holiday launch Only C launched in the holidays. Missed: two of the three hits had no holiday launch.
  2. What agreement gives (The six launches). Agreement narrows the field to the trial, but a shared factor nobody recorded, such as one product manager, would pass the same test.
  3. Difference (Analyst). C and F were alike in every recorded way, a holiday launch and a price under $50, except that C had a free trial. C sold out; F sold 2,000 units.
    • The holiday launch Both had it. Missed: what both cases share cannot explain why they differ.
    • The free trial It is the only recorded way the two differ. Difference: the one circumstance present in the success and absent in the failure.
  4. What difference gives (The six launches). With everything else held the same, the trial is the gap between a hit and a miss. Difference is the logic of a controlled experiment.
  5. Joint method (Analyst). Every success had a free trial. B had an influencer push and a holiday launch; D had a price under $50 and an influencer push; F had a holiday launch and a price under $50. None of the three had a trial.
    • The influencer push B had one and failed. Missed: it shows up among the failures too.
    • The free trial Present in all three hits and absent from all three misses. Joint method: agreement in both groups at once, the strongest pattern of the five.
  6. What joint gives (The six launches). Two groups agree on it from opposite sides. The case for the trial is now strong enough to test on purpose.
  7. Residues (Analyst). E sold 12,000 units. The other launches put a free trial at about 5,000 units on its own. E's only other feature was the marketplace listing.
    • The trial explains all 12,000 It explains about 5,000 of them. Missed: 7,000 units are still unexplained.
    • The marketplace listing, about 7,000 units Subtract the known effect, and 7,000 units remain for the one feature left. Residues: what is left after known causes are removed points at the remaining one.
  8. What residues give (The six launches). The listing becomes a candidate no other method surfaced. It is only as good as the 5,000-unit estimate it rests on.
  9. Concomitant variation (Analyst). Prices across the six launches ran from $35 to $65. Every $10 cut came with about 1,500 more units sold, in hits and misses alike.
    • Price decides which launches sell out C and F were both priced at $45, and one sold out. Missed: price moves units, and it did not decide hit or miss.
    • Price changes sales by degree Sales rise steadily as price falls, across all six. Concomitant variation: two things that vary together are likely linked.
  10. What variation gives (The six launches). Price is a lever on volume. It is not the switch between a hit and a miss; the trial is the candidate for that.

An example

A kitchenware company had launched six products in two years. Three sold out, three did not, and the post-launch reviews had blamed a different thing each time: timing, price, the influencer, the channel. The head of product put all six on one sheet, with every feature each launch had, and read the sheet five ways.

The three hits shared only one feature: a free trial. Two launches, C and F, matched on everything recorded except the trial, and only C sold out. No miss had a trial and every hit did. Subtracting the trial's usual effect from E's sales left about 7,000 units that pointed at E's marketplace listing, a cause nobody had suggested. And sales rose steadily as price fell, in hits and misses alike, so price mattered by degree without deciding which launches succeeded. The next launch ran a free trial in half its regions and none in the other half, and listed on the marketplace from day one. The trial regions sold twice as many units.

Why it works

Post-mortems on single cases tend to blame whatever is most memorable. Mill's methods replace the story with comparison. Each method is a different way of lining up cases with and without the effect so that most candidate causes are ruled out, leaving few standing.

Agreement asks what every case of the effect shares. Difference asks what separates two otherwise matching cases, which is the logic of a controlled experiment. The joint method demands both. Residues subtract what is already explained and look at what is left. Concomitant variation looks for a factor that rises and falls with the effect. None of them proves a cause on its own, because an unrecorded factor can pass any of the tests. Together they narrow the field enough to know what to test properly next.

The thinking behind it

John Stuart Mill John Stuart Mill, the British philosopher and economist, set out the canons of experimental inquiry in A System of Logic, Ratiocinative and Inductive (1843), in Book III, chapter 8: the methods of agreement, difference, residues, and concomitant variation, with a joint method that combines the first two. He wrote them as rules for reasoning from observed cases to their causes. The method of difference, which compares two cases alike in every circumstance but one, is the reasoning that a controlled experiment puts into practice.

How to run it

  1. Collect the cases. Gather every instance with and without the effect, not just the memorable ones.
  2. List the candidate factors. Record each factor that was present or absent in every case, in one table.
  3. Apply agreement. Find what every case of the effect shares, and cross off anything absent from even one.
  4. Apply difference. Find two cases alike in every recorded way but the effect, and name what separates them.
  5. Check the joint pattern. Confirm the candidate is present in every success and absent in every failure.
  6. Subtract and look at the residue. Take away what known causes explain and ask what the leftover effect points to.
  7. Look for variation together. Plot the effect against each factor that comes in degrees, such as price or length.
  8. Test the survivor. Run a deliberate comparison on the strongest candidate before acting on it.

Where it goes wrong

The first failure is the unrecorded factor. If every hit also had the same product manager and nobody wrote that down, agreement will credit the trial for her work. Before trusting a result, ask what else the cases might share that is not on the sheet.

The second is too few cases. Six launches can make a coincidence look like a pattern. Treat the result as a candidate to test.

The third is reading variation together as cause. Sales and price moved together, but a third factor, such as a stronger season, could drive both. Use concomitant variation to find what to test, and the method of difference, run on purpose, to settle it.

When to use something else

Where it sits in SOLVE eX

Go to the source

  • A System of Logic, Ratiocinative and Inductive, John W. Parker, London; Book III, chapter 8, 1843

Back to the catalog