Randomized Controlled Trial

Thinking Tools6.1 Designing and Testing the SolutionAlso known as A/B Test, RCT, Controlled Experiment

Rooted in Francis Bacon (Novum Organum, 1620) and John Stuart Mill (A System of Logic, 1843), made rigorous by Ronald A. Fisher (Statistical Methods for Research Workers, 1925), and set out for online tests by Ron Kohavi, Diane Tang, and Ya Xu (Trustworthy Online Controlled Experiments, 2020).

Assigns units at random to a change or a control, measures an outcome fixed in advance, and estimates whether the change caused the difference.

The same sign-up test run a thousand times, and how often it finds a lift that is really there

A new onboarding email truly lifts sign-ups by the amount set below. Play runs the test a thousand times; each run splits fresh visitors at random between the old email and the new one and reads the result; the bars show how often the test calls a clear win, a maybe, or a loss. At 2,000 visitors a group, fewer than one run in five finds the lift that is really there. Raise the sample until the clear-win bar dominates, which is what a power calculation does before the real test. Illustrative.

1.0
2,000
  • Measured lift in one run: around 1.0 points, give or take 0.9.
Looks worseLooks better, inside the noiseClear winLooks worseLooks better, insidethe noiseClear win

What one run of the test reports

Runs so far: 1,000 of 1,000

Runs that call it a clear win
19%
Runs that see a lift but cannot rule out chance
68%
Runs where the better email looks worse
13%

An example

A meal-kit subscription company rewrote the onboarding email new visitors get after they create an account. A week after launch the new version looked 1.4 points better on seven-day sign-ups, 11.4 percent against 10.0, on about 1,800 visitors each, and the product lead wanted to switch everyone over.

The analyst asked for a proper test first. The team wrote down the outcome (paid sign-up within seven days), the smallest lift worth acting on (one point), and the stopping rule (a fixed sample, no early looks). At a 10 percent base rate, catching a one-point lift four times in five needed about 14,000 visitors per group, so the test ran for eight weeks with visitors assigned at random. The lift came in at 0.8 points, with an interval of 0.1 to 1.5. The company shipped the email, booked the smaller gain in its forecast, and repeated the test in its second market before rolling it out there.

Why it works

Random assignment is the whole trick. When a coin decides who gets the change, everything else that moves the outcome (season, traffic source, mood, the customers who happen to show up) lands on both sides in roughly equal measure. A difference that survives that split is very likely the change at work.

The other half is deciding in advance. Fixing the metric, the sample, and the stopping rule before any data arrives removes the freedom to keep looking until a number looks good. The power calculation sets the sample so a real effect of the size that matters would usually show up; an underpowered test mostly produces maybes, and the occasional big win it reports tends to be luck that shrinks on the next run.

The thinking behind it

John Stuart Mill John Stuart Mill's A System of Logic (1843) set out the canons of experimental inquiry, among them the method of difference: if two cases differ in one circumstance only, and the outcome differs, that circumstance is the cause or part of it. A controlled experiment is an attempt to manufacture Mill's two matched cases on purpose, at a scale where the leftover differences average out.

Ronald A. Fisher Ronald A. Fisher gave working scientists the tests of significance in Statistical Methods for Research Workers (1925), and in The Design of Experiments (1935) argued that randomization, replication, and blocking are what make an experiment's result trustworthy. His answer to the problem of unknown differences between groups was to let chance assign the groups, so that the arithmetic of chance could then judge the result.

Ron Kohavi Ron Kohavi, with Diane Tang and Ya Xu, wrote Trustworthy Online Controlled Experiments (2020), the practitioner's guide to A/B tests at scale, drawn from running experimentation platforms at large technology companies. It covers choosing an overall evaluation metric, checking that randomization actually worked, and the pitfalls that can leave an online test's result untrustworthy without anyone on the team noticing.

How to run it

  1. State the claim as cause and effect. Name the change and the one outcome it should move, such as the new email raising seven-day paid sign-ups.
  2. Fix the plan before any data arrives. Write down the metric, the analysis, the sample size, and the stopping rule, and share them with whoever will judge the result.
  3. Size the test to the smallest lift that matters. Run a power calculation; if the sample it asks for is out of reach, test a bolder change or pick a metric that moves faster.
  4. Let chance assign the units. Randomize by visitor, account, or store, and check the groups match on traffic source and history before reading the outcome.
  5. Run to the planned sample without peeking. Early looks invite early stopping, and early stops on a good day inflate the effect.
  6. Report the lift with its interval. Give the size of the effect and its range, so the decision can be made on the low end as well as the middle.
  7. Replicate before scaling. Run it again in another market or season, since a first result is often larger than the effect that holds.

Where it goes wrong

The most common failure is a test too small to see the effect it is looking for. It shows up as a string of inconclusive results, or a dramatic win that vanishes when repeated. Run the power calculation first and size the test, or accept that the question cannot be answered this quarter.

Peeking is the second. Someone checks the dashboard daily and calls it as soon as the number looks good, which turns noise into launches. The fix is a stopping rule set in writing, or a method built for sequential looks. Third, the randomization itself breaks: a bug sends more returning customers to one group. Check the split before reading the result, and treat a lopsided split as a failed test.

When to use something else

  • Difference-in-Differences. When the change cannot be randomized, but a comparable group and data from before and after the change exist.
  • Design of Experiments. When several factors and their interactions need testing at once, and one change against a control is too slow.
  • Pre-Registered Predictions. When the problem is a team that explains every result after the fact, and the first step is committing to predictions in writing.

Where it sits in SOLVE eX

Go to the source

  • Novum Organum, 1620
  • A System of Logic, 1843
  • Statistical Methods for Research Workers, Oliver and Boyd, 1925
  • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020

Back to the catalog