Design of Experiments

Thinking Tools6.1 Designing and Testing the SolutionAlso known as DOE, Factorial Design

Developed by Ronald A. Fisher (The Design of Experiments, 1935) and carried into industry by George E. P. Box, William G. Hunter, and J. Stuart Hunter (Statistics for Experimenters, 1978).

Varies several factors together in a structured pattern, with replication and randomization, to learn in few runs how each factor and their interactions move an outcome.

Eight runs of a sign-up page test, and what testing one change at a time would have reported

Three changes to a sign-up page, each on or off, make eight versions; each bar is one version's sign-up rate (Mo and Yr: the price shown monthly or yearly). Set how much each change really moves sign-ups and how much the headline and the price display interact. The eight runs recover every effect; testing one change at a time from the old page measures the headline only with the monthly price, and reports the interaction as part of it. Illustrative.

1.0
0.5
0.0
−2.0
Eight runs: headline's effect, averaged over the other changes
1.0 points
Eight runs: headline and price interaction
−2.0 points
Eight runs: headline's effect with the yearly price
−1.0 points
One at a time from the old page: the headline's effect, reported as if it held everywhere
3.0 points

Sign-up rate in each run, percent

5.07.510.012.515.08.211.210.89.88.211.210.89.8Old MoGrayNew MoGrayOld YrGrayNew YrGrayOld MoGreenNew MoGreenOld YrGreenNew YrGreen5.07.510.012.515.08.211.210.89.88.211.210.89.8OldMoGrayNewMoGrayOldYrGrayNewYrGrayOldMoGreenNewMoGreenOldYrGreenNewYrGreen

An example

A software company's growth team had three changes queued for its sign-up page: a new headline, showing the price per year instead of per month, and a green button. The plan was three tests in a row, one change each, over six weeks. The analyst proposed one test instead: all eight combinations at once, each visitor assigned a version at random, run for two weeks with enough traffic in every version to read a one-point difference.

The results split the effects apart. The button did nothing. The new headline added a point of sign-up rate on average, but that average hid two cases: with the monthly price it added three points, and with the yearly price it cost one. A one-at-a-time test from the old page would have measured only the first case and reported a three-point win.

That mattered because finance planned to move the page to yearly pricing next quarter. The team shipped the new headline with the monthly price and booked a retest of the headline for the week the pricing changed.

Why it works

Testing one change at a time measures each change at a single setting of everything else. If the changes act independently, that works; if they interact, each estimate depends on where the other factors happened to sit, and nobody can tell. A factorial design varies the factors together, so every run informs every estimate: each effect is the average difference across half the runs against the other half, and the same runs show whether one factor's effect depends on another's level.

Random assignment of units to runs keeps unmeasured differences (time of day, traffic source, a holiday) from lining up with any factor, and replication shows how large a difference noise alone produces. Together they turn a few runs into estimates that can be trusted, and they surface the interactions that change the decision.

The thinking behind it

Ronald A. Fisher Ronald A. Fisher, statistician and geneticist, set out the principles of experimental design in The Design of Experiments (1935): randomization, replication, and blocking, along with the null hypothesis, which he illustrated with the lady tasting tea. His earlier Statistical Methods for Research Workers (1925) had given working scientists tests of significance and the analysis of variance, the arithmetic that splits a factorial experiment's results into the effect of each factor and their interactions.

George E. P. Box George E. P. Box, a British statistician who worked on quality control, time series, and Bayesian inference, carried these designs into industry with William G. Hunter and J. Stuart Hunter in Statistics for Experimenters (1978), a standard text on designing experiments and analyzing their results. Box is also known for the remark that all models are wrong but some are useful, a fair summary of how experimenters should treat their own estimates.

J. Stuart Hunter J. Stuart Hunter, a statistician born in 1923, was the third author, with George Box and William Hunter, of Statistics for Experimenters (1978), published by Wiley. The book taught generations of engineers and scientists to plan factorial designs, block out nuisance variation, randomize the order of runs, and fit response surfaces. Through it the method became a working tool on factory floors and in laboratories, well outside statistics departments.

How to run it

  1. Name the outcome and its unit. Pick one measured response (sign-up rate, yield, defect count) and how it will be read.
  2. List the factors and their levels. Keep to the few that plausibly matter, each at two levels to start.
  3. Choose the design. Use a full factorial for up to four factors, a fractional one when the runs would be too many.
  4. Size the runs. Work out how many units each run needs to detect the smallest difference worth acting on.
  5. Randomize and block. Assign units to runs at random, and group runs by known nuisances such as shift or week.
  6. Run every combination as planned. Record each run's factor settings and response, and resist stopping early on a good look.
  7. Estimate effects and interactions. Compare the average of each factor's high runs with its low runs, then check each pair for interaction.
  8. Confirm the winner. Rerun the chosen combination before rolling it out, since one experiment can still mislead.

Where it goes wrong

Too many factors at once. Eight factors at two levels is 256 runs, and each one starves for traffic; spot it when every run's sample is too small to read. Screen first with a fractional design, then study the few that matter.

The randomization is not random. Running version A on weekdays and B on weekends builds the calendar into the result. Check that assignment came from a random draw, and that the groups look alike on what was measured before the test.

Interactions get averaged away in the report. A summary that lists only main effects throws out the finding that changes the decision. Report each important effect at each level of the factor it interacts with.

When to use something else

Where it sits in SOLVE eX

Go to the source

  • The Design of Experiments, 1935
  • Statistics for Experimenters, 1978

Back to the catalog