From the mathematics of combinatorics, running from Blaise Pascal (Traité du triangle arithmétique, 1665) through Jacob Bernoulli (Ars Conjectandi, 1713) and Leonhard Euler (Introductio in analysin infinitorum, 1748).
Counts and lists every combination in a decision or test space, using multiplication, permutations, and combinations, so no case is missed and the size of the space is known.
How many test cells a pricing experiment really has, and whether one quarter can cover them
Every dimension multiplies the space. Three prices, two bundles, two regions, one channel, and one welcome email picked from five drafts make 60 test cells, exactly what the team can run this quarter. Now add a second email: the emails are a sequence, so their order counts, and two picked from five make 20 orders where 10 pairs would do if order did not matter. The space jumps to 240 and falls below the line, where it must be sampled or pruned. Illustrative.
Coverage this quarterFull coverage fits: run every combination this quarter.Too many to cover: sample the space on purpose, or cut a dimension.
An example
A meal-planning app's growth team proposed a pricing experiment for the quarter: three monthly prices, two bundles (with and without the grocery add-on), two regions, and the web sign-up flow, each new subscriber getting one welcome email from five drafts. The team could run about 60 test cells at a useful sample size. Someone added that the welcome series should be two emails, not one, since the second drives most conversions.
The analyst counted first. As proposed, the space was 3 × 2 × 2 × 1 × 5, or 60 cells: exactly the capacity. Two emails picked from five, sent in order, made 20 sequences instead of 5, and the space jumped to 240 cells, four quarters of testing. If only the pair mattered and not the order, it would be 10 pairs and 120 cells, still twice the capacity.
The team kept one email in the price test and ran the email sequence as its own small test the next quarter, with the price fixed at the winner. Both finished on time; every cell in each was actually tested.
Why it works
Spaces of possibilities grow by multiplication, and intuition grows them by addition. Adding one more dimension to a test or a rule set feels like a small change, and it doubles or triples the work. Counting first, with the plain rules of combinatorics, turns that hidden growth into a number the team can compare with its capacity before it commits.
The count also forces the question of whether order matters. A sequence of two emails from five drafts has 20 versions; an unordered pair has 10. Mixing the two up either misses cases or wastes effort. Once the size is known, the choice is explicit: cover every case where full coverage matters, as in eligibility rules or safety checks, and where it does not fit, sample or prune the space on purpose, knowing exactly which cases are left untested.
The thinking behind it
Blaise Pascal Blaise Pascal, the French mathematician and philosopher, worked out the counting of combinations in his treatise on the arithmetical triangle, Traité du triangle arithmétique, written in 1654 and printed after his death in 1665. The triangle's rows give the number of ways to choose any number of items from a set. His correspondence with Pierre de Fermat the same year founded the mathematical theory of probability.
Jacob Bernoulli Jacob Bernoulli's Ars Conjectandi, published in Basel in 1713 after his death, treated permutations and combinations systematically as part of founding the theory of probability, and gave the first proof of a law of large numbers. It set out the counting rules, ordered and unordered selections, that any enumeration of cases still uses. Bernoulli, a Swiss mathematician, also found the constant now written e while working out a compound-interest question.
Leonhard Euler Leonhard Euler, in chapter 16 of Introductio in analysin infinitorum (1748), counted the ways a number can be written as a sum by expanding a product of factors as a power series, so that the coefficients did the counting. That method of generating functions is still how combinatorics counts large spaces without listing them one by one. For a test plan, it is the reason a count can be exact even when the list would be too long to write.
How to run it
- List the dimensions that vary independently. Write each thing that can change, such as price, bundle, region, channel, and message, and check that each can vary on its own.
- Count the values on each. Write how many options each dimension has in this test or rule set.
- Decide whether order matters. A sequence counts permutations and a set counts combinations; settle which one applies to every dimension.
- Multiply out the total. Multiply the counts, using permutations or combinations where a dimension is a selection, to get the size of the space.
- Compare it with capacity. Set the total against the cells, cases, or hours available, and see whether full coverage fits.
- Sample or prune on purpose. When it does not fit, drop dimensions, fix some values, or use a designed sample, and write down which cases remain untested.
Where it goes wrong
The common failure is counting by addition. A team adds a dimension and expects a little more work, when the space doubles. It shows up as a test that cannot reach its sample size, or a rule set with cases nobody checked. Multiply before committing.
A second failure is getting order wrong, counting a sequence as a set or the reverse. The first misses real cases; the second wastes effort on duplicates. Ask whether swapping two items would change the result. A third is dimensions that are not independent, such as a bundle that exists only in one region: the plain product overstates the space. List the impossible combinations and subtract them.
When to use something else
- Design of Experiments. When full coverage will not fit and the work is choosing a fraction of the combinations that still measures the main effects.
- Morphological Analysis. When the combinations are design options to generate ideas from, not cases to test.
- Constraint Satisfaction. When most combinations are ruled out by constraints and the job is finding the few that satisfy all of them.
Where it sits in SOLVE eX
Go to the source
- Traité du triangle arithmétique, written 1654, 1665
- Ars Conjectandi, 1713
- Introductio in analysin infinitorum, 1748