Construct Validity

Thinking Tools5.1 Defining the Evaluation CriteriaAlso known as Multitrait-multimethod matrix

Defined by Lee Cronbach and Paul Meehl (Construct Validity in Psychological Tests, 1955), with the multitrait-multimethod matrix of Donald Campbell and Donald Fiske (1959).

Tests whether a measure captures the concept it is named for by checking its content, what it predicts, what it agrees with, and what it should not track.

A proposed engagement score against four tests of whether it measures engagement

HR wants to put a three-question pulse score on every manager's scorecard. Each line is one test with its evidence. Decide whether the score passes or fails it, then select the line to see. One failure is enough to revise the score before it reaches 2,000 people. Illustrative.

Select each item to see the verdict.

  • Half the concept is missing. Nothing asks about effort, so a team that wants to stay and has stopped trying scores well. Add questions on effort before anything else.

  • Engagement should predict leaving, and the score does, by a wide margin, on outcomes recorded after the score was taken.

  • A different method aimed at the same concept gives a similar ranking of teams, which is what a valid measure should do.

  • The score moves with team size while turnover does not, so part of it measures smallness. On a scorecard it would reward managers for small teams. Adjust for size or find out why small teams answer differently.

Tests passed
2 of 4
Tests failed: any one means revise before scaling
2 of 4

An example

The HR team at a 2,000-person logistics company wanted every manager's quarterly scorecard to carry an engagement score: three pulse questions, asked monthly, averaged by team. The head of people analytics asked for one month to test it before it went live.

She wrote down what the company meant by engagement (the effort people choose to put in, and their attachment to the team) and ran four checks. Two passed. Teams in the bottom fifth lost 19 percent of their people within the next year, against 6 percent in the top fifth, and the score ranked 140 teams much as skip-level managers' independent ratings did. Two failed. All three questions asked about staying and having the right tools, and none touched effort. Small teams also scored 15 points higher than large ones with no difference in turnover. The score went on the scorecard a quarter later, with two effort questions added and an adjustment for team size, and the managers of large teams stopped asking to split them.

Why it works

Every measure has a name, and the name makes a promise the numbers may not keep. Once a score sits on a scorecard, people work to raise it, and whatever the score actually tracks is what they will raise. If it tracks team size, the organization gets small teams; if it tracks intent to stay, it gets contented teams that coast.

The tests catch the gap before it is paid for. Coverage asks whether the questions reach the whole concept. Prediction asks whether the score foretells what the concept should foretell. Agreement asks whether a different method gives the same answer. Independence asks whether the score stays away from things it should not track. Each test can fail on its own, and each failure names a specific repair.

The thinking behind it

Lee J. Cronbach Lee Cronbach, an educational psychologist, defined construct validity with Paul Meehl in Construct Validity in Psychological Tests (1955), in the Psychological Bulletin. Their problem was a test of a trait that cannot be observed directly and has no single yardstick to check it against. Their answer was that such a test is validated through a network of predicted relations between the trait and other measures; each prediction that holds supports both the test and the theory behind it.

Paul E. Meehl Paul Meehl, a clinical psychologist, was the paper's coauthor. He had already published Clinical versus Statistical Prediction (1954), which found that simple formulas usually predict as well as expert judgment or better, a habit of checking claims against outcomes that the 1955 paper shares. Construct validity made the meaning of a measure an empirical question: what the score is for is settled by what it predicts and relates to.

Donald T. Campbell Donald Campbell, a methodologist, and Donald Fiske proposed the multitrait-multimethod matrix in Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix (1959). A valid measure agrees with other measures of the same trait taken by different methods, and does not agree too closely with measures of different traits. The second half guards against a common trap: two scores that agree only because they were collected the same way.

How to run it

  1. Define the concept in words. Write what the measure is meant to capture so a stranger could apply it, including its parts.
  2. Specify how the number is made. Name the questions or data, the source, the period, and the arithmetic.
  3. Check coverage. Map each part of the definition to the questions or data that reach it, and mark the parts nothing reaches.
  4. Check prediction. Test whether the score foretells an outcome the concept should drive, using outcomes recorded after the score.
  5. Check agreement. Compare the score with a different measure of the same concept, collected another way.
  6. Check independence. Test it against things it should not track, such as team size, region, or who collected the data.
  7. Revise before scaling. Fix each failure, then rerun the failed test before the score is tied to pay or targets.

Where it goes wrong

The common failure is accepting a single good correlation. A score that predicts turnover can still miss half the concept or ride on something irrelevant. Run all four tests; passing one says little about the others.

The second is testing with the same method. A pulse score that agrees with another pulse score may only show that people answer surveys consistently. Agreement should come from a different kind of evidence, such as exit records or manager ratings.

The third is checking once and never again. A measure tied to pay changes how people answer it, and a score that was valid before it counted may drift once it does. Retest after the first two cycles on the scorecard, and watch for scores that rise while the outcomes they predicted stay flat.

When to use something else

  • Goodhart-Aware Metric Selection. When the measure is valid today and the worry is how people will game it once targets are attached.
  • KPI Design. When the work is choosing which few measures a team should run on, before testing any one of them.
  • Design of Experiments. When the question is whether a change caused an outcome, and the measure itself is already trusted.

Where it sits in SOLVE eX

Go to the source

  • Construct Validity in Psychological Tests, Psychological Bulletin 52, 1955
  • Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix, Psychological Bulletin 56, 1959

Back to the catalog