Thinking Tools3.1 Finding the Current FactsAlso known as Evidence pyramid
From the levels of evidence first published by the Canadian Task Force on the Periodic Health Examination (1979), ranked by David L. Sackett (1986) and carried into evidence-based medicine by Sackett and colleagues (1996).
Ranks kinds of evidence from pooled randomized trials at the top to expert opinion at the bottom, so competing claims are weighed by how well their support rules out confounding.
Eight pieces of evidence on whether remote work raises output, placed by strength
A leadership team had to decide on a remote-work policy, and every side brought evidence. Place each piece on its level; the higher the level, the more of the other explanations it rules out. The verdict rests on the top levels and uses the lower ones only where nothing better exists. The first piece is placed as an example. Illustrative.
Items (select one, then a box)
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Not placedPlacedReviews of many trialsRandomized trialsObservational studiesOpinion and single cases
Strength
Trial reviewsRandomized trialsObservationalOpinion, one case
Reviews of many trials
Randomized trials
Observational studies
Opinion and single cases
Sources
Evidence that remote work raises output
CEO's keynote. A well-known chief executive says remote work kills productivity.Not placed yet.CEO's keynote. A well-known chief executive says remote work kills productivity.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.CEO's keynote. A well-known chief executive says remote work kills productivity.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.CEO's keynote. A well-known chief executive says remote work kills productivity.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.CEO's keynote. A well-known chief executive says remote work kills productivity.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.One person's view, however senior. It rules out nothing.Review of eleven trials. A review pooling eleven randomized trials finds gains of 3 to 5 percent for individual focused work, and none for team tasks.Not placed yet.Review of eleven trials. A review pooling eleven randomized trials finds gains of 3 to 5 percent for individual focused work, and none for team tasks.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.The strongest evidence on the table, and it splits the answer by kind of work.Review of eleven trials. A review pooling eleven randomized trials finds gains of 3 to 5 percent for individual focused work, and none for team tasks.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Review of eleven trials. A review pooling eleven randomized trials finds gains of 3 to 5 percent for individual focused work, and none for team tasks.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Review of eleven trials. A review pooling eleven randomized trials finds gains of 3 to 5 percent for individual focused work, and none for team tasks.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.Call center trial. Staff were assigned to home or office by lottery for nine months; calls handled rose 8 percent at home.Not placed yet.Call center trial. Staff were assigned to home or office by lottery for nine months; calls handled rose 8 percent at home.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Call center trial. Staff were assigned to home or office by lottery for nine months; calls handled rose 8 percent at home.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Assignment by lottery rules out self-selection. A strong result, for one kind of work.Call center trial. Staff were assigned to home or office by lottery for nine months; calls handled rose 8 percent at home.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Call center trial. Staff were assigned to home or office by lottery for nine months; calls handled rose 8 percent at home.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.Software firm trial. Engineers split by lottery for six months; code shipped was flat, and new hires ramped more slowly at home.Not placed yet.Software firm trial. Engineers split by lottery for six months; code shipped was flat, and new hires ramped more slowly at home.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Software firm trial. Engineers split by lottery for six months; code shipped was flat, and new hires ramped more slowly at home.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Also randomized. It agrees with the review: no gain for collaborative work.Software firm trial. Engineers split by lottery for six months; code shipped was flat, and new hires ramped more slowly at home.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Software firm trial. Engineers split by lottery for six months; code shipped was flat, and new hires ramped more slowly at home.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.Forty firms over 3 years. Firms that went remote showed higher revenue per employee than firms that did not.Not placed yet.Forty firms over 3 years. Firms that went remote showed higher revenue per employee than firms that did not.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Forty firms over 3 years. Firms that went remote showed higher revenue per employee than firms that did not.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Forty firms over 3 years. Firms that went remote showed higher revenue per employee than firms that did not.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.A pattern, but the firms that went remote may have been different already.Forty firms over 3 years. Firms that went remote showed higher revenue per employee than firms that did not.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.Survey of 2,000 staff. Most staff say they get more done at home.Not placed yet.Survey of 2,000 staff. Most staff say they get more done at home.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Survey of 2,000 staff. Most staff say they get more done at home.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Survey of 2,000 staff. Most staff say they get more done at home.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Self-reported and observational: it measures belief about output, not output.Survey of 2,000 staff. Most staff say they get more done at home.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.Our best quarter yet. The sales team had a record quarter in its first remote year.Not placed yet.Our best quarter yet. The sales team had a record quarter in its first remote year.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Our best quarter yet. The sales team had a record quarter in its first remote year.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Our best quarter yet. The sales team had a record quarter in its first remote year.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Our best quarter yet. The sales team had a record quarter in its first remote year.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.One case, in one market. It may reflect the market as much as the policy.Vendor white paper. A collaboration software vendor reports large gains among its customers.Not placed yet.Vendor white paper. A collaboration software vendor reports large gains among its customers.Reviews of many trials: Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.Vendor white paper. A collaboration software vendor reports large gains among its customers.Randomized trials: People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.Vendor white paper. A collaboration software vendor reports large gains among its customers.Observational studies: Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.Vendor white paper. A collaboration software vendor reports large gains among its customers.Opinion and single cases: A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.A seller's account of its own customers: opinion, with a reason to be generous.
Your sort matched 0 of 8.Your sort matched 1 of 8.Your sort matched 2 of 8.Your sort matched 3 of 8.Your sort matched 4 of 8.Your sort matched 5 of 8.Your sort matched 6 of 8.Your sort matched 7 of 8.Your sort matched 8 of 8.
Reviews of many trials
Pools every trial on the question. Rules out the luck of a single study, if the trials it pools are sound.
Randomized trials
People assigned to remote or office by lottery. Rules out the people who chose remote being different to begin with.
Observational studies
Groups compared as they were, without assignment. Shows a pattern, but cannot rule out that the groups differed for other reasons.
Opinion and single cases
A view, a story, or one team's result. Rules out nothing; useful for questions to ask, not for answers.
The sort
CEO's keynote: Opinion and single cases. One person's view, however senior. It rules out nothing.
Review of eleven trials: Reviews of many trials. The strongest evidence on the table, and it splits the answer by kind of work.
Call center trial: Randomized trials. Assignment by lottery rules out self-selection. A strong result, for one kind of work.
Software firm trial: Randomized trials. Also randomized. It agrees with the review: no gain for collaborative work.
Forty firms over 3 years: Observational studies. A pattern, but the firms that went remote may have been different already.
Survey of 2,000 staff: Observational studies. Self-reported and observational: it measures belief about output, not output.
Our best quarter yet: Opinion and single cases. One case, in one market. It may reflect the market as much as the policy.
Vendor white paper: Opinion and single cases. A seller's account of its own customers: opinion, with a reason to be generous.
An example
A professional services firm had to settle its remote-work policy, and both camps arrived with evidence. The office camp had a famous chief executive's keynote. The remote camp had a vendor's report, a staff survey, a record quarter in sales, and a study of forty firms whose revenue per employee rose after they went remote.
The chief operating officer asked for every piece to be placed on one chart, by how much of the other explanations it ruled out. The keynote, the record quarter, and the vendor paper went to the bottom. The survey and the forty-firm study went one level up: patterns, but the firms and staff who chose remote may have differed already. Two randomized trials sat above them, and one review that pooled eleven trials sat at the top. The top levels agreed with each other: modest gains for individual focused work, none for team tasks, and slower starts for new hires. The policy followed that line: home days for focused work, and the first six months in the office for new staff.
Why it works
When people argue from evidence, they rarely argue about the same kind. A story, a survey, and a controlled trial get weighed as equals, and the most vivid one wins. Ranking the kinds of evidence by how many other explanations each rules out gives the room a shared standard before anyone looks at what the pieces say.
The ranking follows one question: could something other than the cause in question have produced this result? Opinion cannot answer it. An observational comparison narrows it, but the groups may differ for other reasons. Assigning people by lottery removes that, and pooling many trials removes the luck of any one. Weighing the top more heavily does not discard the rest; it sends the lower levels to work as sources of questions.
The thinking behind it
David L. Sackett David L. Sackett, a founder of clinical epidemiology at McMaster University, set out rules of evidence for grading clinical recommendations in Rules of Evidence and Clinical Recommendations on the Use of Antithrombotic Agents (1986, Chest). The first published levels of evidence had come from the Canadian Task Force on the Periodic Health Examination in 1979; Sackett ranked them, and with Gordon Guyatt later published a hierarchy that put systematic reviews and randomized trials at the top.
William M. C. Rosenberg William M. C. Rosenberg wrote, with David Sackett, J. A. Muir Gray, R. Brian Haynes, and W. Scott Richardson, a 1996 editorial in the BMJ that defined evidence-based medicine as the conscientious use of the best current evidence together with clinical expertise and patient values, in decisions about individual patients. The definition is a guard against using the pyramid mechanically: the best evidence informs the judgment, and does not replace it.
How to run it
State the claim. Write it as a cause and an effect, such as remote work raises output, and say whether a cause is being claimed at all.
Gather every piece on the table. Include the ones each side would prefer to leave out.
Place each piece on its level. Ask how the groups were formed: by lottery, by choice, or not at all.
Judge quality within each level. A large, well-run observational study can outweigh a tiny, flawed trial; note size, length, and who paid.
Check fit to your case. Ask whether the people, the work, and the setting match yours closely enough to carry over.
Read the top levels first. Write the verdict from the strongest evidence that fits, and use the lower levels to add questions.
Look for agreement and conflict. Note where studies replicate and where they disagree, and say what would settle the disagreement.
Where it goes wrong
The level is read as quality. A randomized trial with forty people and a short run can be weaker than a careful study of forty firms. Judge each piece within its level before letting its level decide.
The top evidence does not fit the case. Trials of call center staff say little about a design studio. When the best evidence is about different people or work, say so, and give it less weight.
The absence of a trial becomes a reason to do nothing. Many business questions will never have one. Decide on the best evidence available, state how strong it is, and plan a test of your own where you can.
When to use something else
Bradford Hill Criteria. When there are no trials, and the question is whether an observed association is likely to be causal.
Randomized Controlled Trial. When the firm can run its own controlled test instead of weighing other people's.
Bayesian Updating. When the evidence arrives over time and each new piece should shift the belief by its strength.
The Periodic Health Examination, Canadian Medical Association Journal 121 (Canadian Task Force on the Periodic Health Examination), 1979
Rules of Evidence and Clinical Recommendations on the Use of Antithrombotic Agents, Chest, 1986
Evidence Based Medicine: What It Is and What It Isn't, BMJ (David L. Sackett, William M. C. Rosenberg, J. A. Muir Gray, R. Brian Haynes, and W. Scott Richardson), 1996