Multiple Comparisons: Why Testing 50 Outcomes Can Produce a ‘Significant’ Result by Chance

A “significant” result is much less persuasive when it was selected from dozens of outcomes, time points, models, or subgroups. Credibility depends on the full set of analytical opportunities and how they were controlled.

Family-wise error control limits the chance of one or more false-positive conclusions within a defined family; false discovery rate control limits the expected proportion of false positives among declared findings. They answer different questions.

Positive secondary outcomes are strongest when prespecified and covered by the multiplicity plan. Otherwise, they are generally better treated as hypotheses requiring confirmation.

For subgroup claims, compare effects with a formal interaction test and confidence interval. “Significant” in one subgroup but not another does not prove that the effects differ.

Testing 50 outcomes creates 50 opportunities to find something that looks statistically significant. If all 50 null hypotheses are true, the tests are independent, and each uses a 0.05 threshold, the probability of obtaining at least one false positive is about 92.3%. That does not mean every study with 50 tests is wrong. It means a lone p-value below 0.05 is weak evidence unless readers know how many analytical opportunities existed and how the study controlled them.

The problem is called multiplicity or multiple comparisons. It can arise from testing many outcomes, doses, time points, treatment groups, interim analyses, statistical models, or patient subgroups. The relevant number is not always the count of variables in the main results table; it is the broader set of opportunities to make the claim under consideration. The FDA guidance on multiple endpoints in clinical trials explains how these choices can increase the chance of a false-positive conclusion.

Why the usual 0.05 threshold stops meaning what readers think

A type I error is a false positive: rejecting a null hypothesis even though it is true. A significance threshold of 0.05 limits that error probability to 5% for one test under the assumptions of the analysis. It does not automatically limit the probability of any false positive to 5% across an entire collection of tests.

For m independent tests in which every null hypothesis is true, the probability of at least one p-value below 0.05 is 1 − (1 − 0.05)^m. The increase is steep:

Independent null tests	Chance of at least one false positive
1	5.0%
2	9.8%
10	40.1%
20	64.2%
50	92.3%

The 92.3% figure is an illustration, not a universal estimate for studies reporting 50 outcomes. Medical outcomes are often correlated: blood pressure measured at several visits, for example, does not produce independent tests. Correlation changes the exact probability. Nevertheless, more analytical opportunities generally create more ways for chance variation to cross a significance threshold.

There is another important distinction. A 92.3% chance of at least one false positive does not mean 92.3% of the 50 results will be false positives. Under this simplified setup, the expected number crossing the 0.05 threshold is 50 × 0.05, or 2.5. The 92.3% figure answers a different question: how likely is it that the collection contains one or more false alarms?

Family-wise error and false discovery rate answer different questions

Multiplicity methods depend on what error a study is trying to control. The family-wise error rate, or FWER, is the probability of making at least one false-positive conclusion within a defined family of tests. Confirmatory clinical trials commonly control FWER across primary and key secondary endpoints used to support efficacy claims.

The word “family” matters. Researchers must identify which hypotheses belong together for the intended claim. Testing every endpoint family separately at 0.05 can leave the overall study-wide error rate inflated. A correction applied to five primary endpoints also does not automatically protect an unrelated collection of 30 subgroup analyses.

The false discovery rate, or FDR, addresses a different quantity: the expected proportion of false positives among results declared positive. The Benjamini–Hochberg procedure was developed to control FDR under specified conditions and can retain more power than simple Bonferroni correction when many hypotheses are screened. The original Benjamini–Hochberg paper describes this framework.

FDR control is often useful in broad exploratory work, where finding promising signals may matter more than ensuring almost no false-positive claims appear. But a result surviving an FDR procedure is not automatically confirmation of a clinical benefit. It may still be a signal requiring independent validation in a study designed around that specific hypothesis.

Common ways to control multiplicity

Bonferroni correction

Bonferroni divides the available significance level among the tests. With two equally weighted endpoints and an overall two-sided alpha of 0.05, each could be tested at 0.025. With 50 equally weighted tests, the threshold would be 0.001. The method is broadly applicable, but it can be conservative, particularly when there are many tests or the tests are correlated.

Holm’s step-down procedure

Holm’s method orders the p-values from smallest to largest and compares them with progressively less stringent thresholds. It controls family-wise error while generally offering more power than ordinary Bonferroni. FDA guidance notes that it can be used regardless of the degree of correlation among endpoints.

Fixed-sequence testing and gatekeeping

A fixed-sequence plan tests hypotheses in a prespecified order. Testing moves to the next endpoint only if the earlier endpoint succeeds. This can preserve alpha without dividing it equally among every endpoint, but a failure early in the sequence prevents later results from receiving confirmatory status under that plan—even if their nominal p-values are below 0.05.

Gatekeeping extends this logic to groups of hypotheses. For example, primary endpoints may form one family and secondary endpoints another. Preplanned rules determine whether and how testing proceeds or unused alpha transfers between families. The order must be set before the results are known; choosing a favorable order afterward defeats the protection.

Why a positive secondary outcome may still be exploratory

A primary outcome addresses the trial’s central question. Secondary outcomes add information, but “secondary” does not mean that every one was tested confirmatorily. A statistically significant secondary result may not support a confirmatory claim if it was outside the prespecified multiplicity strategy.

Prespecification helps because it separates planned tests from choices made after the data were visible. It does not guarantee that a study is unbiased or well designed, but it makes selective emphasis harder. Readers can learn more about that distinction in this guide to preregistration in medical research. Reporting standards also call for authors to identify whether additional analyses were prespecified or post hoc.

Subgroup fishing creates a second layer of risk

Subgroup analyses ask whether an effect differs by characteristics such as age, sex, baseline disease severity, or biomarker status. They are especially vulnerable to unstable patterns because splitting a study reduces the effective sample size in each group, while testing many possible splits multiplies opportunities for chance findings. Even prespecified subgroup analyses can be underpowered for the comparison that matters.

A common error is to report that treatment worked in one subgroup because p 0.05. “Significant” in one group and “not significant” in another does not establish that the treatment effects differ. The appropriate evidence usually comes from a formal interaction test, which directly estimates whether the effect changes between groups. The Cochrane Handbook’s subgroup guidance recommends examining interaction estimates and limiting subgroup analyses to a small number supported by a scientific rationale.

Confidence intervals are crucial. They show whether the estimated interaction is precise and whether the range of compatible effects includes clinically unimportant differences. A very small interaction p-value may still describe a trivial difference, while a potentially important difference with a wide interval may be too uncertain to guide care.

Subgroups defined after treatment begins require further caution. If treatment can influence the characteristic used to assign subgroup membership, the comparison can introduce selection bias in addition to the ordinary multiple-testing problem.

How to spot an overinterpreted finding

When a headline rests on a secondary outcome or subgroup, work through these questions:

Was the result for the prespecified primary outcome, a key secondary outcome, or an exploratory analysis?
How many outcomes, time points, doses, models, and subgroups were examined—not merely reported?
Was the specific analysis prespecified in a protocol, registry entry, or statistical analysis plan?
What family of hypotheses did the multiplicity procedure cover? A correction elsewhere in the study may not cover this claim.
Did the authors use family-wise error control, false discovery rate control, or only unadjusted p-values?
For a subgroup claim, was there a formal interaction test with an effect estimate and confidence interval?
Is the effect large enough to matter clinically, or is it only statistically distinguishable from zero?
Was the finding replicated in an independent dataset or a later confirmatory study?

Phrases such as “nominally significant,” “exploratory,” “post hoc,” and “not adjusted for multiple comparisons” are not automatic reasons to discard a result. They are signals to lower the strength of the conclusion. Exploratory analyses can uncover useful hypotheses, rare safety patterns, or effects worth testing next. The mistake is presenting discovery and confirmation as though they were the same stage of evidence.

What multiplicity correction cannot fix

A corrected p-value addresses a specific false-positive problem. It does not repair confounding, biased outcome measurement, missing data, selective nonreporting, unsuitable statistical models, sparse event counts, or poor randomization. Nor does it turn a small biomarker shift into a meaningful improvement in symptoms, function, hospitalization, or survival.

Correction also involves a tradeoff. Stricter thresholds reduce false-positive claims but can increase false negatives, especially in small studies. The answer is not to avoid correction whenever power is limited. It is to design studies around a focused set of clinically important questions, plan the testing strategy in advance, and label additional analyses according to their exploratory role.

The defensible takeaway

Multiple comparisons do not make a medical study invalid. They change what a “significant” result can support. A lone p-value below 0.05 is considerably less persuasive when it emerged from dozens of outcomes, time points, models, or subgroups than when it came from a prespecified primary test with appropriate error control.

Readers should look beyond the smallest p-value. The strongest findings have a clearly defined hypothesis family, a prespecified analysis, an error-control method suited to the claim, an effect size with a useful confidence interval, and conclusions that respect the difference between exploratory signals and confirmatory evidence. When those pieces are missing, the finding may be worth following—but not yet worth treating as established.