The ANOVA Was Significant. What Happens Next? Multiple Comparisons and Post-Hoc Tests

What Did the Omnibus ANOVA Actually Test?

For a one-way ANOVA with I population means, the omnibus null hypothesis is

H₀: μ₁ = μ₂ = ⋯ = μ_I.

The alternative says that the population means are not all equal. The ANOVA F statistic therefore tests the overall equality of the means rather than a collection of separate pairwise hypotheses (Devore & Berk, 2012; Moore et al., 2014, 2021).

Suppose there are four groups. A statistically significant omnibus test does not establish that:

• Group 1 differs from Group 2;
• Group 1 differs from Group 3;
• Group 1 differs from Group 4; or
• every possible pair of groups differs.

It establishes only that the equal-means null hypothesis is not supported at the chosen significance level (Moore et al., 2014, 2021).

This is why the answer to “ANOVA significant—what next?” is not “declare the largest and smallest means different.” Specific differences require specific inferential comparisons.

Why Not Just Run a Separate Test for Every Pair?

With I groups, the number of distinct pairwise comparisons is

I(I − 1) / 2.

The inferential problem changes as the number of comparisons grows. If every hypothesis in a family is tested independently at the ordinary significance level, the probability of making at least one false rejection somewhere in the family can exceed the error probability attached to any single test (Lovric, 2011).

This is the core problem behind multiple comparisons ANOVA. A collection of apparently ordinary pairwise tests is not necessarily equivalent to one ordinary test repeated several times: the conclusions are being made jointly.

First Decision: Were the Research Comparisons Known Before Analysis?

This question should come before choosing a multiple-comparison procedure.

Example of focused planned contrasts

With a control group and two interventions, the substantive hypotheses might have been:

1. Do the two interventions, averaged together, differ from control?
2. Do the two intervention versions differ from each other?

Those are two focused contrasts. They answer a different research plan from automatically examining all three possible pairwise comparisons.

Calling a comparison “post-hoc,” however, does not itself determine the correction. The appropriate method depends on what comparisons are being made.

Match the Procedure to the Comparisons You Actually Need

There is no single universal multiple comparison correction for every significant ANOVA. The sources describe several procedures with different inferential targets.

Tukey: when the target is all pairwise comparisons

Tukey’s method is designed for all pairwise comparisons among treatment or group means while controlling the FWER. The original method is formulated for equal sample sizes; the Tukey–Kramer extension is used conservatively when sample sizes differ (Lovric, 2011).

Devore and Berk (2012) likewise present Tukey’s procedure as a method for simultaneous inference about pairwise differences and describe its experimentwise or family error interpretation.

A Tukey post hoc test therefore maps naturally to the question:

Which pairs of group means differ when every pairwise comparison is of interest?

It is less natural when the research question concerns only a small number of prespecified comparisons or only comparisons with one control.

Dunnett: when every treatment is compared with one control

When the inferential structure is several treatments versus one common control, Dunnett’s method is specifically designed for those treatment-control comparisons (Devore & Berk, 2012; Lovric, 2011).

This distinction matters. If the scientific objective is “Does each intervention differ from placebo?”, comparisons between the active interventions may not belong to the research question. A procedure designed around a common control targets that structure directly rather than expanding the family to every possible pair.

Dmitrienko and Koch (2017) also describe Dunnett procedures as parametric multiple-testing methods that exploit the dependence among comparisons sharing a common control and can provide more powerful inference than a basic Bonferroni procedure in that setting.

Scheffé: when a broad range of contrasts may be examined

Scheffé’s method provides simultaneous inference for all possible contrasts among the group means, not merely the pairwise differences (Devore & Berk, 2012; Lovric, 2011).

That breadth comes with a cost. Lovric (2011) describes Scheffé’s procedure as too conservative when the actual goal is only all pairwise comparisons or comparisons with a control.

Scheffé is therefore most relevant when the analyst genuinely wants protection across a broad set of possible contrasts. Its flexibility should not be confused with greater efficiency for a narrow question.

Bonferroni: when there is a fixed collection of comparisons

For m comparisons, a Bonferroni procedure can test each at level α/m. The method controls the FWER through the Bonferroni inequality and can be applied to a fixed collection of comparisons (Lovric, 2011).

Moore et al. (2021) illustrate this logic for ANOVA pairwise comparisons: if ten comparisons belong to a family with overall α = .05, the Bonferroni approach allocates .05/10 = .005 to each individual test.

Bonferroni is simple and broadly applicable, but simplicity does not imply optimal power. Both Lovric (2011) and Dmitrienko and Koch (2017) describe more specialized or stepwise procedures that can use the structure of a multiple-testing problem more efficiently.

Least significant difference: why repeated ordinary tests are risky

Moore et al. (2021) describe the least-significant-difference approach as using the ordinary pairwise t-test critical value for each comparison. Because it controls the error probability for an individual comparison rather than the probability of any false rejection across the whole family, its overall false-rejection probability can become problematic when many means are compared (Moore et al., 2021).

Error Control and Power Pull in Opposite Directions

Multiplicity protection is not free.

Procedures that make it harder to produce at least one false positive across a family generally require stronger evidence for individual rejections. The International Encyclopedia of Statistical Science explicitly describes a trade-off between FWER control and statistical power for simultaneous or single-step procedures (Lovric, 2011).

This is why the choice of procedure should follow the research question.

A researcher interested only in two prespecified contrasts does not automatically gain anything scientifically by testing every possible pair and correcting across the larger collection. Conversely, a researcher who explores every pairwise difference should not report each ordinary p < .05 result as though no other comparisons were performed. The inferential family should reflect the conclusions that the analysis is intended to support (Dmitrienko & Koch, 2017; Lovric, 2011).

Specialized procedures can also improve efficiency by using the comparison structure. Dunnett procedures, for example, use the dependence generated by several comparisons with a common control, whereas a basic Bonferroni calculation does not exploit that correlation structure (Dmitrienko & Koch, 2017).

The relevant question is therefore not:

Which correction gives me the smallest adjusted p-values?

It is:

Which procedure matches the hypotheses and error-control objective that define this analysis?

How to Interpret Adjusted Inference

Multiplicity adjustment changes the inferential standard. It does not change the observed group means or the estimated mean difference itself.

Dmitrienko and Koch (2017) define an adjusted p-value as the smallest overall significance level at which the corresponding hypothesis would be rejected under the specified multiple-testing procedure.

Prefer adjusted confidence intervals when available

Multiple-comparison procedures can also be expressed through simultaneous confidence intervals. These intervals provide joint coverage across the collection of comparisons rather than treating each interval as an isolated statement (Devore & Berk, 2012; Moore et al., 2021).

Moore et al. (2021) note that simultaneous intervals corresponding to a multiple-comparison procedure are directly connected to its testing decisions: if the interval for a mean difference includes zero, that comparison is not declared statistically different under the same procedure.

Because simultaneous coverage is stronger than isolated interval coverage, multiplicity-adjusted intervals are generally wider than corresponding unadjusted intervals (Moore et al., 2021).

That wider interval should be interpreted as the price of making several inferential statements with joint error protection—not as evidence that the observed difference itself became smaller.

A Practical Workflow After a Significant ANOVA

Use the following sequence rather than automatically clicking a post-hoc option.

A Compact Decision Guide

The table is a map from research questions to procedures, not a universal ranking of methods. The inferential structure should determine the method.

Common Reporting Mistakes

What Should You Report?

A clear ANOVA follow-up report should allow a reader to reconstruct the inferential logic. Report the omnibus hypothesis and result, specify whether the follow-up comparisons were planned or exploratory, identify exactly which comparisons formed the family, name the multiplicity procedure, and present estimated differences with adjusted p-values and/or simultaneous confidence intervals where appropriate (Dmitrienko & Koch, 2017; Moore et al., 2021).

For example:

The omnibus ANOVA provided evidence that the population means were not all equal. Because all pairwise group differences were of substantive interest, pairwise comparisons were evaluated using a Tukey procedure with family-wise error control. Estimated mean differences and multiplicity-adjusted confidence intervals were reported for each pair.

If only two hypotheses had been specified in advance, the reporting should instead describe those two contrasts and the multiplicity strategy used for that planned family.

Conclusion

A significant ANOVA is a gateway to more specific questions, not an answer to all of them.

The omnibus F test asks whether all population means are equal. Planned contrasts address specific hypotheses. Exploratory post-hoc comparisons identify more detailed patterns. Once multiple inferential claims are made, the analysis must also consider the error rate across the relevant family of hypotheses (Devore & Berk, 2012; Lovric, 2011; Moore et al., 2021).

For researchers asking “ANOVA significant—what next?”, the practical sequence is:

hypothesis → comparison structure → number and family of tests → multiplicity method → adjusted inference → reporting.

The best post hoc test after ANOVA is therefore not determined by ANOVA significance alone. It is determined by the scientific questions the researcher intends to answer.

References

Devore, J. L., & Berk, K. N. (2012). Modern mathematical statistics with applications (2nd ed.). Springer.

Dmitrienko, A., & Koch, G. G. (Eds.). (2017). Analysis of clinical trials using SAS: A practical guide (2nd ed.). SAS Institute.

Lovric, M. (Ed.). (2011). International encyclopedia of statistical science. Springer. https://doi.org/10.1007/978-3-642-04898-2

Moore, D. S., McCabe, G. P., & Craig, B. A. (2014). Introduction to the practice of statistics (8th ed.). W. H. Freeman.

Moore, D. S., McCabe, G. P., & Craig, B. A. (2021). Introduction to the practice of statistics (10th ed.). Macmillan Learning.