16. Post-Hoc Tests and Multiple Comparisons

a. Multiple Comparisons for Means
Math 115

Yurk

Vocabulary: Where We Left Off

Do mean vocabulary scores differ across social class — and if so, which classes? In 15b, ANOVA on the GSS vocabulary data (k = 4 classes, n = 795) gave F = 21.73, p < 0.001 — we rejected H_0: \mu_1 = \mu_2 = \mu_3 = \mu_4: at least one class mean differs.

Class n_i \bar{x}_i s_i
Lower 41 5.07 2.24
Working 407 5.75 1.87
Middle 331 6.76 1.89
Upper 16 6.19 2.34

ANOVA Said “At Least One Differs” — Which?

ANOVA gives a single yes/no on the whole table — it says some class mean differs, not which ones.

The open questions:

  • Is Middle significantly different from Lower? From Working? From Upper?
  • Does Lower differ from Working?
  • Do Working and Upper differ?

To pinpoint which pairs differ, we need pairwise comparisons.

How Many Pairwise Comparisons?

The natural follow-up to a significant ANOVA: run a separate t-test for each pair of groups.

For k groups, the number of pairs is

a = \frac{k(k-1)}{2}.

Groups (k) Pairwise tests (a)
3 3
4 6
5 10
6 15

For our k = 4 classes, a = \dfrac{4 \times 3}{2} = 6 tests. The more groups, the more tests — and the more chances to make a false rejection.

The Multiple Testing Problem

Each test at \alpha = 0.05 has its own 5% chance of a Type I error when its null is true. Run several tests and those chances accumulate.

Familywise error rate (FWE): the probability of at least one false rejection across the whole family of tests, when every null is true.

With a tests, the FWE grows quickly:

\text{FWE} \approx 1 - (1 - \alpha)^a

Tests (a) Familywise error rate
1 5%
6 ≈ 26%
10 ≈ 40%
20 ≈ 64%

Same inflation pitfall from 15a — exactly what ANOVA’s single test avoided. Now we do want the pairwise tests, so we correct to keep the FWE at \alpha.

The Bonferroni Correction

The fix: divide the significance level by the number of tests.

For a pairwise comparisons at familywise level \alpha, each individual test uses the adjusted significance level

\alpha^* = \frac{\alpha}{a}.

Each unadjusted p-value must now clear this stricter level — not the original \alpha — to count as significant.

Why it works. Each individual test now has at most an \alpha/a chance of a false rejection. Across a tests, the chance that at least one false rejection occurs is at most a \times \dfrac{\alpha}{a} = \alpha.

Conservative on purpose. Bonferroni guarantees the familywise error rate is at most \alpha; the actual rate is often lower. The trade-off: lower per-test power, so real differences can be missed.

The Six Pairwise Comparisons

The a = 6 pairs for our four classes:

  • Working vs Lower    - Middle vs Lower    - Upper vs Lower
  • Middle vs Working    - Upper vs Working    - Upper vs Middle

Conditions: no new check needed. These tests pool the SD across all four groups — the same pooling we used for two-group t-tests in topic 12 — so they rest on the ANOVA conditions already verified in 15b.

Try it: with 6 comparisons at familywise \alpha = 0.05, what’s the adjusted significance level \alpha^*? \alpha^* = \dfrac{0.05}{6} \approx 0.0083 — each unadjusted p-value must clear this stricter level to count as significant.

Applying the Adjusted Significance Level

Comparison Unadjusted p p < α* = 0.0083?
Working − Lower 0.031
Middle − Lower 1.1e-07 ✓
Upper − Lower 0.048
Middle − Working 1.6e-12 ✓
Upper − Working 0.367
Upper − Middle 0.24

Each unadjusted p-value is compared to the adjusted level \alpha^* = 0.05 / 6 \approx 0.0083.

Two pairs survive correction: Middle vs Lower and Middle vs Working. Two more (Working vs Lower, Upper vs Lower) would have been significant at an uncorrected 0.05 — the correction is what rules them out.

Adjusting the p-Values Instead

Same correction, rearranged. Rather than pulling the significance level down to \alpha^*, push each p-value up:

p_{\text{adj}} = a \times p \quad \text{(capped at 1)},

then compare to the original \alpha.

These are the same comparison, just written two ways:

p < \frac{\alpha}{a} \quad\Longleftrightarrow\quad a \times p < \alpha.

This is the form software reports. Jamovi prints adjusted p-values, so its output is compared directly to \alpha = 0.05.

The Same Six Tests, Adjusted p-Values

Comparison Unadjusted p Adjusted p Significant at α = 0.05?
Working − Lower 0.031 0.183
Middle − Lower 1.1e-07 6.8e-07 ✓
Upper − Lower 0.048 0.285
Middle − Working 1.6e-12 9.8e-12 ✓
Upper − Working 0.367 1
Upper − Middle 0.24 1

Adjusted p = unadjusted \times\ 6 (capped at 1), compared to the usual \alpha = 0.05.

Same two pairs, same conclusion. The bookkeeping changed; the answer didn’t.

Interpretation

What we can say (at familywise \alpha = 0.05):

  • Middle class has a significantly higher mean vocabulary score than Lower class (\bar{x}_{\text{Middle}} \approx 6.76 vs 5.07).
  • Middle class also scores significantly higher than Working class (6.76 vs 5.75).

What we can’t say:

  • No significant difference between Lower and Working, or between any pair involving Upper. Upper’s sample mean (6.19) actually sits well above Lower’s (5.07) — a larger raw gap than Middle vs Working — yet with only n = 16 Upper respondents the test has too little power to call it significant. Absence of evidence isn’t evidence of absence.

Five Groups

A researcher compares 5 groups with ANOVA, rejects H_0, and wants all pairwise comparisons.

  1. How many pairwise tests are there? a = \dfrac{5 \times 4}{2} = 10

  2. Without correction, what is the familywise error rate at \alpha = 0.05 (if all 10 null hypotheses were true)? 1 - 0.95^{10} \approx 0.40 — about a 40% chance of at least one false rejection.

  3. What is the adjusted significance level \alpha^*? \alpha^* = 0.05 / 10 = 0.005

  4. Two pairs give unadjusted p-values of 0.004 and 0.012. Which are significant? 0.004 < 0.005 ✓    0.012 > 0.005 ✗

The Same Two p-Values

The same two results — unadjusted p-values of 0.004 and 0.012 — with a = 10 comparisons at \alpha = 0.05.

  1. Now decide using adjusted p-values instead. 0.004: 10 \times 0.004 = 0.04 < 0.05 ✓    0.012: 10 \times 0.012 = 0.12 > 0.05 ✗

Same verdicts as question 4. The two forms are the same comparison, so they always agree. Note that 0.012 would have been significant at an uncorrected \alpha = 0.05.

Beyond Bonferroni

Bonferroni is the simplest correction, but its conservatism costs power — with many tests, real differences can be missed.

Less conservative alternatives give up less power. Software reports them routinely, but they are harder to compute by hand:

  • Tukey’s HSD — all-pairs comparisons of group means after ANOVA only.
  • Holm’s procedure — works for any set of tests, like Bonferroni.

For this course we’ll default to Bonferroni, but be aware the others exist — you will see them in software output.

Looking Ahead

After a significant ANOVA, we can now compare the groups two at a time, correcting for how many comparisons we run.

Next time (16b): the same idea for proportions — following up a significant chi-square test by comparing groups two at a time, again with a Bonferroni correction.

References

Course Home