Math 115: Introduction to Statistics – 16. Post-Hoc Tests and Multiple Comparisons

Vocabulary: Where We Left Off

Do mean vocabulary scores differ across social class — and if so, which classes? In 15b, ANOVA on the GSS vocabulary data (\(k = 4\) classes, \(n = 795\)) gave \(F = 21.73\), p \(< 0.001\) — we rejected \(H_0: \mu_1 = \mu_2 = \mu_3 = \mu_4\): at least one class mean differs.

Class	\(n_i\)	\(\bar{x}_i\)	\(s_i\)
Lower	41	5.07	2.24
Working	407	5.75	1.87
Middle	331	6.76	1.89
Upper	16	6.19	2.34

ANOVA Said “At Least One Differs” — Which?

ANOVA gives a single yes/no on the whole table — it says some class mean differs, not which ones.

The open questions:

Is Middle significantly different from Lower? From Working? From Upper?
Does Lower differ from Working?
Do Working and Upper differ?

To pinpoint which pairs differ, we need pairwise comparisons.

How Many Pairwise Comparisons?

The natural follow-up to a significant ANOVA: run a separate t-test for each pair of groups.

For \(k\) groups, the number of pairs is

\[a = \frac{k(k-1)}{2}.\]

Groups (\(k\))	Pairwise tests (\(a\))
3	3
4	6
5	10
6	15

For our \(k = 4\) classes, \(a = \dfrac{4 \times 3}{2} = 6\) tests. The more groups, the more tests — and the more chances to make a false rejection.

The Multiple Testing Problem

Each test at \(\alpha = 0.05\) has its own 5% chance of a Type I error when its null is true. Run several tests and those chances accumulate.

Familywise error rate (FWE): the probability of at least one false rejection across the whole family of tests, when every null is true.

With \(a\) tests, the FWE grows quickly:

\[\text{FWE} \approx 1 - (1 - \alpha)^a\]

Tests (\(a\))	Familywise error rate
1	5%
6	≈ 26%
10	≈ 40%
20	≈ 64%

Same inflation pitfall from 15a — exactly what ANOVA’s single test avoided. Now we do want the pairwise tests, so we correct to keep the FWE at \(\alpha\).

Now the problem. Each individual t-test at alpha equals zero point zero five has its own five-percent chance of a false rejection when its null is true. Run several tests and those chances accumulate. The familywise error rate is the probability of at least one false rejection across the whole family of tests, assuming every null is really true. With a tests, that probability is approximately one minus the quantity one minus alpha, raised to the power a. That rate grows quickly: at ten tests it is around forty percent; at twenty, about sixty-four. So running our six pairwise tests without any correction would raise the familywise error rate to twenty-six percent — well above the five percent we intended. That is exactly the inflation ANOVA avoided with a single overall test — but now that we want the pairwise comparisons, we need a correction to bring the familywise error rate back down to alpha.

The Bonferroni Correction

The fix: divide the significance level by the number of tests.

For \(a\) pairwise comparisons at familywise level \(\alpha\), each individual test uses the adjusted significance level

\[\alpha^* = \frac{\alpha}{a}.\]

Each unadjusted p-value must now clear this stricter level — not the original \(\alpha\) — to count as significant.

Why it works. Each individual test now has at most an \(\alpha/a\) chance of a false rejection. Across \(a\) tests, the chance that at least one false rejection occurs is at most \(a \times \dfrac{\alpha}{a} = \alpha\).

Conservative on purpose. Bonferroni guarantees the familywise error rate is at most \(\alpha\); the actual rate is often lower. The trade-off: lower per-test power, so real differences can be missed.

The Bonferroni correction is the simplest fix. For a pairwise comparisons at familywise level alpha, each individual test uses an adjusted significance level: alpha-star equals alpha divided by a. That is the value each unadjusted p-value must now come in under — the stricter adjusted significance level, not the original alpha. You’ll hear this called the adjusted significance level or the Bonferroni cutoff; both refer to the same value. Why does it work? Each individual test now has at most an alpha-over-a chance of a false rejection when its null is true. Add that up across a tests and the chance that at least one false rejection occurs is at most a times alpha over a — which is exactly alpha. So the familywise error rate is controlled at alpha again. One caution is important here. Bonferroni is conservative on purpose. It guarantees the familywise error rate is at most alpha, and in practice the actual rate is often lower than alpha. The trade-off is lower per-test power, so real differences can get missed. That’s a real cost; we’ll mention some less conservative alternatives near the end.

The Six Pairwise Comparisons

The \(a = 6\) pairs for our four classes:

Working vs Lower - Middle vs Lower - Upper vs Lower
Middle vs Working - Upper vs Working - Upper vs Middle

Conditions: no new check needed. These tests pool the SD across all four groups — the same pooling we used for two-group t-tests in topic 12 — so they rest on the ANOVA conditions already verified in 15b.

Try it: with 6 comparisons at familywise \(\alpha = 0.05\), what’s the adjusted significance level \(\alpha^*\)? \(\alpha^* = \dfrac{0.05}{6} \approx 0.0083\) — each unadjusted p-value must clear this stricter level to count as significant.

With four classes there are six pairwise comparisons — Working versus Lower, Middle versus Lower, and so on through the list on the slide. Each one is a t-test between two group means — the same framework as topic twelve, just applied six times. A quick word on conditions: there is no new check to do here. These tests pool the standard deviation across all four groups, rather than using only the two groups in each pair — the same pooling you saw with two-group t-tests in topic twelve. Because of that, they rest on the conditions we already verified for the ANOVA back in fifteen B. Before we run them, work out the adjusted significance level: with six comparisons at a familywise alpha of zero point zero five, what does each individual test have to clear? Advance when you have it. The adjusted significance level is zero point zero five divided by six, about zero point zero zero eight three. Each unadjusted p-value has to come in under that stricter level to count as significant.

Applying the Adjusted Significance Level

Comparison	Unadjusted p	p < α* = 0.0083?
Working − Lower	0.031
Middle − Lower	1.1e-07	✓
Upper − Lower	0.048
Middle − Working	1.6e-12	✓
Upper − Working	0.367
Upper − Middle	0.24

Each unadjusted p-value is compared to the adjusted level \(\alpha^* = 0.05 / 6 \approx 0.0083\).

Two pairs survive correction: Middle vs Lower and Middle vs Working. Two more (Working vs Lower, Upper vs Lower) would have been significant at an uncorrected \(0.05\) — the correction is what rules them out.

Here are all six t-tests, each one’s unadjusted p-value compared to our adjusted significance level of zero point zero zero eight three. Two of them come in under it. Middle versus Lower and Middle versus Working have unadjusted p-values far below zero point zero zero eight three — small enough that the stricter level does not change their verdict. So the Middle class is significantly different from both Lower and Working. The next two are the important cases. Working versus Lower has an unadjusted p-value of zero point zero three one, and Upper versus Lower zero point zero four eight. Both are below zero point zero five, so at an uncorrected five percent level both would have been significant. Against the adjusted significance level of zero point zero zero eight three, neither one is. These two are exactly where the correction changes the answer. The last two are not significant under either standard. Upper versus Working is zero point three seven and Upper versus Middle is zero point two four — both well above zero point zero five, let alone zero point zero zero eight three. And notice that every comparison involving Upper fails: with only sixteen Upper class respondents, those tests don’t have enough power to hold up after correction. So two pairs survive the correction, two more were significant only because we had not yet corrected, and the last two were not significant under either standard.

Adjusting the p-Values Instead

Same correction, rearranged. Rather than pulling the significance level down to \(\alpha^*\), push each p-value up:

\[p_{\text{adj}} = a \times p \quad \text{(capped at 1)},\]

then compare to the original \(\alpha\).

These are the same comparison, just written two ways:

\[p < \frac{\alpha}{a} \quad\Longleftrightarrow\quad a \times p < \alpha.\]

This is the form software reports. Jamovi prints adjusted p-values, so its output is compared directly to \(\alpha = 0.05\).

There is a second way to apply the same correction. Instead of lowering the significance level to alpha-star, we increase each p-value. Multiply each unadjusted p-value by a — the number of comparisons — and cap the result at one, since a probability cannot exceed one. Then compare that adjusted p-value to the original alpha of zero point zero five. These are not two different corrections. They are the same comparison written two ways: asking whether p is less than alpha over a is algebraically equivalent to asking whether a times p is less than alpha. Multiply both sides of the first inequality by a and you get the second. The second form matters because it is what statistical software reports. Jamovi reports adjusted p-values directly, so when you read Jamovi output you compare it to zero point zero five.

The Same Six Tests, Adjusted p-Values

Comparison	Unadjusted p	Adjusted p	Significant at α = 0.05?
Working − Lower	0.031	0.183
Middle − Lower	1.1e-07	6.8e-07	✓
Upper − Lower	0.048	0.285
Middle − Working	1.6e-12	9.8e-12	✓
Upper − Working	0.367	1
Upper − Middle	0.24	1

Adjusted p = unadjusted \(\times\ 6\) (capped at 1), compared to the usual \(\alpha = 0.05\).

Same two pairs, same conclusion. The bookkeeping changed; the answer didn’t.

Now the same six comparisons, run the second way. The unadjusted column is unchanged — those come straight from the six t-tests. The new column multiplies each one by six, caps the result at one, and we compare that to the ordinary zero point zero five. The same two are significant. Middle versus Lower goes from about one point one times ten to the minus seven to about six point eight times ten to the minus seven, and Middle versus Working from about one point six times ten to the minus twelve to about nine point eight times ten to the minus twelve. Multiplying by six changes them very little, and both remain far below zero point zero five. The same two are the important cases. Working versus Lower goes from zero point zero three one to zero point one eight three, and Upper versus Lower from zero point zero four eight to zero point two eight five. Both unadjusted values were below zero point zero five; both adjusted values are above it. Again, these are the two comparisons where the correction changes the answer. The last two show what the cap does. Upper versus Working has an unadjusted p-value of about zero point three six seven. Multiply that by six and we would get about two point two. A probability greater than one is not sensible, so we cap the adjusted p-value at one. Upper versus Middle works the same way: zero point two four times six is about one point four four, again capped at one. Either way, neither is close to significant. And the verdict is identical to the one we just reached: Middle versus Lower and Middle versus Working survive, the other four don’t. This is expected — these are the same correction, so they cannot disagree. Use whichever form you have. If software reports adjusted p-values, compare them to zero point zero five. If you have unadjusted p-values, compare them to alpha over a.

Interpretation

What we can say (at familywise \(\alpha = 0.05\)):

Middle class has a significantly higher mean vocabulary score than Lower class (\(\bar{x}_{\text{Middle}} \approx 6.76\) vs \(5.07\)).
Middle class also scores significantly higher than Working class (\(6.76\) vs \(5.75\)).

What we can’t say:

No significant difference between Lower and Working, or between any pair involving Upper. Upper’s sample mean (\(6.19\)) actually sits well above Lower’s (\(5.07\)) — a larger raw gap than Middle vs Working — yet with only \(n = 16\) Upper respondents the test has too little power to call it significant. Absence of evidence isn’t evidence of absence.

Let’s put the results into context. At the familywise zero point zero five level, Middle class scores significantly higher than Lower class on this vocabulary test — six point seven six versus five point oh seven — and significantly higher than Working class — six point seven six versus five point seven five. Those are the two surviving comparisons. On the other side, we found no significant difference between Lower and Working, or between any pair involving Upper. One detail deserves attention: Upper’s sample mean, six point one nine, is well above Lower’s five point oh seven — a bigger raw gap than the Middle-versus-Working difference that did survive. But Upper has only sixteen respondents, so that gap is too noisy to call significant. Absence of evidence isn’t evidence of absence — Upper’s null result means we can’t tell, not that Upper’s mean equals everyone else’s.

Five Groups

A researcher compares 5 groups with ANOVA, rejects \(H_0\), and wants all pairwise comparisons.

How many pairwise tests are there? \(a = \dfrac{5 \times 4}{2} = 10\)
Without correction, what is the familywise error rate at \(\alpha = 0.05\) (if all 10 null hypotheses were true)? \(1 - 0.95^{10} \approx 0.40\) — about a 40% chance of at least one false rejection.
What is the adjusted significance level \(\alpha^*\)? \(\alpha^* = 0.05 / 10 = 0.005\)
Two pairs give unadjusted p-values of 0.004 and 0.012. Which are significant? \(0.004 < 0.005\) ✓ \(0.012 > 0.005\) ✗

Now it’s your turn to apply the same procedure to a slightly bigger problem. A researcher compares five groups with ANOVA, rejects the null, and wants all pairwise comparisons. Four questions: how many pairwise tests, the uncorrected familywise error rate, the adjusted significance level, and then two p-values to judge against that level. Advance when you are ready to see the answers. For five groups, the number of pairwise comparisons is k times k minus one over two — five times four divided by two, which equals ten. So ten pairwise tests. Without correction at alpha equal to zero point zero five, the familywise error rate is one minus point nine five raised to the tenth power, which is about zero point four zero. That is roughly a forty percent chance of at least one false rejection, just by chance, if all ten null hypotheses were true. The adjusted significance level is alpha over a — zero point zero five divided by ten equals zero point zero zero five. Zero point zero zero four is below zero point zero zero five, so that pair is significant. Zero point zero one two is above zero point zero zero five, so it is not.

The Same Two p-Values

The same two results — unadjusted p-values of 0.004 and 0.012 — with \(a = 10\) comparisons at \(\alpha = 0.05\).

Now decide using adjusted p-values instead. 0.004: \(10 \times 0.004 = 0.04 < 0.05\) ✓ 0.012: \(10 \times 0.012 = 0.12 > 0.05\) ✗

Same verdicts as question 4. The two forms are the same comparison, so they always agree. Note that \(0.012\) would have been significant at an uncorrected \(\alpha = 0.05\).

Beyond Bonferroni

Bonferroni is the simplest correction, but its conservatism costs power — with many tests, real differences can be missed.

Less conservative alternatives give up less power. Software reports them routinely, but they are harder to compute by hand:

Tukey’s HSD — all-pairs comparisons of group means after ANOVA only.
Holm’s procedure — works for any set of tests, like Bonferroni.

For this course we’ll default to Bonferroni, but be aware the others exist — you will see them in software output.

Looking Ahead

After a significant ANOVA, we can now compare the groups two at a time, correcting for how many comparisons we run.

Next time (16b): the same idea for proportions — following up a significant chi-square test by comparing groups two at a time, again with a Bonferroni correction.

References

This topic is not covered by the Introduction to Modern Statistics (2e) textbook.
General Social Survey — vocabulary score (wordsum) by self-identified social class.