16. Post-Hoc Tests and Multiple Comparisons

Math 115

Yurk

The Multiple Testing Problem

When we have 3+ groups and find a significant overall test:

  • ANOVA: At least one mean differs
  • Chi-square: There is an association

Natural follow-up: Which specific groups differ?

This requires pairwise comparisons — but there’s a catch!

Why Not Just Do Many Tests?

With 4 groups, there are \(\frac{4 \times 3}{2} = 6\) pairwise comparisons.

The problem: Each test has a 5% chance of Type I error (if \(\alpha = 0.05\)).

Probability of at least one Type I error (if all null hypotheses are true):

\[1 - 0.95^6 = 0.265 = 26.5\%\]

This inflated error rate is unacceptable!

Familywise Error Rate

Familywise Error Rate (FWE): Probability of making at least one Type I error when performing multiple tests.

For \(a\) independent tests at significance level \(\alpha\):

\[\text{FWE} = 1 - (1-\alpha)^a\]

We need methods that control FWE — keep it at or below \(\alpha\).

The Bonferroni Method

The simplest approach: Divide \(\alpha\) by the number of tests.

For \(a\) pairwise comparisons, use \(\alpha^* = \frac{\alpha}{a}\) for each test.

Why it works:

\[\text{FWE} \leq \frac{\alpha}{a} + \frac{\alpha}{a} + \cdots + \frac{\alpha}{a} = a \cdot \frac{\alpha}{a} = \alpha\]

This guarantees FWE \(\leq \alpha\).

Bonferroni in Practice

Two equivalent ways to apply Bonferroni:

  1. Adjust the significance level: Compare each p-value to \(\alpha/a\)
  2. Adjust the p-values: Multiply each p-value by \(a\), compare to \(\alpha\)

Software typically reports adjusted p-values (Method 2).

Vocabulary Scores by Social Class

Recall: Do vocabulary scores differ by social class?

  • 4 groups: Lower, Middle, Upper, Working
  • ANOVA result: \(F_{3,791} = 21.73\), p-value < 0.001

Conclusion: At least one mean differs.

But which groups differ from each other?

The Data

Class n Mean SD
Lower 41 5.07 2.24
Middle 331 6.76 1.89
Upper 16 6.19 2.34
Working 407 5.75 1.87

Pairwise Comparisons

With 4 groups, there are 6 pairwise comparisons:

  • Lower vs Middle, Lower vs Upper, Lower vs Working
  • Middle vs Upper, Middle vs Working
  • Upper vs Working

Bonferroni adjustment: \(\alpha^* = 0.05/6 = 0.0083\)

(Or multiply each p-value by 6 and compare to 0.05)

Post-Hoc Results: Means

Pairwise T-Tests with Bonferroni adjustment:

Comparison Unadjusted p Adjusted p
MIDDLE vs LOWER 1.1e-07 6.6e-07
UPPER vs LOWER 0.048 0.288
WORKING vs LOWER 0.031 0.186
UPPER vs MIDDLE 0.24 1
WORKING vs MIDDLE 1.6e-12 9.6e-12
WORKING vs UPPER 0.37 1

Note: Adjusted p-value = 6 × Unadjusted p-value (cap at 1)

Significant differences (at \(\alpha = 0.05\)): Middle vs Lower, Middle vs Working

Interpretation: Means

What we can conclude:

  • Middle class has significantly higher mean vocabulary score than Lower class
  • Middle class has significantly higher mean vocabulary score than Working class

What we cannot conclude:

  • No significant difference for any other pair of classes
  • (This doesn’t mean the means are equal — we just lack evidence)

Other Methods for Means

Bonferroni is conservative — it controls FWE but may miss real differences.

Alternative methods (less conservative):

  • Tukey’s procedure: Designed specifically for pairwise comparisons of means
  • Holm’s method: A procedure with more power than Bonferroni

These methods are available in statistical software (e.g., Jamovi).

Following Up a Chi-Square Test

Same idea applies to proportions!

After a significant chi-square test with 3+ groups and a binary response:

  1. The overall test tells us there IS an association
  2. Pairwise comparisons tell us WHICH groups differ
  3. Must adjust for multiple comparisons

We’ll use the Bonferroni method.

Military Spending by Party

From Topic 14: Political party is associated with military spending views.

Follow-up question: Which parties differ in the proportion saying military spending is “Too Much”?

Party n Proportion “Too Much”
Democrat 43 0.279
Independent 72 0.333
Republican 34 0.059

Overall Test

First, test if there’s any difference:

\(H_0\): \(p_{Dem} = p_{Ind} = p_{Rep}\) (same proportion in each party)

\(H_A\): At least one proportion differs

Party Not Too Much Too Much
Democrat 31 12
Independent 48 24
Republican 32 2

\(X^2 = 9.34\), df = 2, p-value = 0.009

Reject \(H_0\): Proportions differ. Now: which pairs?

Pairwise Comparisons for Proportions

With 3 groups, there are \(\frac{3 \times 2}{2} = 3\) pairwise comparisons:

  • Democrat vs Independent
  • Democrat vs Republican
  • Independent vs Republican

Bonferroni adjustment: \(\alpha^* = 0.05/3 = 0.0167\)

Two-Proportion Z-Tests

For each pair, we perform a two-proportion Z-test.

Results with Bonferroni adjustment:

Comparison Unadjusted p-value Adjusted p-value
Democrat vs Independent 0.544 1.000
Democrat vs Republican 0.013 0.039
Independent vs Republican 0.002 0.006

Adjusted p-value = Unadjusted p-value × 3 (capped at 1)

Interpretation: Proportions

Significant differences (at \(\alpha = 0.05\)):

  • Democrat vs Republican: 0.279 vs 0.059
  • Independent vs Republican: 0.333 vs 0.059

Not significant:

  • Democrat vs Independent

Conclusion: Democrats and Independents are both significantly more likely than Republicans to say military spending is “Too Much.” There is no significant difference between Democrats and Independents.

Summary: Multiple Comparisons

The Process:

  1. Perform overall test (ANOVA or chi-square)
  2. If significant, proceed to pairwise comparisons
  3. Apply Bonferroni correction: multiply p-values by number of comparisons

For Means: Pairwise t-tests with Bonferroni (or Tukey/Holm)

For Proportions: Pairwise Z-tests with Bonferroni

Bonferroni for Confidence Intervals

Same idea applies to confidence intervals!

For \(a\) simultaneous confidence intervals at overall confidence \(1 - \alpha\):

Use confidence level \(1 - \frac{\alpha}{a}\) for each interval.

Example: For 6 pairwise CIs at 95% overall confidence:

\[\text{Each CI at } 1 - \frac{0.05}{6} = 0.9917 = 99.17\%\]

Key Takeaways

  1. Multiple testing inflates Type I error rate
  2. Bonferroni method: Divide \(\alpha\) by number of tests
  3. Works for both means and proportions
  4. Conservative but simple and widely applicable
  5. Other methods (Tukey, Holm) may have more power for specific situations

References