16. Multiple Comparisons
Math 115
Example 1: Study Techniques and Exam Performance
A psychology researcher investigated whether different study techniques affect exam performance. A random sample of 120 college students was randomly assigned to one of four study methods: Flashcards, Practice Tests, Re-reading, or Teaching Others. After two weeks of studying the same material using their assigned method, all students took the same exam (scored 0-100).
An ANOVA was conducted to test whether mean exam scores differ across the four study methods. The results showed a significant difference (\(F_{3,116} = 4.87\), p = 0.003).
The researcher now wants to determine which specific pairs of study methods differ. Pairwise t-tests were conducted, and the unadjusted p-values are reported below:
| Comparison | Unadjusted p-value |
|---|---|
| Practice Tests vs Flashcards | 0.142 |
| Re-reading vs Flashcards | 0.621 |
| Teaching Others vs Flashcards | 0.008 |
| Re-reading vs Practice Tests | 0.047 |
| Teaching Others vs Practice Tests | 0.231 |
| Teaching Others vs Re-reading | 0.002 |
Using the Bonferroni method to adjust the p-values, determine which pairs of study methods have significantly different mean exam scores at the \(\alpha = 0.05\) significance level.
Step 1: Determine the number of pairwise comparisons
With \(k = 4\) groups, the number of pairwise comparisons is:
\[a = \frac{k(k-1)}{2} = \frac{4 \times 3}{2} = 6\]
Step 2: Calculate adjusted p-values
The Bonferroni method adjusts p-values by multiplying each unadjusted p-value by the number of comparisons (\(a = 6\)):
\[\text{Adjusted p-value} = \text{Unadjusted p-value} \times a\]
(If the result exceeds 1, we cap it at 1.)
| Comparison | Unadjusted p-value | Adjusted p-value |
|---|---|---|
| Practice Tests vs Flashcards | 0.142 | \(0.142 \times 6 = 0.852\) |
| Re-reading vs Flashcards | 0.621 | \(0.621 \times 6 = 1.000\) |
| Teaching Others vs Flashcards | 0.008 | \(0.008 \times 6 = 0.048\) |
| Re-reading vs Practice Tests | 0.047 | \(0.047 \times 6 = 0.282\) |
| Teaching Others vs Practice Tests | 0.231 | \(0.231 \times 6 = 1.000\) |
| Teaching Others vs Re-reading | 0.002 | \(0.002 \times 6 = 0.012\) |
Step 3: Compare adjusted p-values to \(\alpha = 0.05\)
| Comparison | Adjusted p-value | Significant? |
|---|---|---|
| Practice Tests vs Flashcards | 0.852 | No |
| Re-reading vs Flashcards | 1.000 | No |
| Teaching Others vs Flashcards | 0.048 | Yes |
| Re-reading vs Practice Tests | 0.282 | No |
| Teaching Others vs Practice Tests | 1.000 | No |
| Teaching Others vs Re-reading | 0.012 | Yes |
Step 4: Conclusion
After adjusting for multiple comparisons using the Bonferroni method, there is convincing evidence of a difference in mean exam scores between:
- Teaching Others vs Flashcards (adjusted p = 0.048)
- Teaching Others vs Re-reading (adjusted p = 0.012)
There is not sufficient evidence of a difference between the other pairs of study methods.
Interpretation: The “Teaching Others” method appears to produce different exam performance compared to both “Flashcards” and “Re-reading” methods. (We would need to look at the sample means to determine which method produces higher scores.)
Example 2: Exercise Type and Resting Heart Rate
A health researcher studied whether resting heart rate differs among people who primarily engage in different types of exercise. A random sample of 95 adults was categorized by their primary exercise type: Running, Swimming, or Weightlifting. Resting heart rate (beats per minute) was measured for each participant.
An ANOVA was conducted to test whether mean resting heart rate differs across the three exercise types. The results showed a significant difference (\(F_{2,92} = 5.23\), p = 0.007).
Pairwise t-tests were conducted to compare each pair of exercise types. The unadjusted p-values are:
| Comparison | Unadjusted p-value |
|---|---|
| Swimming vs Running | 0.089 |
| Weightlifting vs Running | 0.002 |
| Weightlifting vs Swimming | 0.031 |
Using the Bonferroni method to adjust the significance level, determine which pairs of exercise types have significantly different mean resting heart rates. Use an overall significance level of \(\alpha = 0.05\).
Step 1: Determine the number of pairwise comparisons
With \(k = 3\) groups, the number of pairwise comparisons is:
\[a = \frac{k(k-1)}{2} = \frac{3 \times 2}{2} = 3\]
Step 2: Calculate the adjusted significance level
The Bonferroni method adjusts the significance level by dividing \(\alpha\) by the number of comparisons:
\[\alpha^* = \frac{\alpha}{a} = \frac{0.05}{3} = 0.0167\]
Step 3: Compare unadjusted p-values to \(\alpha^* = 0.0167\)
| Comparison | Unadjusted p-value | p-value < 0.0167? | Significant? |
|---|---|---|---|
| Swimming vs Running | 0.089 | No | No |
| Weightlifting vs Running | 0.002 | Yes | Yes |
| Weightlifting vs Swimming | 0.031 | No | No |
Step 4: Conclusion
Using the Bonferroni-adjusted significance level of \(\alpha^* = 0.0167\), there is convincing evidence of a difference in mean resting heart rate between:
- Weightlifting vs Running (p = 0.002 < 0.0167)
There is not sufficient evidence of a difference between Swimming and Running or between Weightlifting and Swimming at the adjusted significance level.
Note: The comparison “Weightlifting vs Swimming” has a p-value of 0.031, which would be significant at the unadjusted \(\alpha = 0.05\) level but is NOT significant after the Bonferroni adjustment. This illustrates why the adjustment is necessary — without it, we would have a higher chance of making a Type I error.
Example 3: Preferred News Source by Generation
A media research company surveyed a random sample of 450 American adults about their preferred primary news source. Respondents were classified by generation: Gen Z, Millennials, or Gen X. The response variable was whether they prefer online news (websites, apps, social media) or traditional news (TV, print, radio).
A chi-square test was conducted to determine if there is an association between generation and preference for online news. The results showed a significant association (\(X^2 = 28.4\), df = 2, p < 0.001).
The proportions preferring online news were:
| Generation | Sample size | Proportion preferring online |
|---|---|---|
| Gen Z | 142 | 0.718 |
| Millennials | 168 | 0.583 |
| Gen X | 140 | 0.414 |
To determine which generations differ, pairwise two-proportion Z-tests were conducted. The unadjusted p-values are:
| Comparison | Unadjusted p-value |
|---|---|
| Millennials vs Gen Z | 0.012 |
| Gen X vs Gen Z | < 0.001 |
| Gen X vs Millennials | 0.003 |
Using the Bonferroni method to adjust the p-values, determine which pairs of generations have significantly different proportions preferring online news. Use \(\alpha = 0.05\).
Step 1: Determine the number of pairwise comparisons
With \(k = 3\) groups, the number of pairwise comparisons is:
\[a = \frac{k(k-1)}{2} = \frac{3 \times 2}{2} = 3\]
Step 2: Calculate adjusted p-values
The Bonferroni method adjusts p-values by multiplying each unadjusted p-value by the number of comparisons (\(a = 3\)):
For the comparison with p < 0.001, we’ll use p = 0.0001 as a conservative estimate for the calculation.
| Comparison | Unadjusted p-value | Adjusted p-value |
|---|---|---|
| Millennials vs Gen Z | 0.012 | \(0.012 \times 3 = 0.036\) |
| Gen X vs Gen Z | < 0.001 | \(< 0.001 \times 3 = < 0.003\) |
| Gen X vs Millennials | 0.003 | \(0.003 \times 3 = 0.009\) |
Step 3: Compare adjusted p-values to \(\alpha = 0.05\)
| Comparison | Adjusted p-value | Significant? |
|---|---|---|
| Millennials vs Gen Z | 0.036 | Yes |
| Gen X vs Gen Z | < 0.003 | Yes |
| Gen X vs Millennials | 0.009 | Yes |
Step 4: Conclusion
After adjusting for multiple comparisons using the Bonferroni method, there is convincing evidence that the proportion preferring online news differs between:
- All three pairs of generations
Specifically:
- Gen Z (71.8%) has a significantly higher proportion preferring online news than Millennials (58.3%)
- Gen Z (71.8%) has a significantly higher proportion preferring online news than Gen X (41.4%)
- Millennials (58.3%) have a significantly higher proportion preferring online news than Gen X (41.4%)
Interpretation: There is a clear generational pattern — younger generations are significantly more likely to prefer online news sources compared to older generations, with each generation differing significantly from the others.