16. Post-Hoc Tests and Multiple Comparisons

b. Multiple Comparisons for Proportions
Math 115

Yurk

From Means to Proportions

In 16a we followed up a significant ANOVA with pairwise t-tests + Bonferroni to ask which group means differ.

The exact same logic applies after a significant chi-square test:

Overall test (topic 14) tells us an association exists.
Pairwise tests tell us which groups differ.
Bonferroni keeps the familywise error rate at \alpha.

Same plan, different statistic — pairwise two-proportion z-tests instead of t-tests.

Recall: Military Spending by Party

GSS 2016 — a random sample of 149 U.S. adults, cross-classified by party and opinion on military spending. The 14a/14b table:

Party	Too Little	About Right	Too Much	Total
Democrat	17	14	12	43
Independent	20	28	24	72
Republican	24	8	2	34
Total	61	50	38	149

Topic 14 result: X^2 = 18.97, df = 4, p-value < 0.001 → reject H_0. Party and opinion are associated.

That verdict covers the whole table at once. It doesn’t say which parties differ.

Simplifying the Response

Our pairwise tool is the two-proportion z-test (topic 10) — and it needs a binary response. Opinion has three levels.

The fix: pick one focal level and collapse the other levels together. We’ll use “Too Little.”

Party	n_i	“Too Little”	\hat{p}_i
Democrat	43	17	0.395
Independent	72	20	0.278
Republican	34	24	0.706

Narrower question, so it gets its own overall test: X^2 = 17.55, df = 2, p-value < 0.001 → rejected.

Today’s question: which pairs differ? Eyeball the chart first — which looks clear-cut, which looks shakiest? Both Republican pairs look clear-cut (71% vs 40%, 71% vs 28%). Democrat vs Independent is the close one — is 12 points on 43 and 72 people a real difference, or just sampling variation? You can’t answer that from the chart — that’s what the test is for.

Three Pairwise Comparisons

With k = 3 parties there are a = \dfrac{k(k-1)}{2} = \dfrac{3 \times 2}{2} = 3 pairs: Democrat vs Independent, Democrat vs Republican, Independent vs Republican.

Each pair uses the two-proportion z-test from topic 10, applied three times.

Two equivalent ways to apply the correction (16a):

Adjust the significance level. Compare each unadjusted p-value to \alpha^* = 0.05 / 3 \approx 0.0167.
Adjust the p-values. Multiply each unadjusted p-value by 3 (cap at 1), then compare to \alpha = 0.05.

Same verdict either way. The results ahead use the second form — it’s what software reports.

Conditions for Each Pair

Independence — GSS random sample of U.S. adults ✓
Success-failure — at least 10 expected successes and failures per group, pooled under H_0

Unlike 16a. There the pairwise t-tests pooled the SD across all groups, so they inherited the ANOVA’s conditions — no new check. A z-test pools only its own two groups, so every pair needs checking separately.

Each pair is its own 2×2 table, with its own pooled \hat{p} — so the check runs three times.

Checking All Three Pairs

D vs I	Too Little	Not	n
Democrat	17	26	43
Independent	20	52	72
Total	37	78	115

D vs R	Too Little	Not	n
Democrat	17	26	43
Republican	24	10	34
Total	41	36	77

I vs R	Too Little	Not	n
Independent	20	52	72
Republican	24	10	34
Total	44	62	106

D vs I. \hat{p}_{pool} = \frac{37}{115} \approx 0.322
Dem: 43 \times 0.322 \approx 13.8, 43 \times 0.678 \approx 29.2
Ind: 72 \times 0.322 \approx 23.2, 72 \times 0.678 \approx 48.8 — all ≥ 10 ✓

Your turn — D vs R. Work it from the middle table.

\hat{p}_{pool} = \frac{41}{77} \approx 0.532
Dem: 43 \times 0.532 \approx 22.9, 43 \times 0.468 \approx 20.1
Rep: 34 \times 0.532 \approx 18.1, 34 \times 0.468 \approx 15.9 — all ≥ 10 ✓

I vs R. \hat{p}_{pool} = \frac{44}{106} \approx 0.415 → expected 29.9, 42.1, 14.1, 19.9 — all ≥ 10 ✓

Pairwise Z-Tests with Bonferroni

Comparison	Sample proportions	Unadjusted p-value	Adjusted p-value	Sig. at α = 0.05?
Democrat vs Independent	0.395 vs 0.278	0.192	0.576
Democrat vs Republican	0.395 vs 0.706	0.00669	0.0201	✓
Independent vs Republican	0.278 vs 0.706	2.98e-05	8.94e-05	✓

Adjusted p-value = unadjusted × 3 (cap at 1).

Reading the Results

At familywise \alpha = 0.05:

Republicans are significantly more likely than both Democrats and Independents to say military spending is “Too Little” (71% vs 40% and 28%).
Democrats and Independents do not differ significantly from each other (40% vs 28%).

Story: The difference is between Republicans and the other two parties. This matches 14a, where the Republican “Too Little” cell was the single largest contributor to X^2.

Scope: GSS random sample → conclusion generalizes to U.S. adults (2016). Observational data → no causation; we’re describing a real association.

Confidence in Government

Same 149 GSS respondents, new question: how much confidence do you have in the executive branch of the federal government? Focal level: “Hardly Any.”

Party	\hat{p}_i (“Hardly Any”)
Democrat	0.326
Independent	0.444
Republican	0.676

Overall test: X^2 = 9.6, df = 2, p-value = 0.0082 → reject. Success-failure holds (smallest expected count 16.3).

Unadjusted pairwise p-values: D vs I: 0.21 D vs R: 0.0022 I vs R: 0.026

Apply Bonferroni (a = 3) — what are the adjusted p-values? Adjusted = unadjusted × 3: D vs I: 0.63 D vs R: 0.0066 I vs R: 0.078

Which Pairs Differ?

Pair	Unadjusted	Adjusted (× 3)
Democrat vs Independent	0.21	0.63
Democrat vs Republican	0.0022	0.0066
Independent vs Republican	0.026	0.078

Using the adjusted p-values, which pairs are significantly different at familywise \alpha = 0.05?

Only Democrat vs Republican — adjusted 0.0066, below 0.05. The other two adjusted p-values, 0.63 and 0.078, are both above 0.05.

Notice Independent vs Republican. Unadjusted, 0.026 is significant; adjusted, 0.078 is not. That is Bonferroni’s cost — controlling the familywise error rate means less power on each test.

Bonferroni for Confidence Intervals

Same idea applies to simultaneous confidence intervals: when you want a intervals that together have 1 - \alpha confidence, raise the per-interval confidence to 1 - \alpha/a.

Example. For today’s 3 pairwise differences in proportions, at overall 95% confidence:

\text{Each CI uses confidence level } 1 - \frac{0.05}{3} \approx 0.9833 = 98.33\%.

Same arithmetic scales up: 16a’s 4-group ANOVA needed 6 intervals, so each would be built at 1 - 0.05/6 \approx 99.17\%.

In practice, statistical software builds these for you when you request “simultaneous” or “Bonferroni-corrected” intervals.

The same trade-off applies: wider intervals individually, but the family of intervals is guaranteed to cover at the stated overall confidence level.

Workflow: Multiple Comparisons

The full pattern across both 16a and 16b:

Overall test (ANOVA or chi-square) — does any group differ?
If not significant, stop. No basis for pairwise claims.
If significant, run pairwise comparisons — checking each test’s own conditions:
- Means → pairwise t-tests
- Proportions → pairwise two-proportion z-tests (success-failure, pair by pair)
Apply Bonferroni — multiply each unadjusted p-value by the number of tests a (cap at 1); compare to \alpha.
Interpret only the comparisons that survive correction.

Looking Ahead

We’ve closed out the inference-with-groups arc: ANOVA / chi-square overall tests, plus pairwise follow-ups with Bonferroni for the natural “which groups differ?” question.

Topic 17: a new kind of question entirely — when both variables are numerical, can we describe and predict one from the other with a regression line?

References

This topic is not covered by the Introduction to Modern Statistics (2e) textbook.
General Social Survey, 2016 — National Opinion Research Center.