Math 115: Introduction to Statistics – 16. Post-Hoc Tests and Multiple Comparisons

From Means to Proportions

In 16a we followed up a significant ANOVA with pairwise t-tests + Bonferroni to ask which group means differ.

The exact same logic applies after a significant chi-square test:

Overall test (topic 14) tells us an association exists.
Pairwise tests tell us which groups differ.
Bonferroni keeps the familywise error rate at \(\alpha\).

Same plan, different statistic — pairwise two-proportion z-tests instead of t-tests.

Recall: Military Spending by Party

GSS 2016 — a random sample of 149 U.S. adults, cross-classified by party and opinion on military spending. The 14a/14b table:

Party	Too Little	About Right	Too Much	Total
Democrat	17	14	12	43
Independent	20	28	24	72
Republican	24	8	2	34
Total	61	50	38	149

Topic 14 result: \(X^2 = 18.97\), \(df = 4\), p-value \(< 0.001\) → reject \(H_0\). Party and opinion are associated.

That verdict covers the whole table at once. It doesn’t say which parties differ.

Before we run anything new, let’s put the topic fourteen table back on the screen. The General Social Survey sampled one hundred forty-nine U.S. adults in twenty sixteen and asked each one two things: their party identification, and whether the country spends too little, about right, or too much on the military. Cross-classifying those two answers gives this three by three table. In fourteen B we ran the chi-square test of independence on it, and got a statistic of about eighteen point nine seven on four degrees of freedom, with a p-value well under zero point zero zero one. We rejected the null. Party and military-spending opinion are associated. But notice what that test does and doesn’t give us. Rejecting independence is a claim about the whole table at once. It says the two variables are related; it doesn’t say which specific parties differ from which. That’s the question this lecture answers.

Simplifying the Response

Our pairwise tool is the two-proportion z-test (topic 10) — and it needs a binary response. Opinion has three levels.

The fix: pick one focal level and collapse the other levels together. We’ll use “Too Little.”

Party	\(n_i\)	“Too Little”	\(\hat{p}_i\)
Democrat	43	17	0.395
Independent	72	20	0.278
Republican	34	24	0.706

Narrower question, so it gets its own overall test: \(X^2 = 17.55\), \(df = 2\), p-value \(< 0.001\) → rejected.

Today’s question: which pairs differ? Eyeball the chart first — which looks clear-cut, which looks shakiest? Both Republican pairs look clear-cut (71% vs 40%, 71% vs 28%). Democrat vs Independent is the close one — is 12 points on 43 and 72 people a real difference, or just sampling variation? You can’t answer that from the chart — that’s what the test is for.

Our pairwise tool is the two-proportion z-test from topic ten, and it compares exactly two proportions. Our response has three levels, so we simplify: pick one level to focus on, and collapse the other two together. We’ll use “too little.” The proportions saying “too little” are about forty percent for Democrats, twenty-eight for Independents, and seventy-one for Republicans. This is a narrower question than the one topic fourteen answered, so it gets its own overall test: chi-square about seventeen point five five on two degrees of freedom, with a p-value well under zero point zero zero one. Rejected — so at least one party differs, and we can start looking at pairs. Before we test, take a few seconds with the chart. Which comparison looks clear-cut, and which looks shakiest? Advance when you have a guess. Both of the pairs involving Republicans look clear-cut — seventy-one percent against forty, and seventy-one against twenty-eight. Those are big gaps. The close one is Democrat versus Independent: forty percent against twenty-eight, a twelve-point gap, resting on forty-three and seventy-two people. Is that a real difference between the parties, or just sampling variation? You can’t answer that from the chart alone. That is why we run the test. Hold onto your guesses; in a moment we’ll see what the tests say.

Three Pairwise Comparisons

With \(k = 3\) parties there are \(a = \dfrac{k(k-1)}{2} = \dfrac{3 \times 2}{2} = 3\) pairs: Democrat vs Independent, Democrat vs Republican, Independent vs Republican.

Each pair uses the two-proportion z-test from topic 10, applied three times.

Two equivalent ways to apply the correction (16a):

Adjust the significance level. Compare each unadjusted p-value to \(\alpha^* = 0.05 / 3 \approx 0.0167\).
Adjust the p-values. Multiply each unadjusted p-value by 3 (cap at 1), then compare to \(\alpha = 0.05\).

Same verdict either way. The results ahead use the second form — it’s what software reports.

With three parties the pairwise count is smaller than in sixteen A: k times k minus one over two, so three times two over two equals three. The pairs are Democrat versus Independent, Democrat versus Republican, and Independent versus Republican. Each pair gets its own two-proportion z-test — the same test from topic ten. So we run that test three times, once for each pair of parties. Sixteen A gave us two ways to apply the Bonferroni correction, and both are on the slide. The first is to adjust the significance level. With three tests at a familywise error rate of zero point zero five, the adjusted significance level is zero point zero five divided by three, about zero point zero one six seven, and each unadjusted p-value has to come in under that. The second is to adjust the p-values instead. Multiply each unadjusted p-value by three, cap the result at one, and compare it to the original zero point zero five. These are the same correction written two ways, so they always agree. The results coming up use the second form, because that is what software reports.

Conditions for Each Pair

Independence — GSS random sample of U.S. adults ✓
Success-failure — at least 10 expected successes and failures per group, pooled under \(H_0\)

Unlike 16a. There the pairwise t-tests pooled the SD across all groups, so they inherited the ANOVA’s conditions — no new check. A z-test pools only its own two groups, so every pair needs checking separately.

Each pair is its own 2×2 table, with its own pooled \(\hat{p}\) — so the check runs three times.

Checking All Three Pairs

D vs I	Too Little	Not	n
Democrat	17	26	43
Independent	20	52	72
Total	37	78	115

D vs R	Too Little	Not	n
Democrat	17	26	43
Republican	24	10	34
Total	41	36	77

I vs R	Too Little	Not	n
Independent	20	52	72
Republican	24	10	34
Total	44	62	106

D vs I. \(\hat{p}_{pool} = \frac{37}{115} \approx 0.322\)
Dem: \(43 \times 0.322 \approx 13.8\), \(43 \times 0.678 \approx 29.2\)
Ind: \(72 \times 0.322 \approx 23.2\), \(72 \times 0.678 \approx 48.8\) — all ≥ 10 ✓

Your turn — D vs R. Work it from the middle table.

\(\hat{p}_{pool} = \frac{41}{77} \approx 0.532\)
Dem: \(43 \times 0.532 \approx 22.9\), \(43 \times 0.468 \approx 20.1\)
Rep: \(34 \times 0.532 \approx 18.1\), \(34 \times 0.468 \approx 15.9\) — all ≥ 10 ✓

I vs R. \(\hat{p}_{pool} = \frac{44}{106} \approx 0.415\) → expected 29.9, 42.1, 14.1, 19.9 — all ≥ 10 ✓

Here are all three pairs, each as its own two by two table. Let me work the first one, then you’ll do the second. Democrat versus Independent. Start by pooling the two groups: thirty-seven of these one hundred fifteen people said “too little,” so the pooled p hat is about zero point three two two. That is our best estimate of the rate both groups share, if the null is true and Democrats and Independents really do say “too little” equally often. Now apply that rate to each group in turn. Among the forty-three Democrats, if the null is true, we would expect about thirteen point eight of them to say “too little,” and the remaining twenty-nine point two to say something else. Among the seventy-two Independents, we would expect about twenty-three point two to say “too little,” and about forty-eight point eight not to. Those four numbers are the expected counts, and every one of them is at least ten — so this pair satisfies the condition. Now you try the next one — Democrat versus Republican. Everything you need is in the middle table. Advance when you have it. Forty-one out of seventy-seven gives a pooled p hat of about zero point five three two — a much higher pooled rate than the first pair, because Republicans say “too little” far more often. Democrats give about twenty-two point nine and twenty point one; Republicans give about eighteen point one and fifteen point nine. All four clear ten again. And the third pair, Independent versus Republican, works the same way — I’ll skip the multiplications this time. Forty-four out of one hundred six is a pooled p hat of about zero point four one five, and the four expected counts come out at about twenty-nine point nine, forty-two point one, fourteen point one, and nineteen point nine. So across all three pairs the smallest expected count anywhere is about thirteen point eight, comfortably above ten. Every pair satisfies the condition, and the normal model is valid for all three tests. That is not automatic. Republicans are the smallest group at thirty-four people, and if the proportion were closer to zero or one, those expected counts could drop below ten. Topic ten C covers that case: when success-failure fails, use a randomization test instead.

Pairwise Z-Tests with Bonferroni

Comparison	Sample proportions	Unadjusted p-value	Adjusted p-value	Sig. at α = 0.05?
Democrat vs Independent	0.395 vs 0.278	0.192	0.576
Democrat vs Republican	0.395 vs 0.706	0.00669	0.0201	✓
Independent vs Republican	0.278 vs 0.706	2.98e-05	8.94e-05	✓

Adjusted p-value = unadjusted × 3 (cap at 1).

Here are the three pairwise z-tests, each with its own adjusted p-value. Democrat versus Independent has an unadjusted p-value of about zero point one nine two — not close to significant. Multiplying by three gives about zero point five seven six. Nowhere near zero point zero five, so Democrats and Independents don’t differ significantly in the proportion saying “too little.” Democrat versus Republican has an unadjusted p-value of about zero point zero zero six seven. Multiplied by three, the adjusted p-value is about zero point zero two — still below zero point zero five, so this one is significant. Independent versus Republican has an unadjusted p-value of about three in one hundred thousand — three times ten to the minus five. Even after tripling it, the adjusted value is about nine times ten to the minus five. Overwhelmingly significant.

Reading the Results

At familywise \(\alpha = 0.05\):

Republicans are significantly more likely than both Democrats and Independents to say military spending is “Too Little” (71% vs 40% and 28%).
Democrats and Independents do not differ significantly from each other (40% vs 28%).

Story: The difference is between Republicans and the other two parties. This matches 14a, where the Republican “Too Little” cell was the single largest contributor to \(X^2\).

Scope: GSS random sample → conclusion generalizes to U.S. adults (2016). Observational data → no causation; we’re describing a real association.

With the familywise error rate held at zero point zero five, Republicans are significantly more likely than both Democrats and Independents to say military spending is “too little” — about seventy-one percent against forty and twenty-eight. Democrats and Independents don’t differ significantly from each other. So the difference is between Republicans and the other two parties. That matches fourteen A, where the Republican “too little” cell contributed most to the chi-square statistic. Fourteen A showed that cell stood out; the pairwise tests now tell us the gap is bigger than sampling variation alone would produce. On scope: the General Social Survey samples U.S. adults at random, so this generalizes to U.S. adults in twenty sixteen. The data are observational, so we describe an association, not a cause.

Confidence in Government

Same 149 GSS respondents, new question: how much confidence do you have in the executive branch of the federal government? Focal level: “Hardly Any.”

Party	\(\hat{p}_i\) (“Hardly Any”)
Democrat	0.326
Independent	0.444
Republican	0.676

Overall test: \(X^2 = 9.6\), \(df = 2\), p-value \(= 0.0082\) → reject. Success-failure holds (smallest expected count 16.3).

Unadjusted pairwise p-values: D vs I: \(0.21\) D vs R: \(0.0022\) I vs R: \(0.026\)

Apply Bonferroni (\(a = 3\)) — what are the adjusted p-values? Adjusted = unadjusted × 3: D vs I: \(0.63\) D vs R: \(0.0066\) I vs R: \(0.078\)

Now it’s your turn, with the same one hundred forty-nine General Social Survey respondents but a different question. This time: how much confidence do you have in the executive branch of the federal government? The focal level is “hardly any.” The proportions saying “hardly any” are about thirty-three percent for Democrats, forty-four percent for Independents, and sixty-eight percent for Republicans — a steady climb across the three groups. The overall chi-square is about nine point six on two degrees of freedom, with a p-value of about zero point zero zero eight, so we reject and we’re entitled to look at pairs. Success-failure has been checked for you — the smallest expected count anywhere is about sixteen, comfortably above ten. The unadjusted pairwise p-values are: Democrat versus Independent about zero point two one, Democrat versus Republican about zero point zero zero two two, and Independent versus Republican about zero point zero two six. Apply Bonferroni and work out the three adjusted p-values. Advance when you are ready. Multiply each unadjusted p-value by three. Democrat versus Independent: zero point two one times three is zero point six three. Democrat versus Republican: zero point zero zero two two times three is zero point zero zero six six. Independent versus Republican: zero point zero two six times three is zero point zero seven eight.

Which Pairs Differ?

Pair	Unadjusted	Adjusted (× 3)
Democrat vs Independent	0.21	0.63
Democrat vs Republican	0.0022	0.0066
Independent vs Republican	0.026	0.078

Using the adjusted p-values, which pairs are significantly different at familywise \(\alpha = 0.05\)?

Only Democrat vs Republican — adjusted \(0.0066\), below 0.05. The other two adjusted p-values, \(0.63\) and \(0.078\), are both above 0.05.

Notice Independent vs Republican. Unadjusted, \(0.026\) is significant; adjusted, \(0.078\) is not. That is Bonferroni’s cost — controlling the familywise error rate means less power on each test.

Here are the unadjusted and adjusted p-values side by side. Using the adjusted column, work out which pairs are significantly different at a familywise error rate of zero point zero five. Advance when you have it. Only Democrat versus Republican. Its adjusted p-value is zero point zero zero six six, below zero point zero five. The other two are zero point six three and zero point zero seven eight, both above zero point zero five, so neither is significant. Now look at Independent versus Republican. Unadjusted, its p-value is zero point zero two six, which would be significant on its own. Adjusted, it is zero point zero seven eight, which is not. That is what the correction costs: holding the familywise error rate at zero point zero five means less power on each individual test. And failing to reject here doesn’t mean the two groups are the same — only that this sample can’t separate them.

Bonferroni for Confidence Intervals

Same idea applies to simultaneous confidence intervals: when you want \(a\) intervals that together have \(1 - \alpha\) confidence, raise the per-interval confidence to \(1 - \alpha/a\).

Example. For today’s 3 pairwise differences in proportions, at overall 95% confidence:

\[\text{Each CI uses confidence level } 1 - \frac{0.05}{3} \approx 0.9833 = 98.33\%.\]

Same arithmetic scales up: 16a’s 4-group ANOVA needed 6 intervals, so each would be built at \(1 - 0.05/6 \approx 99.17\%\).

In practice, statistical software builds these for you when you request “simultaneous” or “Bonferroni-corrected” intervals.

The same trade-off applies: wider intervals individually, but the family of intervals is guaranteed to cover at the stated overall confidence level.

The Bonferroni idea isn’t limited to hypothesis tests — it applies just as naturally to confidence intervals. If you want a family of a intervals to collectively have ninety-five percent confidence, raise each individual interval’s confidence to one minus alpha over a. Take today’s analysis: three pairwise differences in proportions, and we want all three intervals to hold together at ninety-five percent. Each individual interval gets built at one minus zero point zero five over three — about ninety-eight point three three percent confidence. The same arithmetic scales up with the number of comparisons. Sixteen A’s four-group ANOVA had six pairwise differences, so each of those intervals would be built at one minus zero point zero five over six, about ninety-nine point one seven percent. Either way, each individual interval comes out wider than a standalone ninety-five-percent interval would be, but the family as a whole is guaranteed to cover at the ninety-five percent overall confidence level. Statistical software can produce these — usually called “simultaneous” or “Bonferroni-corrected” intervals. Same trade-off as the hypothesis-test version: less precision on each interval, in exchange for guaranteed coverage across the whole family.

Workflow: Multiple Comparisons

The full pattern across both 16a and 16b:

Overall test (ANOVA or chi-square) — does any group differ?
If not significant, stop. No basis for pairwise claims.
If significant, run pairwise comparisons — checking each test’s own conditions:
- Means → pairwise t-tests
- Proportions → pairwise two-proportion z-tests (success-failure, pair by pair)
Apply Bonferroni — multiply each unadjusted p-value by the number of tests \(a\) (cap at 1); compare to \(\alpha\).
Interpret only the comparisons that survive correction.

Looking Ahead

We’ve closed out the inference-with-groups arc: ANOVA / chi-square overall tests, plus pairwise follow-ups with Bonferroni for the natural “which groups differ?” question.

Topic 17: a new kind of question entirely — when both variables are numerical, can we describe and predict one from the other with a regression line?

References

This topic is not covered by the Introduction to Modern Statistics (2e) textbook.
General Social Survey, 2016 — National Opinion Research Center.