10. Inference for Two Proportions (Normal Model)

c. Comparing Methods
Math 115

Yurk

Where We Left Off

Across 9a–10b we worked two real datasets four ways each — permutation, bootstrap, normal H-test, normal CI:

CPR (blood thinners, survival): all four → fail to reject, intervals include 0
Stents (treatment, stroke within 30 days): all four → reject, intervals exclude 0

This deck steps back: how do the methods compare, when do we pick each, and what does the full inference workflow look like end-to-end?

This deck pulls together the work we have done on two proportions over the last four lessons. Across nine A, nine B, ten A, and ten B we applied four inference methods to two real datasets — the CPR study and the stents study. The four methods are the permutation test, the bootstrap confidence interval, the normal-model hypothesis test, and the normal-model confidence interval. For the CPR study, every method failed to reject and every interval included zero, so we had no convincing evidence that blood thinners affect survival. For the stents study, every method rejected and every interval excluded zero, so we had convincing evidence that stents affect short-term stroke rate. Today we step back to compare the methods across both cases, decide when to pick each one, and walk through a full inference workflow from start to finish.

Comparing Methods: Results

Method	Purpose	CPR (fail to reject)	Stents (reject)
Permutation test	H-test	p-value ≈ 0.27	p-value ≈ 0.004
Normal approximation	H-test	p-value ≈ 0.171	p-value ≈ 0.0016
Bootstrap CI	Estimation	(-0.064, 0.325)	(0.04, 0.15)
Normal CI	Estimation	(-0.057, 0.317)	(0.035, 0.145)

Across both datasets, the two families don’t match down to the decimal — but they agree on the conclusion. (CPR fail-to-reject, intervals include 0; Stents reject, intervals exclude 0.)

Here are all four results across both running examples — CPR, where every method failed to reject, and Stents, where every method rejected. Let’s walk through them one row at a time. First, the permutation test. The p-value for CPR was about zero point two seven, and the p-value for Stents was about zero point zero zero four. The normal-model hypothesis test gave a p-value of about zero point one seven for CPR and zero point zero zero one six for Stents. These p-values are different from the permutation p-values, but the conclusions are the same in both cases. The bootstrap confidence interval estimates the true difference in proportions, treatment minus control. For CPR, the bootstrap interval ran from about negative zero point zero six to zero point three three. For Stents, it ran from zero point zero four to zero point one five. The normal-model confidence interval estimates the same parameter — the difference in proportions, treatment minus control — and the intervals are almost identical to the bootstrap: negative zero point zero six to zero point three two for CPR, and zero point zero three five to zero point one four five for Stents. The two families do not match exactly, but in both of these examples they agree on the conclusion: the CPR intervals straddle zero, and the Stents intervals stay positive. In general, when the conditions for the normal model hold, the two families usually agree — though for borderline cases they can land on different sides of the cutoff.

When to Use Each Method

	Randomization	Normal Model
Conditions	Independence	Independence + SF (≥ 10 expected/group)
Computation	Many simulations	Formula-based
Small samples	✓	✗ — approximation unreliable
Large samples	Either works	Slightly faster

Rule of thumb: both methods need independence. If success-failure also holds, either method works and they give the same conclusion. If success-failure fails, randomization is the safe choice.

Resume Callbacks — Hypotheses & Data

Does the perceived race signaled by a name affect callback rates? Bertrand & Mullainathan (2004) responded to 1,300 job ads in Boston and Chicago with 4,870 fictitious resumes — first name (white- or black-sounding) randomly assigned; other resume features balanced across the two groups. Outcome: did the employer call back?

Name	Callback	No callback	Total
Black-sounding	157	2278	2435
White-sounding	235	2200	2435
Total	392	4478	4870

State \(H_0\) and \(H_A\) for \(p_W - p_B\) (two-sided). \(H_0: p_W - p_B = 0\) \(H_A: p_W - p_B \neq 0\)
Compute \(\hat{p}_W\), \(\hat{p}_B\), and the observed difference \(\hat{p}_W - \hat{p}_B\). \(\hat{p}_W = 235/2435 \approx 0.097\); \(\hat{p}_B = 157/2435 \approx 0.064\); \(\hat{p}_W - \hat{p}_B \approx 0.032\)

Your turn — a full inference workflow on a famous economics field experiment. In two thousand four, Bertrand and Mullainathan responded to thirteen hundred job ads in Boston and Chicago with forty-eight hundred seventy fictitious resumes. Resume content varied across the study — different qualifications and work histories — but the first name was randomly assigned to be either white-sounding or black-sounding, so any other resume feature is balanced on average between the two groups. To choose names, the researchers used Massachusetts birth-certificate data to identify first names that were most distinctively associated with white births or with black births, and then validated those associations with a separate survey asking people to guess the race a name signaled. The outcome was whether the employer called back. The design: names were randomly assigned, but the job ads themselves are not a random sample — they’re newspaper ads from two cities in two thousand one and two thousand two. Two quick steps to set up — advance when you are ready to see the answers. This is a two-sided test on the difference in callback rates. The null hypothesis is that p W minus p B equals zero — that white-sounding and black-sounding names produce the same callback rate. The alternative is that p W minus p B is not zero — that the two rates differ, with no committed direction. White-sounding callback rate is two hundred thirty-five over twenty-four hundred thirty-five, about zero point zero nine seven. Black-sounding is one hundred fifty-seven over twenty-four hundred thirty-five, about zero point zero six four. Observed difference is about zero point zero three two — white-sounding names got about three more callbacks per hundred resumes than black-sounding names.

Resume Callbacks — H-test Conditions

Name	Callback	No callback	Total
Black-sounding	157	2278	2435
White-sounding	235	2200	2435
Total	392	4478	4870

Compute \(\hat{p}_{pool}\). \(\hat{p}_{pool} = 392/4870 \approx 0.080\)
Check success-failure: are all four expected counts ≥ 10? White: \(2435 \times 0.080 \approx 195\) callbacks; \(2435 \times 0.920 \approx 2240\) no callbacks.
Black: \(2435 \times 0.080 \approx 195\) callbacks; \(2435 \times 0.920 \approx 2240\) no callbacks. All four ≥ 10 ✓

Same data, repeated so you can read the counts off as you compute. Two quick steps for the hypothesis test conditions — advance when you are ready to see the answers. P pooled is total callbacks over total resumes — three hundred ninety-two over forty-eight hundred seventy — about zero point zero eight. Now check success-failure with expected counts under the null. Both groups have the same n of twenty-four hundred thirty-five, so the expected counts are identical. Twenty-four hundred thirty-five times zero point zero eight is about one hundred ninety-five expected callbacks per group, and twenty-four hundred thirty-five times zero point nine two is about twenty-two hundred forty expected no-callbacks per group. All four are well above ten, so the success-failure condition is satisfied.

Resume Callbacks — Z and Conclusion

Compute the pooled SE. \(SE = \sqrt{0.080 \times 0.920 \times (1/2435 + 1/2435)} \approx 0.0078\)
Compute \(Z\). \(Z = 0.032 / 0.0078 \approx 4.10\)
Two-sided p-value is about \(0.00004\) (\(< 0.001\)). Conclusion at \(\alpha = 0.05\)? \(p \ll 0.05\) → reject \(H_0\). Convincing evidence that callback rates differ by name. White-sounding names received about 50% more callbacks (9.7% vs 6.4%) than black-sounding names.

Conditions are met, so we can proceed with the normal model. Three quick steps to wrap up the hypothesis test — advance when you are ready to see the solutions. Plug into the pooled SE formula — zero point zero eight times zero point nine two, times one over twenty-four hundred thirty-five plus one over twenty-four hundred thirty-five, then take the square root. SE comes out to about zero point zero zero seven eight — small, because the sample sizes are large. Then Z is zero point zero three two over zero point zero zero seven eight, about four point one — a very large Z. A Z of four point one gives a two-sided p-value of about zero point zero zero zero zero four — far below zero point zero five — so we reject the null. The data provide convincing evidence that callback rates differ by perceived race of the name. In practical terms, white-sounding names received about fifty percent more callbacks than black-sounding names.

Resume Callbacks — Building the CI

Check success-failure for the CI (observed counts). White: 235 callbacks, 2200 no callbacks (both ≥ 10) ✓ Black: 157 callbacks, 2278 no callbacks (both ≥ 10) ✓. Uses observed counts, not pooled.
Compute the SE (using separate proportions). \(SE = \sqrt{\dfrac{0.097 \times 0.903}{2435} + \dfrac{0.064 \times 0.936}{2435}} \approx 0.0078\)
Compute the 95% margin of error and the CI. \(ME = 1.96 \times 0.0078 \approx 0.015\); \(0.032 \pm 0.015 = (0.017, 0.047)\)

Now the confidence interval side. Three quick steps — advance when you are ready to see the solutions. The success-failure check for a confidence interval uses observed counts, not pooled expected counts. White-sounding has two hundred thirty-five callbacks and twenty-two hundred no-callbacks; black-sounding has one hundred fifty-seven and twenty-two hundred seventy-eight. All four are well above ten, so success-failure is satisfied for the confidence interval. SE for the confidence interval uses separate proportions. Zero point zero nine seven times zero point nine zero three over twenty-four hundred thirty-five, plus zero point zero six four times zero point nine three six over twenty-four hundred thirty-five, all under a square root — about zero point zero zero seven eight. Almost identical to the pooled SE in this case. Margin of error is one point nine six times zero point zero zero seven eight, about zero point zero one five. The confidence interval is the observed difference of zero point zero three two plus or minus zero point zero one five, giving an interval from zero point zero one seven up to zero point zero four seven.

Resume Callbacks — CI & Scope

Interpret the 95% CI in context. 95% confident the true \(p_W - p_B\) is in \((0.017, 0.047)\) — white-sounding names received between 1.7 and 4.7 percentage points more callbacks than black-sounding names. Excludes 0, consistent with rejecting \(H_0\).
Scope of inference — what claims can we make? Causation: Names randomly assigned → name caused the gap. Evidence of name-based hiring discrimination. Generalization: Not a random sample of ads (Boston + Chicago newspapers, 2001–2002) → applies to similar labor markets only.

Last slide on resume callbacks — wrap up the confidence interval. Two questions — advance when you are ready to see the answers. We’re ninety-five percent confident the true difference in callback rates — white-sounding minus black-sounding — is between zero point zero one seven and zero point zero four seven. In plain terms: white-sounding names received somewhere between one point seven and four point seven percentage points more callbacks than black-sounding names. The interval excludes zero, consistent with rejecting in the hypothesis test. On scope of inference: names were randomly assigned, and since we rejected, we can say that the name on the resume caused the difference in callback rates. That is evidence of name-based hiring discrimination by the employers in this study. On generalization: the job ads were not a random sample — they were newspaper ads in Boston and Chicago between two thousand one and two thousand two — so the conclusion applies to similar labor markets in similar eras, not to all employers everywhere.

Pick the Method

For each scenario, pick randomization or the normal model, and explain why.

Sex discrimination study (1974) (H-test for difference in promotion rates). 48 bank supervisors randomly assigned to evaluate identical resumes labeled male or female. Result: 21/24 promoted males, 14/24 promoted females. H-test check uses pooled \(\hat{p}\): \(\hat{p}_{pool} \approx 0.73\), expected counts \(\hat{p}_{pool} \cdot 24 = 17.5\) ✓ but \((1-\hat{p}_{pool}) \cdot 24 = 6.5\) ✗ — fails success-failure. Use randomization (permutation).
National household internet survey (95% CI for urban − rural adoption-rate difference). Random sample of 5,000 households: 3,200 urban (88% adopt) and 1,800 rural (76% adopt). CI check uses observed counts: smallest is urban non-adopters \(\approx 3200 \times 0.12 = 384\) ≥ 10 ✓. Normal model is fine and faster; bootstrap would agree.
Pilot trial of a rare-disease therapy (both H-test and CI). 20 patients randomly assigned to two experimental treatments, 2/10 vs 6/10 improvement. Very small sample — H-test expected counts and CI observed counts both well below 10, so the normal model is unreliable. Use randomization for both questions.

Here’s another your-turn — picking the right method for three different studies. For each scenario, pause and decide which method you’d use and why before advancing for the answer. The first is the classic sex discrimination study from nineteen seventy-four — a hypothesis test for the difference in promotion rates. Forty-eight bank supervisors were randomly assigned to evaluate identical resumes labeled either male or female. The H-test check uses the pooled p hat, which is about zero point seven three; expected successes are seventeen point five — fine — but expected failures are only six point five, which falls below the ten threshold. The success-failure condition fails, so we use a permutation test instead of the normal model. The second is a national household internet survey — five thousand households, with thirty-two hundred urban and eighteen hundred rural — and the question is a confidence interval for the urban-minus-rural adoption-rate difference. The CI check uses observed counts, not pooled. The smallest is urban non-adopters: thirty-two hundred times zero point one two, about three hundred eighty-four — far above ten. So the normal model is fine here, and the formula is more efficient than bootstrap. The third is a pilot trial for a rare-disease therapy with both questions on the table — is there a difference, and how big is it. Only twenty patients, ten in each group. The H-test expected counts and the CI observed counts both fall well below ten. The normal-model approximation breaks down at this size, so randomization is required for both — a permutation test for the comparison and a bootstrap interval for the estimate.

Factors Affecting Tests and CIs

H-tests — smaller p-value (stronger evidence):

Effect size: larger \(|\hat{p}_1 - \hat{p}_2|\) → larger \(|Z|\)
Sample size: larger \(n\) → smaller SE → larger \(|Z|\)
Variability: \(\hat{p}\) near 0.5 → larger SE → weaker test

CIs — narrower interval (more precise):

Sample size: larger \(n\) → smaller SE → narrower CI
Variability: \(\hat{p}\) near 0.5 → larger SE → wider CI
Confidence level: 90→95→99% → larger \(z^*\) → wider CI

Stepping back, let’s look at what makes evidence strong and what makes a confidence interval precise. Three factors affect the p-value of a hypothesis test. The first is effect size, because a bigger observed difference pushes Z further from zero. The second is sample size, because a larger sample shrinks the standard error and that inflates Z. The third is variability, because proportions near one half produce the largest SE, and that weakens the test. For a confidence interval, sample size and variability play the same role through the SE. A larger sample shrinks the SE and narrows the interval. Proportions near one half inflate the SE and widen the interval. The new factor for confidence intervals is the confidence level itself. Moving from ninety percent to ninety-five to ninety-nine multiplies z-star and widens the interval. Two your-turns are coming up, first on test strength and then on confidence-interval width, to put these rules into practice.

Evidence Strength — Sample Size & Effect Size

For each pair, predict which study has stronger evidence (smaller p-value).

Sample size. Two market-research firms test whether new packaging improves purchase rate compared to the old packaging. Both find 60% buy with the new packaging vs 48% with the old. Firm A surveys \(n = 50\) shoppers per group; Firm B surveys \(n = 200\) per group. Firm B. Larger \(n\) → smaller SE → larger \(|Z|\) → smaller p-value. Rule: more data, stronger evidence.
Effect size. Two universities study whether a tutoring program raises the introductory-stats pass rate. Both have \(n = 100\) students in the tutored and untutored groups. School A: 72% pass (tutored) vs 68% (untutored). School B: 85% vs 65%. School B. Larger observed difference → larger \(|Z|\) → smaller p-value. Rule: bigger effect, stronger evidence.

Now a series of your-turns on what makes evidence strong and what makes a confidence interval narrow. For each pair, predict which study has the smaller p-value. Advance when you are ready to see the answers. First, sample size. Two market-research firms see the same purchase rates — sixty percent with new packaging versus forty-eight percent with old — but Firm A surveys fifty per group and Firm B surveys two hundred per group. Firm B has the smaller p-value. A larger n shrinks the standard error, which makes Z larger and the p-value smaller. Second, effect size. Two universities test whether tutoring raises the intro-stats pass rate, both with one hundred per group. School A sees a four percentage-point gap and School B sees a twenty percentage-point gap. School B has the smaller p-value. A bigger observed difference pushes Z further from zero, which shrinks the p-value.

Evidence Strength — Variability

Variability. Two universities test whether an email reminder boosts the response rate on an end-of-semester course survey. Both have \(n = 150\) students in the reminder and no-reminder conditions, and both observe a 10 percentage-point gap. University A serves a population with middling engagement: 52% respond with the reminder vs 42% without. University B serves a highly engaged population: 92% vs 82%. University B. Proportions farther from 0.5 → smaller SE → larger \(|Z|\) → smaller p-value. Rule: \(\hat{p}\) near 0.5 inflates SE.

CI Width — Sample Size & Confidence Level

For each pair, predict which CI is wider.

Sample size. Back to the packaging study from earlier. Two market-research firms each build a 95% CI for the true new − old purchase-rate difference, and both observe the same sample proportions: 60% buy with new vs 48% with old. Firm A surveys \(n = 50\)/group; Firm B surveys \(n = 200\)/group. Firm A. Smaller \(n\) → larger SE → wider CI. Rule: more data, narrower CI.
Confidence level. Back to School A’s tutoring data — 72% pass rate (tutored) vs 68% (untutored), \(n = 100\)/group. School A builds two intervals for the true tutored − untutored difference: a 90% CI and a 99% CI. 99% CI. Higher confidence → larger \(z^*\) → wider CI. Trade-off: more confidence, less precision.

Same kind of your-turn for confidence intervals, three scenarios across two slides. Advance when you are ready to see the answers. First, sample size. Back to the market-research packaging study from earlier. Two firms each build a ninety-five percent confidence interval for the true new minus old purchase-rate difference, and both happen to observe the same sample proportions: sixty percent buy with new versus forty-eight percent with old. Firm A surveys fifty per group; Firm B surveys two hundred per group. Firm A has the wider interval — a smaller n means a larger SE and a wider interval. Second, confidence level. Back to School A’s tutoring data — seventy-two percent pass rate among tutored students versus sixty-eight percent among untutored, one hundred per group. School A builds two intervals for the true difference: a ninety percent CI and a ninety-nine percent CI. The ninety-nine percent interval is wider. Higher confidence requires a larger z-star multiplier, which inflates the margin of error. That’s the classic trade-off — more confidence, less precision.

CI Width — Variability

Variability. Back to the email-reminder survey-response study — \(n = 150\) in the reminder and no-reminder conditions, both with a 10 percentage-point gap. University A serves a middling-engagement population: 52% respond with the reminder vs 42% without. University B serves a low-engagement population: 12% vs 2%. Both build a 95% CI for the true reminder − no-reminder difference. University A. \(\hat{p}\) near 0.5 → larger SE → wider CI. Rule: \(\hat{p}\) near 0.5 inflates SE.

Sample size and variability affect both tests and CIs. Effect size drives test strength only; confidence level drives CI width only.

Third scenario, variability. Back to the email-reminder survey-response study — one hundred fifty students per group, both with a ten percentage-point gap. University A serves a middling-engagement population, fifty-two percent versus forty-two percent. University B serves a low-engagement population, twelve percent versus two percent. Both build a ninety-five percent confidence interval for the true reminder minus no-reminder difference. Predict which is wider, then advance for the answer. University A has the wider interval. Proportions near zero or one have smaller standard errors than proportions near one half. Wrapping up the parallel: sample size and variability affect both hypothesis tests and confidence intervals; effect size affects only the test, and confidence level affects only the interval.

Summary

Topic 10 (normal model):

H-test SE uses \(\hat{p}_{pool}\) (null assumes equal rates).
CI SE uses separate \(\hat{p}_1, \hat{p}_2\).
Check success-failure. If it fails — use randomization.

Topic 9 + 10 together:

Permutation ↔︎ Normal H-test: “is there a difference?”
Bootstrap ↔︎ Normal CI: “how big?”
When conditions are met, all four methods agree.
Every conclusion needs a scope check (random assignment → causation; random sample → generalization).

Here’s topic ten in one slide. The hypothesis test SE uses the pooled proportion because the null assumes the two rates are equal. The confidence interval SE uses separate sample proportions because a confidence interval makes no such assumption. Always check success-failure before reaching for the normal model — if it fails, randomization is the safe fallback. Stepping back to topic nine plus topic ten together: permutation and the normal hypothesis test both answer “is there a difference?” Bootstrap and the normal confidence interval both answer “how big is the difference?” When the normal model’s conditions are met, all four methods agree. And no inference example is complete without a scope check. Random assignment is needed to claim that the treatment caused the difference, and random sampling is needed to generalize the conclusion to a broader population.

References

Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
Section 17.3
Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg More Employable than Lakisha and Jamal? American Economic Review, 94(4), 991–1013.