10. Inference for Two Proportions (Normal Model)

c. Comparing Methods
Math 115

Yurk

Where We Left Off

Across 9a–10b we worked two real datasets four ways each — permutation, bootstrap, normal H-test, normal CI:

  • CPR (blood thinners, survival): all four → fail to reject, intervals include 0
  • Stents (treatment, stroke within 30 days): all four → reject, intervals exclude 0

This deck steps back: how do the methods compare, when do we pick each, and what does the full inference workflow look like end-to-end?

Comparing Methods: Results

Method Purpose CPR (fail to reject) Stents (reject)
Permutation test H-test p-value ≈ 0.27 p-value ≈ 0.004
Normal approximation H-test p-value ≈ 0.171 p-value ≈ 0.0016
Bootstrap CI Estimation (-0.064, 0.325) (0.04, 0.15)
Normal CI Estimation (-0.057, 0.317) (0.035, 0.145)

Across both datasets, the two families don’t match down to the decimal — but they agree on the conclusion. (CPR fail-to-reject, intervals include 0; Stents reject, intervals exclude 0.)

When to Use Each Method

Randomization Normal Model
Conditions Independence Independence + SF (≥ 10 expected/group)
Computation Many simulations Formula-based
Small samples ✓ ✗ — approximation unreliable
Large samples Either works Slightly faster

Rule of thumb: both methods need independence. If success-failure also holds, either method works and they give the same conclusion. If success-failure fails, randomization is the safe choice.

Resume Callbacks — Hypotheses & Data

Does the perceived race signaled by a name affect callback rates? Bertrand & Mullainathan (2004) responded to 1,300 job ads in Boston and Chicago with 4,870 fictitious resumes — first name (white- or black-sounding) randomly assigned; other resume features balanced across the two groups. Outcome: did the employer call back?

Name Callback No callback Total
Black-sounding 157 2278 2435
White-sounding 235 2200 2435
Total 392 4478 4870

  1. State H_0 and H_A for p_W - p_B (two-sided). H_0: p_W - p_B = 0    H_A: p_W - p_B \neq 0

  2. Compute \hat{p}_W, \hat{p}_B, and the observed difference \hat{p}_W - \hat{p}_B. \hat{p}_W = 235/2435 \approx 0.097; \hat{p}_B = 157/2435 \approx 0.064; \hat{p}_W - \hat{p}_B \approx 0.032

Resume Callbacks — H-test Conditions

Name Callback No callback Total
Black-sounding 157 2278 2435
White-sounding 235 2200 2435
Total 392 4478 4870

  1. Compute \hat{p}_{pool}. \hat{p}_{pool} = 392/4870 \approx 0.080

  2. Check success-failure: are all four expected counts ≥ 10? White: 2435 \times 0.080 \approx 195 callbacks; 2435 \times 0.920 \approx 2240 no callbacks.
    Black: 2435 \times 0.080 \approx 195 callbacks; 2435 \times 0.920 \approx 2240 no callbacks. All four ≥ 10 ✓

Resume Callbacks — Z and Conclusion

  1. Compute the pooled SE. SE = \sqrt{0.080 \times 0.920 \times (1/2435 + 1/2435)} \approx 0.0078

  2. Compute Z. Z = 0.032 / 0.0078 \approx 4.10

  3. Two-sided p-value is about 0.00004 (< 0.001). Conclusion at \alpha = 0.05? p \ll 0.05 → reject H_0. Convincing evidence that callback rates differ by name. White-sounding names received about 50% more callbacks (9.7% vs 6.4%) than black-sounding names.

Resume Callbacks — Building the CI

  1. Check success-failure for the CI (observed counts). White: 235 callbacks, 2200 no callbacks (both ≥ 10) ✓   Black: 157 callbacks, 2278 no callbacks (both ≥ 10) ✓. Uses observed counts, not pooled.

  2. Compute the SE (using separate proportions). SE = \sqrt{\dfrac{0.097 \times 0.903}{2435} + \dfrac{0.064 \times 0.936}{2435}} \approx 0.0078

  3. Compute the 95% margin of error and the CI. ME = 1.96 \times 0.0078 \approx 0.015;   0.032 \pm 0.015 = (0.017, 0.047)

Resume Callbacks — CI & Scope

  1. Interpret the 95% CI in context. 95% confident the true p_W - p_B is in (0.017, 0.047) — white-sounding names received between 1.7 and 4.7 percentage points more callbacks than black-sounding names. Excludes 0, consistent with rejecting H_0.

  2. Scope of inference — what claims can we make? Causation: Names randomly assigned → name caused the gap. Evidence of name-based hiring discrimination. Generalization: Not a random sample of ads (Boston + Chicago newspapers, 2001–2002) → applies to similar labor markets only.

Pick the Method

For each scenario, pick randomization or the normal model, and explain why.

  1. Sex discrimination study (1974) (H-test for difference in promotion rates). 48 bank supervisors randomly assigned to evaluate identical resumes labeled male or female. Result: 21/24 promoted males, 14/24 promoted females. H-test check uses pooled \hat{p}: \hat{p}_{pool} \approx 0.73, expected counts \hat{p}_{pool} \cdot 24 = 17.5 ✓ but (1-\hat{p}_{pool}) \cdot 24 = 6.5 ✗ — fails success-failure. Use randomization (permutation).

  2. National household internet survey (95% CI for urban − rural adoption-rate difference). Random sample of 5,000 households: 3,200 urban (88% adopt) and 1,800 rural (76% adopt). CI check uses observed counts: smallest is urban non-adopters \approx 3200 \times 0.12 = 384 ≥ 10 ✓. Normal model is fine and faster; bootstrap would agree.

  3. Pilot trial of a rare-disease therapy (both H-test and CI). 20 patients randomly assigned to two experimental treatments, 2/10 vs 6/10 improvement. Very small sample — H-test expected counts and CI observed counts both well below 10, so the normal model is unreliable. Use randomization for both questions.

Factors Affecting Tests and CIs

H-tests — smaller p-value (stronger evidence):

  • Effect size: larger |\hat{p}_1 - \hat{p}_2| → larger |Z|
  • Sample size: larger n → smaller SE → larger |Z|
  • Variability: \hat{p} near 0.5 → larger SE → weaker test

CIs — narrower interval (more precise):

  • Sample size: larger n → smaller SE → narrower CI
  • Variability: \hat{p} near 0.5 → larger SE → wider CI
  • Confidence level: 90→95→99% → larger z^* → wider CI

Evidence Strength — Sample Size & Effect Size

For each pair, predict which study has stronger evidence (smaller p-value).

  1. Sample size. Two market-research firms test whether new packaging improves purchase rate compared to the old packaging. Both find 60% buy with the new packaging vs 48% with the old. Firm A surveys n = 50 shoppers per group; Firm B surveys n = 200 per group. Firm B. Larger n → smaller SE → larger |Z| → smaller p-value. Rule: more data, stronger evidence.

  2. Effect size. Two universities study whether a tutoring program raises the introductory-stats pass rate. Both have n = 100 students in the tutored and untutored groups. School A: 72% pass (tutored) vs 68% (untutored). School B: 85% vs 65%. School B. Larger observed difference → larger |Z| → smaller p-value. Rule: bigger effect, stronger evidence.

Evidence Strength — Variability

  1. Variability. Two universities test whether an email reminder boosts the response rate on an end-of-semester course survey. Both have n = 150 students in the reminder and no-reminder conditions, and both observe a 10 percentage-point gap. University A serves a population with middling engagement: 52% respond with the reminder vs 42% without. University B serves a highly engaged population: 92% vs 82%. University B. Proportions farther from 0.5 → smaller SE → larger |Z| → smaller p-value. Rule: \hat{p} near 0.5 inflates SE.

CI Width — Sample Size & Confidence Level

For each pair, predict which CI is wider.

  1. Sample size. Back to the packaging study from earlier. Two market-research firms each build a 95% CI for the true new − old purchase-rate difference, and both observe the same sample proportions: 60% buy with new vs 48% with old. Firm A surveys n = 50/group; Firm B surveys n = 200/group. Firm A. Smaller n → larger SE → wider CI. Rule: more data, narrower CI.

  2. Confidence level. Back to School A’s tutoring data — 72% pass rate (tutored) vs 68% (untutored), n = 100/group. School A builds two intervals for the true tutored − untutored difference: a 90% CI and a 99% CI. 99% CI. Higher confidence → larger z^* → wider CI. Trade-off: more confidence, less precision.

CI Width — Variability

  1. Variability. Back to the email-reminder survey-response study — n = 150 in the reminder and no-reminder conditions, both with a 10 percentage-point gap. University A serves a middling-engagement population: 52% respond with the reminder vs 42% without. University B serves a low-engagement population: 12% vs 2%. Both build a 95% CI for the true reminder − no-reminder difference. University A. \hat{p} near 0.5 → larger SE → wider CI. Rule: \hat{p} near 0.5 inflates SE.

Sample size and variability affect both tests and CIs. Effect size drives test strength only; confidence level drives CI width only.

Summary

Topic 10 (normal model):

  • H-test SE uses \hat{p}_{pool} (null assumes equal rates).
  • CI SE uses separate \hat{p}_1, \hat{p}_2.
  • Check success-failure. If it fails — use randomization.

Topic 9 + 10 together:

  • Permutation ↔︎ Normal H-test: “is there a difference?”
  • Bootstrap ↔︎ Normal CI: “how big?”
  • When conditions are met, all four methods agree.
  • Every conclusion needs a scope check (random assignment → causation; random sample → generalization).

References

  • Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
  • Section 17.3
  • Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg More Employable than Lakisha and Jamal? American Economic Review, 94(4), 991–1013.
Course Home