9. Inference for Two Proportions (Randomization)
a. Permutation Test
Math 115
What We’ve Done vs. What We’ll Do
Previously: Inference for a single proportion
- One binary categorical variable
- Hypothesis testing: H_0: p = p_0 (single value)
- Confidence interval: What is p?
Now: Inference for comparing two proportions
- Two binary categorical variables
- Hypothesis testing: H_0: p_1 - p_2 = 0 (no difference between groups)
- Confidence interval: What is p_1 - p_2?
Two Binary Categorical Variables
When we have two binary categorical variables:
- One explanatory variable (defines the groups)
- One response variable (defines success/failure)
We compare the proportion of successes between groups
Examples:
- Stent vs No Stent \rightarrow Stroke rate
- Drug vs Placebo \rightarrow Symptom improvement
- With sign vs Without sign \rightarrow Recycling contamination
CPR Study Introduction
Researchers studied whether blood thinners affect survival during CPR
- 90 patients who received CPR for cardiac arrest
- Patients randomly assigned to treatment or control
- Explanatory: Group (treatment = blood thinner, control = none)
- Response: Outcome (survived or died within 24 hours)
Research question: Does blood thinner affect survival rate?
CPR Study Hypotheses
Let p_T = true survival rate for treatment, p_C = true survival rate for control
Hypotheses:
- H_0: p_T - p_C = 0 (no difference in survival rates)
- H_A: p_T - p_C \neq 0 (survival rates differ between groups)
Sometimes we state the hypotheses in words as
- H_0: There is no association between blood thinner use and survival
- H_A: There is an association between blood thinner use and survival
CPR Study Data
| Control |
39 |
11 |
50 |
| Treatment |
26 |
14 |
40 |
| Total |
65 |
25 |
90 |
Survival proportions:
- Treatment: \hat{p}_T = \frac{14}{40} = 0.35 (35%)
- Control: \hat{p}_C = \frac{11}{50} = 0.22 (22%)
- Observed difference: \hat{p}_T - \hat{p}_C = 0.13
- This difference is the statistic of interest
The New Challenge
For single proportion: We used a spinner to simulate outcomes under the null
- H_0: p = 0.10 \rightarrow set spinner to 10% success
- Generate many samples, build null distribution
For two proportions: We’re testing H_0: p_1 - p_2 = 0
- We don’t know p_1 or p_2, just that they’re equal under null
- Can’t build a spinner without knowing the probabilities
- Need a different approach: permutation test
The Key Insight
Under the null hypothesis (p_T = p_C):
- The groups have the same survival rate
- Whether a patient is in treatment or control doesn’t affect their outcome
- The outcome is independent of group assignment
Key idea: If group doesn’t matter, we can shuffle the outcomes!
Permutation: Shuffling Response Values
A permutation randomly reassigns the response values (outcomes):
- Group assignments stay fixed (40 treatment, 50 control)
- We randomly shuffle which outcome goes with which patient
- This simulates “what if the outcome had nothing to do with group”
After shuffling, we calculate the difference in proportions
This gives us ONE value from the null distribution
Permutation Demo
View interactive permutation demo
The demo shows:
- Original data with treatment/control groups
- Shuffling outcomes between groups (simulating null)
- Calculating difference in proportions after each shuffle
- Building the null distribution one permutation at a time
Building the Null Distribution
- Repeat the permutation many times (1000+)
- Each permutation gives a difference in proportions
- Distribution of all these differences = null distribution
The null distribution shows what differences we’d expect if there’s truly no effect (i.e., if H_0 is true)
CPR Null Distribution
![]()
1000 permutations. Centered at 0 (as expected under null). Shaded region shows the smaller tail used for p-value.
Calculating the P-value
Rule: Split the null at the observed value, then double the smaller-tail proportion.
For the CPR null distribution (1000 permutations), observed difference = 0.13:
- Right tail (\geq 0.13): about 137 of 1000 permutations
- Smaller-tail proportion: 137 / 1000 = 0.137
- Two-sided p-value: 2 \times 0.137 \approx 0.274
CPR Conclusion
Results:
- Observed difference: \hat{p}_T - \hat{p}_C = 0.13
- P-value \approx 0.27
- Using \alpha = 0.05: Do not reject H_0
Conclusion: The data do not provide convincing evidence that blood thinners affect survival rate during CPR.
Scope of inference:
- Causation: random assignment → a significant result would support a causal claim.
- Generalization: patients were not a random sample, so any conclusion extends only to similar cardiac-arrest patients in similar settings.
Active Learning Study
A study randomly assigned 80 students to learn a topic either by active learning (working problems with peers) or traditional lecture, then quizzed them.
| Active learning |
28 |
12 |
40 |
| Traditional lecture |
20 |
20 |
40 |
| Total |
48 |
32 |
80 |
Research question: Does active learning improve pass rates compared to traditional lecture?
Hypotheses & Statistics
For the active learning study (active: 28/40 pass; traditional: 20/40 pass):
State H_0 and H_A for p_A - p_T (pass rate, active − traditional). H_0: p_A - p_T = 0
H_A: p_A - p_T \neq 0
no association vs. association between teaching method and passing
Compute \hat{p}_A and \hat{p}_T. \hat{p}_A = 28/40 = 0.70
\hat{p}_T = 20/40 = 0.50
Compute the observed difference \hat{p}_A - \hat{p}_T. \hat{p}_A - \hat{p}_T = 0.70 - 0.50 = 0.20 — a 20 percentage-point higher pass rate in the active group
Null Distribution — Active vs Traditional
![]()
1000 permutations under H_0: p_A = p_T. Each bar’s height is the number of shuffles whose \hat{p}_A - \hat{p}_T landed there. Shaded right tail: shuffles with a gap \geq 0.20 (our observed).
P-Value & Conclude
![]()
Compute the two-sided p-value. Smaller-tail sum: 43 + 18 + 3 = 64 of 1000
Proportion: 64 / 1000 = 0.064
p-value: 2 \times 0.064 = 0.128
State your conclusion at \alpha = 0.05, including scope of inference. Fail to reject H_0 at \alpha = 0.05. The data do not provide convincing evidence that active learning affects pass rates. If active and traditional truly had the same pass rate, we’d see a difference this extreme or more — in either direction — about 12.8% of the time by chance. Causation: randomly assigned → a significant result would have supported a causal claim. Generalization: not a random sample → conclusion applies to similar students in similar settings only.
Stents and Stroke
A study randomly assigned 451 patients with brain blockages to receive either a stent or standard care alone. Outcome: stroke within 30 days.
| Control |
214 |
13 |
227 |
| Treatment |
191 |
33 |
224 |
| Total |
405 |
46 |
451 |
State H_0 and H_A for p_T - p_C (stroke rate, treatment − control). H_0: p_T - p_C = 0
H_A: p_T - p_C \neq 0
no association vs. association between stent use and stroke
Compute the observed difference \hat{p}_T - \hat{p}_C (stroke rate). \hat{p}_T = 33/224 \approx 0.147
\hat{p}_C = 13/227 \approx 0.057
\hat{p}_T - \hat{p}_C \approx 0.147 - 0.057 = 0.090 — stents are associated with a 9 percentage-point higher stroke rate
Stents — Conclusion
From the stent study: observed diff \approx 0.090, permutation p-value \approx 0.004.
- State the conclusion at \alpha = 0.05, including scope of inference. Reject H_0 at \alpha = 0.05. The data provide convincing evidence that stents affect short-term stroke rate. Causation: randomized → causal. Generalization: not a random sample → applies to similar patient populations only.
Why Permutation Works
The permutation test assumes exchangeability under the null:
- If null is true, outcomes are independent of group
- Shuffling outcomes simulates this independence
- The permuted differences show what’s expected by chance alone
- If observed difference is rare in this distribution \rightarrow evidence against null
Looking Ahead
The permutation test answers: “Is there a difference?”
Next time: a bootstrap CI for p_1 - p_2 — “How big is the difference?”
