9. Inference for Two Proportions (Randomization)
a. Permutation Test
What We’ve Done vs. What We’ll Do
Previously : Inference for a single proportion
One binary categorical variable
Hypothesis testing: \(H_0: p = p_0\) (single value)
Confidence interval: What is \(p\) ?
Now : Inference for comparing two proportions
Two binary categorical variables
Hypothesis testing: \(H_0: p_1 - p_2 = 0\) (no difference between groups)
Confidence interval: What is \(p_1 - p_2\) ?
So far we’ve made inferences about a single proportion — testing it against a specific value, or estimating it with a confidence interval. After our recent work, we can describe the difference in proportions between two groups, but we can’t yet make inferences about that difference. That changes now. The hypothesis shifts from “p equals some number” to “the difference between the two proportions is zero,” and a confidence interval gives a range for that difference rather than for a single proportion. Today we handle the test; the interval comes next.
Two Binary Categorical Variables
When we have two binary categorical variables:
One explanatory variable (defines the groups)
One response variable (defines success/failure)
We compare the proportion of successes between groups
Examples:
Stent vs No Stent \(\rightarrow\) Stroke rate
Drug vs Placebo \(\rightarrow\) Symptom improvement
With sign vs Without sign \(\rightarrow\) Recycling contamination
The data structure here is familiar — one categorical variable defines the groups, another records success or failure. The same pattern shows up across many studies: stent versus standard care with stroke as the response, drug versus placebo with symptom improvement, or signs versus no signs measuring recycling contamination. Until now we only described how different the two proportions were; here we add the inference question — is the difference more than chance would produce?
CPR Study Introduction
Researchers studied whether blood thinners affect survival during CPR
90 patients who received CPR for cardiac arrest
Patients randomly assigned to treatment or control
Explanatory: Group (treatment = blood thinner, control = none)Response: Outcome (survived or died within 24 hours)
Research question: Does blood thinner affect survival rate?
Our running example studies ninety patients undergoing CPR for cardiac arrest, randomly assigned to a blood thinner or control care. The question was whether giving blood thinners during the procedure affects survival. Forty patients got the thinner, fifty got control care, and the response was whether they were alive twenty-four hours later. This is exactly the two-proportion setup we just framed.
CPR Study Hypotheses
Let \(p_T\) = true survival rate for treatment, \(p_C\) = true survival rate for control
Hypotheses:
\(H_0: p_T - p_C = 0\) (no difference in survival rates)\(H_A: p_T - p_C \neq 0\) (survival rates differ between groups)
Sometimes we state the hypotheses in words as
\(H_0:\) There is no association between blood thinner use and survival\(H_A:\) There is an association between blood thinner use and survival
Now we restate the research question as formal hypotheses. The null says the true survival rates in the treatment and control groups are equal — no effect, difference of zero. The alternative says those survival rates differ — survival depends on which group a patient was in. Equivalently in words: the null says blood thinner use and survival are unassociated; the alternative says they are associated.
CPR Study Data
Control
39
11
50
Treatment
26
14
40
Total
65
25
90
Survival proportions:
Treatment: \(\hat{p}_T = \frac{14}{40} = 0.35\) (35%)
Control: \(\hat{p}_C = \frac{11}{50} = 0.22\) (22%)
Observed difference: \(\hat{p}_T - \hat{p}_C = 0.13\) This difference is the statistic of interest
The table summarizes the CPR outcomes — how many survived and how many died in each group. Let’s pull out the survival proportions and the difference. Of the forty patients on the blood thinner, fourteen survived — a survival rate of zero point three five. Of the fifty control patients, only eleven survived — a rate of zero point two two. The observed difference — treatment minus control — is about zero point one three, in favor of treatment. That’s the number our test will focus on. Is a difference that large unusual under the null?
The New Challenge
For single proportion : We used a spinner to simulate outcomes under the null
\(H_0: p = 0.10\) \(\rightarrow\) set spinner to 10% successGenerate many samples, build null distribution
For two proportions : We’re testing \(H_0: p_1 - p_2 = 0\)
We don’t know \(p_1\) or \(p_2\) , just that they’re equal under null
Can’t build a spinner without knowing the probabilities
Need a different approach: permutation test
Here’s the wrinkle. For a single proportion we used a spinner — set it to whatever value the null specified, ten percent or fifty percent or something else, and then simulated. But the CPR null only says the treatment and control survival rates are equal — it doesn’t say what value they’re equal to. Without a specific probability we can’t build the spinner. So we need a different approach — the permutation test.
The Key Insight
Under the null hypothesis (\(p_T = p_C\) ):
The groups have the same survival rate
Whether a patient is in treatment or control doesn’t affect their outcome
The outcome is independent of group assignment
Key idea: If group doesn’t matter, we can shuffle the outcomes!
Here’s the trick that makes the permutation test work. If the null is true — if the two groups really have the same rate — then the group label doesn’t determine the outcome. Whether a patient was randomized to treatment or control is irrelevant; their outcome would have been the same either way. And if labels are irrelevant, we can swap them around — shuffle who’s in which group — and the data should look just as plausible as it did originally.
Permutation: Shuffling Response Values
A permutation randomly reassigns the response values (outcomes):
Group assignments stay fixed (40 treatment, 50 control)
We randomly shuffle which outcome goes with which patient
This simulates “what if the outcome had nothing to do with group”
After shuffling, we calculate the difference in proportions
This gives us ONE value from the null distribution
Concretely, we keep the group sizes fixed at forty treatment and fifty control, but we randomly reassign which outcome goes with which patient. Imagine taking all ninety outcome labels — survived or died — putting them in a hat, shuffling, and dealing them back out without regard to who was originally in which group. By shuffling, we’re simulating what the data would look like if outcome had nothing to do with group — exactly the scenario the null hypothesis describes. Then we recompute the difference in proportions, giving us one possible value of the statistic under the null.
Permutation Demo
View interactive permutation demo
The demo shows:
Original data with treatment/control groups
Shuffling outcomes between groups (simulating null)
Calculating difference in proportions after each shuffle
Building the null distribution one permutation at a time
There’s an interactive demo linked here that walks through this visually. It starts with the actual data, runs a shuffle, recomputes the difference, and stacks the results into a null distribution one permutation at a time. Worth a couple of minutes if the verbal description hasn’t quite clicked.
Building the Null Distribution
Repeat the permutation many times (1000+)
Each permutation gives a difference in proportions
Distribution of all these differences = null distribution
The null distribution shows what differences we’d expect if there’s truly no effect (i.e., if \(H_0\) is true)
One shuffle isn’t enough — it could land anywhere by chance. So we repeat the shuffle one thousand times. For the CPR data, that’s one thousand random reassignments of survived-or-died across the ninety patients, each producing a fresh treatment-minus-control difference. The stack of those one thousand differences is the null distribution — the full range of differences we’d expect by chance alone if blood thinner and survival had nothing to do with each other.
CPR Null Distribution
1000 permutations. Centered at 0 (as expected under null). Shaded region shows the smaller tail used for p-value.
This histogram is what a world without a blood thinner effect looks like. Each value is the treatment-minus-control survival difference from one random shuffle of survived-or-died labels across the ninety patients, and we’ve stacked a thousand of them. The bulk sits near zero — expected, because under the null the two groups truly share the same survival rate. The shaded right tail represents shuffles that produced a difference at least as large as our observed zero point one three. That shaded area becomes our p-value.
Calculating the P-value
Rule: Split the null at the observed value, then double the smaller-tail proportion.
For the CPR null distribution (1000 permutations), observed difference \(= 0.13\) :
Right tail (\(\geq 0.13\) ): about 137 of 1000 permutationsSmaller-tail proportion : \(137 / 1000 = 0.137\) Two-sided p-value : \(2 \times 0.137 \approx 0.274\)
Now let’s actually compute the two-sided p-value for the blood thinner question. The rule splits the null at our observed difference of zero point one three and doubles the smaller side. The smaller side is the right tail — counting permutations from the histogram, it comes to about one hundred thirty-seven out of a thousand. So the smaller-tail proportion is about zero point one three seven, and doubling gives a two-sided p-value of about zero point two seven four. In plain terms: if blood thinner truly had no effect on survival, we’d still see a treatment-control gap this extreme or more — in either direction — about twenty-seven percent of the time just by chance, which isn’t unusual enough to call the gap real.
CPR Conclusion
Results:
Observed difference: \(\hat{p}_T - \hat{p}_C = 0.13\)
P-value \(\approx 0.27\)
Using \(\alpha = 0.05\) : Do not reject \(H_0\)
Conclusion: The data do not provide convincing evidence that blood thinners affect survival rate during CPR.
Scope of inference:
Causation: random assignment → a significant result would support a causal claim.Generalization: patients were not a random sample, so any conclusion extends only to similar cardiac-arrest patients in similar settings.
Putting it together: the observed difference is about zero point one three and the p-value is zero point two seven, well above our cutoff of zero point zero five. We fail to reject the null. The data do not provide convincing evidence that blood thinners affect survival rate during CPR. There’s still the scope-of-inference question. Patients were randomly assigned to treatment or control, so if we had found a significant difference, we could have concluded that blood thinners caused it. But these patients were not a random sample of all cardiac-arrest cases — they were drawn from a specific clinical setting — so any conclusion applies only to similar cardiac-arrest patients in similar settings, not the broader population.
Active Learning Study
A study randomly assigned 80 students to learn a topic either by active learning (working problems with peers) or traditional lecture , then quizzed them.
Active learning
28
12
40
Traditional lecture
20
20
40
Total
48
32
80
Research question: Does active learning improve pass rates compared to traditional lecture?
Now it’s your turn — we’re switching to a study on teaching methods. Eighty students were randomly assigned to learn a topic either by active learning, where they worked problems with peers, or by traditional lecture, then quizzed on the material. The table shows the results: twenty-eight of forty in the active group passed, and twenty of forty in the traditional group passed. Over the next few slides you’ll set up the hypotheses, compute the sample statistics, count up the p-value from the null distribution, and reach a conclusion.
Hypotheses & Statistics
For the active learning study (active: 28/40 pass; traditional: 20/40 pass):
State \(H_0\) and \(H_A\) for \(p_A - p_T\) (pass rate, active − traditional). \(H_0: p_A - p_T = 0\) \(H_A: p_A - p_T \neq 0\) no association vs. association between teaching method and passing
Compute \(\hat{p}_A\) and \(\hat{p}_T\) . \(\hat{p}_A = 28/40 = 0.70\) \(\hat{p}_T = 20/40 = 0.50\)
Compute the observed difference \(\hat{p}_A - \hat{p}_T\) . \(\hat{p}_A - \hat{p}_T = 0.70 - 0.50 = 0.20\) — a 20 percentage-point higher pass rate in the active group
Three quick questions to set up the inference for this study. Advance when you are ready to see the answers. For the hypotheses, the null says the difference between active and traditional pass rates is zero — no association between teaching method and whether a student passed. The alternative says the difference is not zero — there is an association in some direction. For the sample proportions: active learning had twenty-eight of forty students pass, so p-hat-A is zero point seven. Traditional had twenty of forty pass, so p-hat-T is zero point five. The observed difference, active minus traditional, is zero point seven minus zero point five, which equals zero point two. Active learning shows a twenty percentage-point higher pass rate than traditional lecture.
Null Distribution — Active vs Traditional
1000 permutations under \(H_0: p_A = p_T\) . Each bar’s height is the number of shuffles whose \(\hat{p}_A - \hat{p}_T\) landed there. Shaded right tail: shuffles with a gap \(\geq 0.20\) (our observed).
Here’s the null distribution for the active learning study. We built it by shuffling the eighty pass-or-fail outcomes randomly across the two groups a thousand times, each time computing the difference in pass rates. The bars show how many shuffles produced a difference in that range — most land near zero. This is expected, since under the null the active and traditional groups truly share the same pass rate. The shaded right tail includes the shuffles whose difference was at least as large as our observed zero point two — that’s the area we’ll use for the p-value on the next slide.
P-Value & Conclude
Compute the two-sided p-value. Smaller-tail sum: \(43 + 18 + 3 = 64\) of 1000\(64 / 1000 = 0.064\) \(2 \times 0.064 = 0.128\)
State your conclusion at \(\alpha = 0.05\) , including scope of inference . Fail to reject \(H_0\) at \(\alpha = 0.05\) . The data do not provide convincing evidence that active learning affects pass rates. If active and traditional truly had the same pass rate, we’d see a difference this extreme or more — in either direction — about 12.8% of the time by chance. Causation: randomly assigned → a significant result would have supported a causal claim. Generalization: not a random sample → conclusion applies to similar students in similar settings only.
Two questions. First, compute the two-sided p-value from the histogram. Second, state your conclusion at alpha equals zero point zero five, including scope of inference. Advance when you are ready to see the answers. The shaded tail counts sum to sixty-four out of one thousand permutations, giving a smaller-tail proportion of about zero point zero six four. Doubling gives a two-sided p-value of about zero point one two eight. Putting it together: at alpha equals zero point zero five, we fail to reject the null. The data do not provide convincing evidence that active learning affects pass rates. In plain terms: if active learning and traditional lecture truly had the same pass rate, we’d see a difference this extreme or more — in either direction — about 12.8 percent of the time just by chance. On scope of inference: students were randomly assigned to teaching method, so a significant result would have supported a causal claim. But they were not a random sample, so the conclusion would apply only to similar students in similar settings.
Stents and Stroke
A study randomly assigned 451 patients with brain blockages to receive either a stent or standard care alone . Outcome: stroke within 30 days.
Control
214
13
227
Treatment
191
33
224
Total
405
46
451
State \(H_0\) and \(H_A\) for \(p_T - p_C\) (stroke rate, treatment − control). \(H_0: p_T - p_C = 0\) \(H_A: p_T - p_C \neq 0\) no association vs. association between stent use and stroke
Compute the observed difference \(\hat{p}_T - \hat{p}_C\) (stroke rate). \(\hat{p}_T = 33/224 \approx 0.147\) \(\hat{p}_C = 13/227 \approx 0.057\) \(\hat{p}_T - \hat{p}_C \approx 0.147 - 0.057 = 0.090\) — stents are associated with a 9 percentage-point higher stroke rate
Try this next one yourself. A real study randomly assigned four hundred fifty-one patients with brain blockages to either receive a stent or just standard care, and tracked whether they had a stroke within thirty days. This is the same SAMMPRIS trial from eight B — we used the one-year follow-up there, so the stroke counts are a bit different; today we focus on the thirty-day window. The counts are in the table — advance when you are ready to see the answers. Let’s write out the hypotheses. The null says the difference between treatment and control stroke rates is zero — no association between stent use and stroke. The alternative says the difference is not zero — there is an association in some direction. Now we compute the observed difference. The treatment stroke rate is thirty-three out of two hundred twenty-four, or about zero point one four seven. The control stroke rate is thirteen out of two hundred twenty-seven, or about zero point zero five seven. Treatment minus control is zero point one four seven minus zero point zero five seven, which comes out to about zero point zero nine — a nine percentage-point higher stroke rate in the stent group than in the control group.
Stents — Conclusion
From the stent study: observed diff \(\approx 0.090\) , permutation p-value \(\approx 0.004\) .
State the conclusion at \(\alpha = 0.05\) , including scope of inference . Reject \(H_0\) at \(\alpha = 0.05\) . The data provide convincing evidence that stents affect short-term stroke rate. Causation: randomized → causal. Generalization: not a random sample → applies to similar patient populations only.
The observed difference here is about zero point zero nine, and a permutation test — the same procedure we ran for CPR, a thousand shuffles under the null with the smaller tail doubled for the two-sided p-value — gives about zero point zero zero four. With those numbers in hand, state the conclusion at alpha equals zero point zero five, including scope of inference. Advance when you are ready to see the answer. In plain terms: if stents had no effect on stroke rate, we’d see a difference this extreme or more — in either direction — only about four times in a thousand by chance. That’s well below zero point zero five, so we reject the null — the data provide convincing evidence that stents affect short-term stroke rate. On scope of inference: random assignment means stents caused the difference — counterintuitively, more strokes, not fewer. But the patients were not a random sample — they were volunteers with specific eligibility — so the conclusion applies only to similar patient populations, not to all stroke patients.
Why Permutation Works
The permutation test assumes exchangeability under the null:
If null is true, outcomes are independent of group
Shuffling outcomes simulates this independence
The permuted differences show what’s expected by chance alone
If observed difference is rare in this distribution \(\rightarrow\) evidence against null
Before we conclude, let’s step back and name what’s going on. The permutation test rests on what’s called exchangeability under the null. If blood thinner truly has no effect — the null in our CPR study — then whether a patient survived is independent of which group they were in, and shuffling labels gives us another equally valid version of the data. The permuted differences show the spread of treatment-minus-control values we’d expect by pure chance; if our observed difference is rare in that picture, that rarity is the evidence against the null.
Looking Ahead
The permutation test answers: “Is there a difference?”
Next time: a bootstrap CI for \(p_1 - p_2\) — “How big is the difference?”
That handles “is there a difference?” for two proportions. Next time we’ll tackle the second question — how big is it? Same data, same family of ideas, but a bootstrap confidence interval instead of a permutation test.
