9. Inference for Two Proportions (Randomization)
b. Bootstrap CI
Recap: CPR Study
Randomized trial: 90 cardiac-arrest patients receiving CPR, assigned to blood thinner (treatment, \(n = 40\) ) or standard care (control, \(n = 50\) ). Response: survived at 24 hours?
Control
39
11
50
Treatment
26
14
40
Total
65
25
90
Observed difference: \(\hat{p}_T - \hat{p}_C = 0.35 - 0.22 = 0.13\)
9a permutation test: p-value \(\approx 0.27\) → failed to reject \(H_0\)
Before we get into bootstrap, a quick recap of the study we’re working with. Recall the CPR trial — ninety cardiac-arrest patients randomly assigned to blood thinner or standard care, with twenty-four-hour survival as the response. Treatment had fourteen survivors out of forty; control had eleven out of fifty — an observed gap of about zero point one three in favor of the thinner. The permutation test gave a p-value around zero point two seven, so we failed to reject the null — chance alone could plausibly produce a gap that size.
From Testing to Estimation
The permutation test answers: “Is there a difference?”
But we also want to know: “How big is the difference?”
Solution: Bootstrap confidence interval for \(p_1 - p_2\)
Gives a range of plausible values for the true difference
More informative than just a yes/no answer
The permutation test answered the yes-or-no question — is there a difference in proportions. But there’s a second question we haven’t tackled. How big is that difference, even if we can’t rule out zero? That’s an estimation problem, and the natural tool is a confidence interval. To build one we’ll switch from permutation to bootstrap.
Bootstrap Approach
Same idea as the single-proportion bootstrap from topic 4:
Throw each case (group + outcome) into a hat
Draw 90 cases with replacement — some come up more than once, some not at all
Compute the new \(\hat{p}_T - \hat{p}_C\)
Repeat 1000 times → bootstrap distribution
Unlike permutation: each case keeps its group and its outcome. No shuffling to break the association.
Same mechanics as the single-proportion bootstrap from topic four. We throw each case into a hat, then draw a new sample one case at a time — replacing each case after it’s drawn, so some cases come up more than once and some don’t come up at all. Notice — unlike the permutation test, each case keeps its group and its outcome. There’s no shuffling to break the association. A thousand resamples gives us the spread of statistics we call the bootstrap distribution.
Why Does Bootstrap Work?
Goal: Know how \(\hat{p}_T - \hat{p}_C\) would vary across repeated samples from the population.
Problem: We only have one sample — the population is out of reach.
Bootstrap idea: The sample is our best stand-in for the population. Resampling from the sample mimics drawing repeated samples from the population.
→ Spread of bootstrap statistics ≈ spread of \(\hat{p}_T - \hat{p}_C\) across real repeated samples → use that spread to build a CI.
Step back for a moment — why does this resampling trick actually work? Here’s the goal: we want to know how the difference in proportions would vary if we could draw fresh samples from the population. The problem is we only have one sample — the population is out of reach. The bootstrap idea is that the sample we have is our best estimate of the population. So resampling from the sample mimics drawing repeated samples from the population. The spread of bootstrap statistics gives us a good approximation of how the statistic actually varies across real repeated samples — and that spread is exactly what we need to build a confidence interval.
Row Mechanics
Permutation
Group stays · outcomes shuffled
1
Tx
S D
2
Tx
D D
3
Tx
D D
4
Tx
S D
5
Tx
S D
6
Tx
D S
7
Ctl
D S
8
Ctl
D S
9
Ctl
S D
10
Ctl
D S
11
Ctl
D S
12
Ctl
S D
Bootstrap
Rows drawn with replacement
1
Tx
S
—
2
Tx
D
—
3
Tx
D
—
4
Tx
S
—
5
Tx
S
—
6
Tx
D
—
7
Ctl
D
—
8
Ctl
D
—
9
Ctl
S
—
10
Ctl
D
—
11
Ctl
D
—
12
Ctl
S
—
draw rows with replacement
To see what makes permutation and bootstrap different at the row level, we’ll work with an illustration — a toy stand-in for the CPR sample, just twelve cases instead of ninety so every row fits on screen. Each row carries a group label and an outcome. The mechanics we’re about to demonstrate apply the same way to the full ninety-patient study. Permutation shuffles the outcomes — the group column holds, every outcome moves once and lands somewhere new. No outcome is duplicated or skipped. That’s how it simulates the null: the link between group and outcome is broken. Bootstrap draws whole rows with replacement — group and outcome stay together. Some rows get picked multiple times, some not at all. We’re sampling new datasets from the original, treating it as a stand-in for the population.
CPR Bootstrap Demo
View interactive bootstrap demo
The demo shows:
Original 90 patients — shapes = group (○ treatment, □ control), colors = outcome (green survived, gray died)
Resampling WITH replacement from entire sample
Group sizes vary in each bootstrap sample
Building the bootstrap distribution
There’s an interactive demo linked here that shows the resampling visually. All ninety CPR patients pooled together, then sampled with replacement to make a new bootstrap sample. Notice the group sizes wobble a bit each round — that’s expected and matches what the bootstrap is doing.
CPR Bootstrap Distribution
1000 bootstraps. Centered near observed difference (0.13), NOT at 0
And here’s the bootstrap distribution itself for the CPR data — a thousand resamples. Notice it’s centered near the observed difference, about zero point one three, not at zero. That’s the key difference from the permutation distribution from nine A. The bootstrap doesn’t assume the null is true — it estimates uncertainty around what the data actually showed.
CPR Confidence Interval
95% Bootstrap Percentile CI:
Find the 2.5th and 97.5th percentiles of the bootstrap distribution:
\[(-0.064, 0.325)\]
Interpretation: We are 95% confident that the true difference in survival rates (treatment − control) is between -0.064 and 0.325.
To get the ninety-five percent confidence interval, we read off the two and a half percentile and the ninety-seven and a half percentile of the bootstrap distribution. For CPR that’s about negative zero point zero six four up to zero point three two five. We’re ninety-five percent confident the true difference in survival rates falls in this range. Next slide we’ll connect this back to the hypothesis test from nine A.
CI and Test Agreement
The confidence interval and hypothesis test give consistent results :
For CPR study:
95% CI: (-0.064, 0.325) includes 0 P-value \(\approx 0.27 > 0.05\) \(\rightarrow\) Don’t reject \(H_0\) Both results indicate that 0 is a plausible value for \(p_T-p_C\)
Now let’s check whether the confidence interval and the hypothesis test from nine A tell the same story for CPR. The interval runs from about negative zero point zero six four up to zero point three two five — it includes zero, meaning zero is a plausible value for the true difference. The p-value from nine A was about zero point two seven, well above zero point zero five, so we don’t reject the null. Both methods agree — “no effect of blood thinner on survival” is still a plausible explanation.
Estimate the CI Bounds
The next slide shows the bootstrap distribution for the active-learning study from 9a (1000 resamples; observed difference \(\approx 0.20\) ).
Estimate the 95% CI bounds by counting bars from each tail.
Hint: a 95% CI excludes about 25 resamples from each tail (\(2.5\% \times 1000 = 25\) ). Advance for the walkthrough.
Time to try the percentile method yourself. The next slide shows the bootstrap distribution from the same active-learning study you worked on in nine A — a thousand resamples. Each bar is labeled with how many resamples landed in it. To get the ninety-five percent confidence interval bounds, count bars from each tail until the cumulative total reaches about twenty-five — that’s two and a half percent of one thousand. Write down both bounds, then advance for the walkthrough.
Reading the Bounds
Left tail counts placeholder
Left tail counts: 1 + 2 + 4 + 3 + 2 + 13 = 25
Left 95% bound \(\approx -0.015\)
Right tail counts (from right): 1 + 5 + 7 + 10 = 23
95% CI: \((-0.015,\ 0.411)\) — includes 0; fail to reject (matches 9a).
Here’s the bootstrap distribution for the active-learning study. Pause and count bars from each tail until the cumulative total reaches about twenty-five, write down both bounds, and advance when you’re ready to walk through it together. Let’s read the bounds together. Counting from the left, the first few bars accumulate to about twenty-five resamples — 1, 2, 4, 3, 2, 13 adds to 25. That’s our two and a half percent tail. So the left ninety-five percent bound sits right where that tail ends, at about -0.015. Counting from the right the same way, the first four bars get close to twenty-five, but not quite — 1, 5, 7, 10 adds to 23. (Just short of twenty-five, but the next bar over would push us past.) The right ninety-five percent bound sits at about 0.411. So the ninety-five percent confidence interval for the difference in pass rates between active learning and traditional lecture runs from about -0.015 to 0.411. The interval just barely includes zero, which lines up with the permutation hypothesis test from nine A — fail to reject.
Stents — Interpret a Bootstrap CI
For the stents-and-stroke study from 9a (treatment vs. control, stroke within 30 days), a bootstrap procedure gives a 95% confidence interval for \(p_T - p_C\) (stroke rate):
\[(0.040, \; 0.150)\]
Does the CI include 0? No. Both endpoints are positive.
Interpret the CI in context. We are 95% confident that the true difference in stroke rates (treatment − control) is between 0.040 and 0.150 — stents are associated with a 4 to 15 percentage-point higher stroke rate.
Is this consistent with the H-test result (p-value \(\approx 0.004\) )? Yes. Both reject \(H_0\) : the CI excludes 0, and the p-value is below \(\alpha = 0.05\) . They agree that 0 is not a plausible value for the true difference.
Now for your turn — same stent study from nine A, but now we’re computing a confidence interval. A bootstrap procedure gave a ninety-five percent confidence interval for the difference in stroke rates — treatment minus control — running from zero point zero four to zero point one five. There are three quick questions to work through before clicking ahead. First, does the confidence interval include zero? It does not. Both endpoints are positive — the lower bound is zero point zero four and the upper is zero point one five, so zero sits outside the interval. Next we interpret the confidence interval in context. We’re ninety-five percent confident that the true difference in stroke rates — treatment minus control — is between zero point zero four and zero point one five. Stents are associated with somewhere between a four and fifteen percentage-point higher stroke rate than standard care. Finally, does this agree with the hypothesis test, which had a p-value of about zero point zero zero four? Yes — both reject the null. The confidence interval excludes zero, and the p-value is below alpha. They agree that zero is not a plausible value for the true difference.
Permutation vs Bootstrap
Purpose Hypothesis testing
Estimation
Question Is there a difference?
How big is the difference?
Assumes Null is true
Nothing about null
Method Shuffle response values
Resample with replacement
Distribution center 0
Observed difference
Result P-value
Confidence interval
A side-by-side of the two methods. Permutation is for testing — it assumes the null is true and asks whether the observed difference is unusual under that assumption. Bootstrap is for estimation — it makes no claim about the null and just describes the spread around our actual sample. That difference in purpose is why the permutation distribution is centered at zero while the bootstrap distribution is centered at the observed difference. On the CPR data both methods landed on fail-to-reject — the permutation p-value was about zero point two seven and the bootstrap confidence interval included zero. Topic ten will redo both of these with the normal model — a mathematical shortcut that gets us to the same answers when conditions are met.
