9. Inference for Two Proportions

What We’ve Done vs. What We’ll Do

Previously: Inference for a single proportion

• One binary categorical variable
• Hypothesis testing: \(H_0: p = p_0\) (single value)
• Confidence interval: What is \(p\)?

Now: Inference for comparing two proportions

• Two binary categorical variables
• Hypothesis testing: \(H_0: p_1 - p_2 = 0\) (no difference between groups)
• Confidence interval: What is \(p_1 - p_2\)?

Two Binary Categorical Variables

When we have two binary categorical variables:

• One explanatory variable (defines the groups)
• One response variable (defines success/failure)

We compare the proportion of successes between groups

Examples:

• Stent vs No Stent \(\rightarrow\) Stroke rate
• Drug vs Placebo \(\rightarrow\) Symptom improvement
• With sign vs Without sign \(\rightarrow\) Recycling contamination

A Different Approach for Null Distributions

For single proportion: We used a spinner to simulate outcomes under the null

• \(H_0: p = 0.10\) \(\rightarrow\) set spinner to 10% success
• Generate many samples, build null distribution

For two proportions: We’re testing \(H_0: p_1 - p_2 = 0\)

• We don’t know \(p_1\) or \(p_2\), just that they’re equal under null
• Can’t build a spinner without knowing the probabilities
• Need a different approach: permutation test

The Key Insight

Under the null hypothesis (\(p_T = p_C\)):

• The groups have the same survival rate
• Whether a patient is in treatment or control doesn’t affect their outcome
• The outcome is independent of group assignment

Key idea: If group doesn’t matter, we can shuffle the outcomes!

Permutation: Shuffling Response Values

A permutation randomly reassigns the response values (outcomes):

• Group assignments stay fixed (40 treatment, 50 control)
• We randomly shuffle which outcome goes with which patient
• This simulates “what if the outcome had nothing to do with group”

After shuffling, we calculate the difference in proportions

This gives us ONE value from the null distribution

Building the Null Distribution

• Repeat the permutation many times (1000+)
• Each permutation gives a difference in proportions
• Distribution of all these differences = null distribution

The null distribution shows what differences we’d expect if there’s truly no effect (i.e., if \(H_0\) is true)

Calculating the P-value

Method: Double the smaller tail

For a two-sided test (\(H_A: p_T - p_C \neq 0\)):

• Count permutations to the LEFT of 0.13: 951
• Count permutations to the RIGHT of 0.13: 117
• Smaller tail: 117
• P-value = \(2 \times \frac{117}{1000} = 0.234\)

Why Permutation Works

The permutation test assumes exchangeability under the null:

• If null is true, outcomes are independent of group
• Shuffling outcomes simulates this independence
• The permuted differences show what’s expected by chance alone
• If observed difference is rare in this distribution \(\rightarrow\) evidence against null

From Testing to Estimation

The permutation test answers: “Is there a difference?”

But we also want to know: “How big is the difference?”

Solution: Bootstrap confidence interval for \(p_1 - p_2\)

• Gives a range of plausible values for the true difference
• More informative than just a yes/no answer

Bootstrap Approach

Similar to single proportion bootstrap:

1. Resample WITH replacement from the original sample
2. Calculate statistic of interest (now: difference in proportions)
3. Repeat many times to build the bootstrap distribution

CI and Test Agreement

The confidence interval and hypothesis test give consistent results:

For CPR study:

• 95% CI: (-0.065, 0.311) includes 0
• P-value \(\approx 0.23 > 0.05\) \(\rightarrow\) Don’t reject \(H_0\)
• Both results indicate that 0 is a plausible value for \(p_T-p_C\)

In-class Exploration: Treating Warts

• Research question: Is application of duct tape as effective as cryotherapy (freezing with liquid nitrogen) after 2 months of treatment for common warts?
• 2 variables
  • Explanatory–treatment: Duct Tape or Cryotherapy
  • Response–cured : Cured or Not
• 51 individuals (26 treated with duct tape, 25 with cryotherapy)

Results (EDA)

Difference in proportions

• Success: result = Cured
• Statistic of interest: difference in proportions \[\hat{p}_T-\hat{p}_C\]
• Observed difference: \[\frac{22}{26}-\frac{15}{25}=0.2462\]

Hypotheses

Simulation of a Null Distribution Using Cards

• We can perform a physical simulation of the duct tape therapy vs. cryotherapy treatment of warts
• Form six groups with a deck of playing cards for each group
• There are 51 observations in this study, so remove one of the aces and the jokers from your deck
• There are 37 patients who were cured and 14 patients who were not.
• We’ll use the red cards from 2 through 8 to represent the 14 who weren’t cured
• The other cards represent the 37 who were cured

• To simulate the null hypothesis being true (no difference between the therapies), shuffle the cards and put them in two piles of 26 (duct tape) and 25 (cryotherapy)
• Count the number not cured in the duct tape sample (the pile of 26)
• Determine how many were cured and calculate the proportion cured in each pile (group)
• And the sample statistic which is the difference in proportions, \[\hat{p}_T-\hat{p}_C\]
• Do this twice in your group and record your statistics in the IE 9 MyOpenMath worksheet

Null Distribution Using Random Permutation

• Mixing up the values of the response variable we simulated for the treating warts study is called random permutation
• We used random permutation with the playing cards to create a null distribution
• Now do this using Jamovi with 1,000 random permutations
• Find the p-value and report it in your IE9 worksheet

Minimizing Confounding Variables

Permutation vs Bootstrap

References

• Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
• Chapter 11
• Sections 17.1, 17.2