9. Two Proportions (Randomization)

Math 115 - Examples for Topic 9

Author

Yurk

These worked examples demonstrate inference for comparing two proportions using randomization-based methods.

Example 1: Reading P-Values from a Null Distribution (Two-Sided Test)

Scenario: Researchers conducted a randomized experiment to evaluate a new vaccine for the common flu. They randomly assigned 150 volunteers to receive either the vaccine or a placebo. Over the following flu season, they tracked whether each participant contracted the flu.

The results were:

Group Got Flu No Flu Total
Vaccine 12 63 75
Placebo 21 54 75
Total 33 117 150

Proportions:

  • Vaccine group: \(\hat{p}_V = \frac{12}{75} = 0.16\)
  • Placebo group: \(\hat{p}_P = \frac{21}{75} = 0.28\)
  • Observed difference: \(\hat{p}_V - \hat{p}_P = -0.12\)

Research question: Is there evidence that the flu rate differs between the vaccine and placebo groups?

Given: Below is a null distribution showing 1000 permuted differences in proportions under the assumption that there is no association between group assignment and flu outcome.

Task: Use the histogram above to estimate the two-sided p-value and make a conclusion about whether the vaccine affects flu rates. Use \(\alpha = 0.05\).

Step 1: Identify the parameter

Let \(p_V\) = the true proportion of people who get the flu when given the vaccine

Let \(p_P\) = the true proportion of people who get the flu when given the placebo

The parameter of interest is \(p_V - p_P\), the difference in flu rates between groups.

Step 2: State the hypotheses

Since we want to test whether the vaccine affects flu rates (in either direction), this is a two-sided test:

  • \(H_0: p_V - p_P = 0\) (no difference in flu rates between groups)
  • \(H_A: p_V - p_P \neq 0\) (flu rates differ between groups)

Step 3: Calculate the sample statistic

From the sample:

\[\hat{p}_V - \hat{p}_P = 0.16 - 0.28 = -0.12\]

Step 4: Estimate the p-value using “double the smaller tail”

For a two-sided test, we use the “double the smaller tail” method:

  1. Count how many permuted differences are to the LEFT of (less than or equal to) the observed value of -0.12
  2. Count how many are to the RIGHT of (greater than or equal to) the observed value
  3. Take the smaller of these two counts
  4. Double it and divide by the total number of permutations

Reading from the Randomize output:

The Randomize module computes the two-sided p-value by doubling the smaller tail area. From the simulation results:

P-value = 0.11

Step 5: Interpret the p-value

Interpretation: If there were truly no difference in flu rates between the vaccine and placebo groups, there is approximately a 11% chance of observing a difference as extreme as -0.12 (or more extreme) in a random sample.

Step 6: Make a decision

Using the significance level \(\alpha = 0.05\):

  • P-value \(\approx 0.11\)
  • Compare: Is 0.11 < 0.05? NO

Decision: We fail to reject \(H_0\) at the \(\alpha = 0.05\) level.

Step 7: State the conclusion in context

The data do not provide convincing evidence that there is a difference in flu rates between the vaccine and placebo groups. While the vaccine group had a lower flu rate (16% vs 28%), this difference could reasonably occur by chance.

Example 2: Reading Confidence Intervals from a Bootstrap Distribution

Scenario: Using the same vaccine study from Example 1, construct a 95% confidence interval for the true difference in flu rates between the vaccine and placebo groups.

Recall:

  • Vaccine group: 12 out of 75 got the flu (\(\hat{p}_V = 0.16\))
  • Placebo group: 21 out of 75 got the flu (\(\hat{p}_P = 0.28\))
  • Observed difference: \(\hat{p}_V - \hat{p}_P = -0.12\)

Given: Below is a bootstrap distribution showing 1000 bootstrap differences in proportions, created by resampling with replacement from the original sample.

Task: Use the bootstrap distribution above to estimate a 95% confidence interval for \(p_V - p_P\).

Step 1: Identify the parameter

The parameter of interest is \(p_V - p_P\), the true difference in flu rates between those who receive the vaccine and those who receive the placebo.

Step 2: Calculate the sample statistic (point estimate)

From the sample:

\[\hat{p}_V - \hat{p}_P = -0.12\]

This is our best single estimate for the true difference.

Step 3: Understand the bootstrap distribution

The histogram shows 1000 bootstrap differences in proportions. Each bootstrap sample:

  1. Resamples the original 150 participants with replacement
  2. Calculates the difference in flu proportions for that bootstrap sample
  3. The distribution shows how the sample difference would vary due to sampling variability

Key observation: The distribution is centered near the observed difference (-0.12), not at 0.

Step 4: Find the percentiles for the 95% CI

For a 95% confidence interval, we need the middle 95% of the bootstrap distribution:

  • Lower bound: 2.5th percentile (cuts off bottom 2.5%)
  • Upper bound: 97.5th percentile (cuts off top 2.5%)

Helpful calculation: With 1000 bootstrap samples:

  • 2.5% of 1000 = 25 samples
  • The lower bound is approximately where you’ve counted 25 observations from the left
  • The upper bound is approximately where you’ve counted 25 observations from the right

Reading from the distribution:

Looking at the histogram:

  • The 2.5th percentile appears to be around -0.253
  • The 97.5th percentile appears to be around 0.011

Step 5: State the confidence interval

95% Bootstrap Percentile Confidence Interval:

\[(-0.253, 0.011)\]

Step 6: Interpret the confidence interval

Interpretation: We are 95% confident that the true difference in flu rates (vaccine - placebo) is between -0.253 and 0.011.

In practical terms:

  • The confidence interval includes 0, so we cannot determine a clear direction of effect
  • The difference could be anywhere from -25.3 to 1.1 percentage points
  • This suggests the vaccine may reduce flu rates, have no effect, or potentially increase them

Step 7: Connect to hypothesis test

Does the 95% CI include 0?

  • CI: (-0.253, 0.011)
  • 0 IS in the interval

This is consistent with our hypothesis test result: since 0 is a plausible value, we failed to reject \(H_0\).

Example 3: Hypothesis Test Using Jamovi (One-Sided Test)

Scenario: An online education platform wants to know if requiring discussion participation improves course completion rates. They randomly assigned 200 students to two groups:

  • Required group: Discussion posts are mandatory (100 students)
  • Optional group: Discussion posts are optional (100 students)

At the end of the semester, they recorded whether each student completed the course.

Group Completed Did Not Complete Total
Required 68 32 100
Optional 54 46 100
Total 122 78 200

Proportions:

  • Required group: \(\hat{p}_R = \frac{68}{100} = 0.68\)
  • Optional group: \(\hat{p}_O = \frac{54}{100} = 0.54\)
  • Observed difference: \(\hat{p}_R - \hat{p}_O = 0.14\)

Research question: Is there evidence that required discussion participation leads to higher completion rates than optional participation?

Task: Use Jamovi to conduct a hypothesis test at the \(\alpha = 0.05\) level.

Jamovi Setup

Step 1: Prepare the data

Create a data file with two variables:

  • discussion: values “required” or “optional” (100 of each)
  • completed: values “yes” or “no”

Your data should have:

  • 68 rows with discussion = “required” and completed = “yes”
  • 32 rows with discussion = “required” and completed = “no”
  • 54 rows with discussion = “optional” and completed = “yes”
  • 46 rows with discussion = “optional” and completed = “no”

Step 2: Access the Randomize Module

  1. Click on the Randomize icon in the Analyses tab
  2. Select Two Proportions - Hypothesis Test from the list

Step 3: Configure the analysis

  • Columns: Drag completed into the Columns box
  • Rows: Drag discussion into the Rows box
  • Compare: Select “rows”
  • Alternative hypothesis: Select “Group 1 > Group 2” (since we want to test if required > optional)

Under Cells, check “Row percentages” to see the proportions.

Under Simulations:

  • Set “Number of simulated samples” to 100 initially
  • Select “Dot plot” for visualization
  • Optionally seed the random number generator for reproducibility

Step 4: Interpret the visualization (100 simulations)

The dot plot shows:

  • Each dot represents the difference in proportions from one permutation
  • The vertical red dashed line shows the observed difference (0.14)
  • Red dots are permuted differences at or above the observed value
  • For a one-sided test (Group 1 > Group 2), we only count the right tail

Step 5: Increase precision

After examining the dot plot:

  • Change number of simulations to 1000
  • Change plot type to Histogram

Step 6: Example output with 1000 simulations

Simulation Results:

Simulation Results
Reps p-value
1000 0.024
Note: HA: prequired > poptional

Step 1: Identify the parameter

Let \(p_R\) = the true completion rate when discussion is required

Let \(p_O\) = the true completion rate when discussion is optional

Step 2: State the hypotheses

Since we specifically want to test if required participation leads to higher completion rates, this is a one-sided (upper-tail) test:

  • \(H_0: p_R - p_O = 0\) (no difference in completion rates)
  • \(H_A: p_R - p_O > 0\) (required participation has higher completion rate)

Step 3: Calculate the sample statistic

From the sample:

\[\hat{p}_R - \hat{p}_O = 0.68 - 0.54 = 0.14\]

Step 4: Obtain the p-value from Jamovi

From the Simulation Results table:

  • Number of simulations: 1000
  • P-value = 0.024

For a one-sided test, this p-value represents the proportion of permuted differences that are at or above the observed value of 0.14.

Step 5: Interpret the p-value

Interpretation: If there were truly no difference in completion rates between the two groups, there is approximately a 2.4% chance of observing a difference of 0.14 or larger in a random sample.

Step 6: Make a decision

Using the significance level \(\alpha = 0.05\):

  • P-value = 0.024
  • Compare: Is 0.024 < 0.05? YES

Decision: We reject \(H_0\) at the \(\alpha = 0.05\) level.

Step 7: State the conclusion in context

The data provide convincing evidence that required discussion participation leads to higher course completion rates than optional participation. Since this was a randomized experiment, we can conclude that requiring discussion causes an increase in completion rates. The difference of 14 percentage points is statistically significant.

Example 4: Confidence Interval Using Jamovi (90% Bootstrap CI)

Scenario: Using the same online learning study from Example 3, construct a 90% confidence interval for the true difference in completion rates between the required and optional discussion groups.

Recall:

  • Required group: 68 out of 100 completed (\(\hat{p}_R = 0.68\))
  • Optional group: 54 out of 100 completed (\(\hat{p}_O = 0.54\))
  • Observed difference: \(\hat{p}_R - \hat{p}_O = 0.14\)

Task: Use Jamovi to construct a 90% bootstrap percentile confidence interval.

Jamovi Setup

Step 1: Access the Randomize Module

  1. Click on the Randomize icon in the Analyses tab
  2. Select Two Proportions - Confidence Interval from the list

Step 2: Configure the analysis

  • Columns: Drag completed into the Columns box
  • Rows: Drag discussion into the Rows box
  • Compare: Select “rows”
  • Confidence level: Enter 90
  • Type: Select “bootstrap percentile”
  • Number of bootstraps: Start with 100, then increase to 1000

Step 3: Understanding the bootstrap process

For two proportions, each bootstrap sample:

  1. Resamples from the required group with replacement (100 observations)
  2. Resamples from the optional group with replacement (100 observations)
  3. Calculates the difference in completion proportions
  4. Repeats many times to build the bootstrap distribution

Step 4: Example output with 1000 bootstraps

Simulation Results:

Simulation Results
Bootstraps Lower Upper
1000 0.024 0.251
90% Bootstrap Percentile Confidence Interval

Step 1: Identify the parameter

The parameter of interest is \(p_R - p_O\), the true difference in completion rates between students with required discussion participation and those with optional participation.

Step 2: Calculate the sample statistic (point estimate)

From the sample:

\[\hat{p}_R - \hat{p}_O = 0.14\]

Step 3: Obtain the confidence interval from Jamovi

From the Simulation Results table:

  • Number of bootstraps: 1000
  • Lower bound (5th percentile): 0.024
  • Upper bound (95th percentile): 0.251

90% Bootstrap Percentile Confidence Interval: (0.024, 0.251)

Note: For a 90% CI, we use the 5th and 95th percentiles (leaving 5% in each tail).

Step 4: Interpret the confidence interval in context

Interpretation: We are 90% confident that the true difference in completion rates (required - optional) is between 0.024 and 0.251.

In practical terms:

  • The positive values indicate required discussion leads to higher completion rates
  • We’re confident the increase is somewhere between 2.4 and 25.1 percentage points

Step 5: Connect to hypothesis test (two-sided)

Does the 90% CI include 0?

  • CI: (0.024, 0.251)
  • 0 is NOT in the interval

Since 0 is not in the interval, this provides evidence against the null hypothesis of no difference for a two-sided test. However, it is important to note that a confidence level of 90% corresponds to a signficance level of 0.1. Thus, based on the confidence interval alone we would only be able to conclude that there is evidence against the null at the α=0.1 signifance level.

Step 6: Compare 90% and 95% confidence levels

If we had calculated a 95% CI instead:

  • 95% CI: (0.004, 0.266)
  • 90% CI: (0.024, 0.251)

Trade-off: The 90% CI is narrower (more precise) but we have less confidence that it contains the true parameter. The 95% CI is wider but provides more confidence.

Key Takeaways

Permutation Test for Two Proportions

  1. Purpose: Test whether there’s a difference between groups (\(H_0: p_1 - p_2 = 0\))
  2. Method: Shuffle the response values between groups (simulates null)
  3. P-value calculation:
    • One-sided: Proportion of permutations in the direction of \(H_A\)
    • Two-sided: Double the smaller tail
  4. Result: P-value for decision making

Bootstrap CI for Two Proportions

  1. Purpose: Estimate the true difference \(p_1 - p_2\)
  2. Method: Resample with replacement from each group separately
  3. CI calculation: Find the appropriate percentiles
    • 95% CI: 2.5th and 97.5th percentiles
    • 90% CI: 5th and 95th percentiles
  4. Result: Range of plausible values for \(p_1 - p_2\)

CI and Test Agreement

  • If a 95% CI excludes 0 → Reject \(H_0\) at \(\alpha = 0.05\) (two-sided)
  • If a 95% CI includes 0 → Fail to reject \(H_0\) at \(\alpha = 0.05\)
  • Same logic applies for 90% CI and \(\alpha = 0.10\)
Back to top