Randomize Guide: Hypothesis Test for Two Proportions

Filter, recode, and run a permutation test for a difference in two proportions in Jamovi with the Randomize module, using practice survey data

Note

This guide runs a permutation test for a difference between two proportions in Jamovi with the Randomize module: filter out the blanks, recode a survey question into two levels, run the test, and read the p-value. The example uses two questions from the practice survey data. With that file open you should get the same table and the same difference. Your p-value will be close to the one shown but not identical, because it comes from a simulation.

The worked example

The survey asked how much sleep you typically get on a school night (sleep_weeknight, in hours) and whether you ate breakfast today (ate_breakfast, Yes or No). Of the 156 students who answered both, 17 of the 32 who sleep less than seven hours skipped breakfast (53%), compared with 30 of the 124 who sleep seven or more (24%). The difference is 0.29. The two categorical variables guide shows how to get those numbers.

Research question: is a gap that size more than chance would produce if sleep and breakfast were unrelated?

\[H_0: p_1 - p_2 = 0 \qquad H_A: p_1 - p_2 \neq 0\]

where $p_1$ is the proportion who skip breakfast among students who sleep less than seven hours, and $p_2$ is the proportion among those who sleep seven or more.

How the test works

If $H_0$ is true, sleep has nothing to do with breakfast, and the 47 “skipped” answers could have belonged to any 47 of the 156 students. The test keeps the two groups the same size (32 and 124), hands the 47 “skipped” and 109 “ate” answers out to the students at random, and computes the difference in proportions. It does that 1000 times. The 1000 differences show what chance alone produces when there is no association. For a two-sided test, the p-value is twice the fraction of them at least as large as the 0.29 we observed.

What you need

Jamovi 2.7.38 with the Randomize module installed. If Randomize is not in your Analyses ribbon, see the Jamovi Setup Guide.
The practice survey file, practice_survey.csv. Open it with the hamburger menu (three lines, top left), Open, then Browse.

Step 1: Filter out the blanks

A few students skipped one of the two questions. Their rows have to come out before anything is counted.

Click the Data tab, then the Filters icon.
In the field next to the $f_x$ button, type:

sleep_weeknight != NA and ate_breakfast != NA
Make sure the toggle labeled active is green.

Figure 1: The filter keeps the students who answered both questions.

Tip

Check your work. The status bar below the spreadsheet should read Row count 160 and Filtered 4, which leaves 156 students.

Step 2: Recode sleep into two levels

The survey recorded sleep in hours, and the test needs two groups.

Click the Variables tab. Type sleep_weeknight in the search box and check its box.
Click Transform.
Change the name of the new variable to sleep7.
In the using transform pull-down, choose Create New Transform….
In the field next to $f_x$, type:

IF($source < 7, "less than seven", "seven or more")

Figure 2: The recode. Anything under 7 hours becomes “less than seven” and everything else “seven or more”.

Randomize takes the first level of the row variable as group 1 and counts the first level of the column variable. Jamovi puts levels in alphabetical order, so the labels set which group comes first and which level is counted:

less than seven sorts before seven or more, so the short sleepers are group 1.
For ate_breakfast, No sorts before Yes, so the test counts the students who skipped breakfast.

If your own question is about the other level, the two categorical variables guide shows how to change the order.

Step 3: Open the test

Click the Analyses tab, then Randomize, then Two Proportions - Hypothesis Test.

Step 4: Set up the test

Use the search box above the variable list to find each variable, select it, and click the arrow next to the box it belongs in. Then expand Simulations.

Option	Value	Why
Rows	`sleep7`	the explanatory variable, recoded
Columns	`ate_breakfast`	the response variable
Compare	rows	compare the two sleep groups
Alternative hypothesis	Group 1 ≠ Group 2	$H_A: p_1 - p_2 \neq 0$
Number of simulated samples	`1000`	enough to see the shape of the null distribution
Plot differences in proportions	Histogram	1000 dots would overlap, so bin them

Leave Counts (optional) empty, and leave Show counts on plot and Seed random number generator unchecked.

Figure 4: The setup and the first part of the output: the table of counts and the observed difference.

Tip

Check your work. The contingency table should show 17, 15, and 32 in the first row, 30, 94, and 124 in the second, and 156 in total. Difference in 2 proportions should be 0.28931, which is $17/32 - 30/124$.

Step 5: Read the output

Figure 5: The simulation settings and the histogram of 1000 shuffled differences. The dashed line is the observed difference.

The null distribution. The histogram shows the 1000 differences from shuffling. It is centered at 0, because shuffling breaks any link between sleep and breakfast, and most of it falls between about −0.2 and 0.2. The dashed red line marks the observed difference, 0.29, out in the right tail. Almost nothing lies past it.

The p-value. For a two-sided test, Randomize doubles the proportion of shuffles at least as large as the observed difference, as the note under the plot says.

In this run, 1 of the 1000 shuffles reached 0.29, so the p-value is $2 \times 1/1000 = 0.002$. Each run shuffles again, so yours will differ. With 1000 shuffles, values from below 0.001 to about 0.01 are all normal here. The exact value, over every possible shuffle, is about 0.004.

The conclusion

The p-value is well below 0.05, so we reject $H_0$. There is strong evidence that students who sleep less than seven hours skip breakfast more often than those who sleep seven or more.

The test doesn’t tell us why short sleepers skip breakfast more often. Nobody was assigned an amount of sleep, so short sleep might lead to skipping breakfast, or something else, like an early class, might lead to both.

If your numbers don’t match

What you see	What went wrong
The status bar doesn’t read Filtered 4	The filter is not active, or it mentions only one of the two variables. Both `!= NA` conditions belong in one filter line
Difference in 2 proportions is −0.28931	One level order is flipped: `seven or more` is the first row, or Yes is the first column. The two-sided p-value doesn’t change, but the difference now means something else. Fix the labels so the order matches your question.
A table with many rows, and no difference or histogram	`sleep_weeknight` is in Rows instead of `sleep7`. The test needs two levels in each variable.
A p-value about half the size you expected	The alternative hypothesis is Group 1 > Group 2, a one-sided test.
A p-value near 1	The alternative hypothesis is Group 1 < Group 2.
A p-value different from 0.002	That’s expected, because each run shuffles again.
No Randomize in the Analyses ribbon	The module is not installed. See the Jamovi Setup Guide.

Practice on a second question

Do coffee drinkers nap more? The survey columns are coffee_drink (Do you regularly drink coffee?) and nap_past3d (Have you taken a nap in the past 3 days?). Both are already Yes or No, so neither needs a recode.

Filter: coffee_drink != NA and nap_past3d != NA, which leaves 158 students.
Rows coffee_drink, Columns nap_past3d, Compare rows, Group 1 ≠ Group 2, 1000 simulated samples, Histogram.

No sorts before Yes in both columns, so group 1 is the students who don’t drink coffee, and the level counted is No, didn’t nap. You should get 45 of 85 who don’t drink coffee and didn’t nap (0.529), against 33 of 73 coffee drinkers (0.452), a difference of 0.077. Counting the students who napped instead gives 55% of coffee drinkers against 47% of everyone else, and the gap is the same.

Your p-value should be around 0.42; the exact value is 0.418. That is well above 0.05, so we fail to reject $H_0$. With groups this size, a gap of about 8 percentage points is something shuffling produces fairly often.