7. Two-Sided Tests and Decision Errors

Math 115 - Examples for Topic 7

Author

Yurk

These worked examples demonstrate two-sided hypothesis tests using both simulation-based and model-based approaches, the connection between confidence intervals and hypothesis tests, and the power of a test.

Example 1: Simulation-Based Two-Sided Test

Scenario: A basketball coach claims that one of her players makes 85% of free throws. During a practice session, you observe the player attempting 25 free throws and making 18 of them.

You want to test whether the player’s true free throw percentage is different from the claimed 85%—perhaps she’s having an off day, or perhaps the coach overestimated her ability.

Question: Is there convincing evidence that the player’s true free throw percentage differs from 85%? Use a significance level of α = 0.05.

Step 1: State the hypotheses

Let \(p\) = the true proportion of free throws this player makes.

Since we want to detect if the rate is different from 85% (in either direction), this is a two-sided test:

  • \(H_0: p = 0.85\) (the coach’s claim is correct)
  • \(H_A: p \neq 0.85\) (the true rate differs from 85%)

Step 2: Calculate the sample statistic

From the sample:

\[\hat{p} = \frac{18}{25} = 0.72\]

Step 3: Check technical conditions for normal model

Independence: Each free throw attempt is independent. ✓

Success-Failure Condition (using \(p_0 = 0.85\)):

  • Expected successes: \(n \times p_0 = 25 \times 0.85 = 21.25\) ≥ 10 ✓
  • Expected failures: \(n \times (1-p_0) = 25 \times 0.15 = 3.75\) ≥ 10 ✗

The success-failure condition is NOT met (expected failures = 3.75 < 10).

We cannot use the normal model. Instead, we’ll use a simulation-based approach.

Step 4: Conduct a simulation-based hypothesis test

Using Jamovi’s Randomize Module

  1. Prepare the data: Enter a variable called made with values yes (made the shot) and no (missed). You should have 25 observations: 18 with yes and 7 with no.

  2. Access the Randomize Module: Click on Randomize → Single Proportion - Hypothesis Test

  3. Configure the analysis:

    • Variable: Drag made into the Variable box
    • Test value: Enter 0.85
    • Alternative hypothesis: Select Not equal (for two-sided test)
    • Number of simulated samples: Use 1000
    • Plot type: Select Histogram

  4. Interpret the output: Jamovi will calculate a two-sided p-value by doubling the smaller tail area.

Step 5: View the null distribution

Below is the null distribution generated by 1000 simulations, assuming the true free throw rate is 85%:

Reading the histogram:

  • Each bar shows how often a particular sample proportion occurred in simulations
  • The vertical red dashed line marks our observed proportion (0.72)
  • Jamovi counts results below the observed proportion (to the left of the line) and above the observed proportion (to the right of the line) and determines which count is smaller
  • The red-shaded bars show simulated proportions in the smaller left tail, which will be used to calculate the p-value

Step 6: Calculate the p-value

The Single Proportion - Hypothesis Test analysis produces a table of simulation results similar to the following:

Simulation Results
Reps p-value
1000 0.11
Note: HA: p ≠ 0.85

Two-sided p-value: The Randomize module computes the two-sided p-value by doubling the smaller tail area. The resulting p-value is 0.11.

Step 7: Interpret the p-value

Interpretation: If the player truly makes 85% of her free throws, there is approximately a 11% chance of observing a sample proportion as extreme as 0.72 (in either direction) in a random sample of 25 free throws.

Step 8: Make a decision

Using the significance level α = 0.05:

  • P-value = 0.11
  • Compare: Is 0.11 < 0.05? NO

Decision: We fail to reject \(H_0\) at the α = 0.05 level.

Step 9: State the conclusion in context

The data do not provide convincing evidence that the player’s true free throw percentage differs from 85%. While the observed rate of 72% is lower than 85%, this difference could reasonably occur by chance (p-value = 0.11).

Example 2: Model-Based Two-Sided Test

Scenario: A political analyst is studying whether public opinion on a local policy has changed since a major news event. Historically, 55% of voters in the district supported this policy. After the news event, a random sample of 400 voters finds that 196 support the policy.

Question: Is there evidence that support for the policy has changed from the historical 55%? Use α = 0.05.

Step 1: State the hypotheses

Let \(p\) = the true proportion of voters who currently support the policy.

Since we want to detect if support has changed (increased OR decreased), this is a two-sided test:

  • \(H_0: p = 0.55\) (support unchanged from historical rate)
  • \(H_A: p \neq 0.55\) (support has changed)

Step 2: Calculate the sample statistic

From the sample:

\[\hat{p} = \frac{196}{400} = 0.49\]

Step 3: Check conditions for normal model

Independence: Random sample of voters. ✓

Success-Failure Condition (using \(p_0 = 0.55\)):

  • Expected successes: \(n \times p_0 = 400 \times 0.55 = 220\) ≥ 10 ✓
  • Expected failures: \(n \times (1-p_0) = 400 \times 0.45 = 180\) ≥ 10 ✓

Both conditions are met, so we can use the normal model.

Step 4: Calculate the test statistic

Standard Error (using \(p_0\)):

\[SE = \sqrt{\frac{p_0(1-p_0)}{n}} = \sqrt{\frac{0.55 \times 0.45}{400}} = 0.0249\]

Z-score:

\[Z = \frac{\hat{p} - p_0}{SE} = \frac{0.49 - 0.55}{0.0249} = -2.41\]

Step 5: Calculate the two-sided p-value using Jamovi

Randomize module → Model-Based Inference

To calculate the p-value in Jamovi:

  1. Open the Randomize module and select Model-Based Inference
  2. Under Distribution, choose Standard normal (Z)
  3. Check “Calculate area (p-value)”
  4. Enter the Z-score: -2.41
  5. Select tail direction: Both (for a two-sided test)

Jamovi displays the two-sided p-value ≈ 0.0159

Step 6: Make a decision

Using the significance level α = 0.05:

  • P-value = 0.0159
  • Compare: Is 0.0159 < 0.05? YES

Decision: We reject \(H_0\) at the α = 0.05 level.

Step 7: State the conclusion in context

The data provide convincing evidence that voter support for the policy has changed from the historical rate of 55%. The current sample proportion of 0.49 is significantly different from 0.55 (Z = -2.41, p-value = 0.0159). Support appears to have decreased.

Example 3: CI and Two-Sided Test Connection

A confidence interval and a two-sided hypothesis test provide consistent information. This example explores that relationship.

Key Principle

For a two-sided test at significance level α:

  • If the \((1-\alpha)\) CI contains \(p_0\) → Fail to reject \(H_0\) (p-value > α)
  • If the \((1-\alpha)\) CI excludes \(p_0\) → Reject \(H_0\) (p-value ≤ α)

Part A: From CI to Hypothesis Test Decision

Given: A 95% confidence interval for a population proportion is (0.42, 0.58).

Question: At the α = 0.05 significance level, would you reject or fail to reject \(H_0: p = 0.40\) in a two-sided test?

Check: Does the 95% CI contain the null value of 0.40?

  • CI: (0.42, 0.58)
  • Null value: 0.40
  • Is 0.40 inside the interval (0.42, 0.58)? NO (0.40 < 0.42)

Answer: Since the 95% CI excludes 0.40, we would reject \(H_0: p = 0.40\) at the α = 0.05 level. The two-sided p-value is less than 0.05.

Part B: From CI to Hypothesis Test Decision

Given: A 95% confidence interval for a population proportion is (0.52, 0.68).

Question: At the α = 0.05 significance level, would you reject or fail to reject \(H_0: p = 0.55\) in a two-sided test?

Check: Does the 95% CI contain the null value of 0.55?

  • CI: (0.52, 0.68)
  • Null value: 0.55
  • Is 0.55 inside the interval (0.52, 0.68)? YES (0.52 < 0.55 < 0.68)

Answer: Since the 95% CI contains 0.55, we would fail to reject \(H_0: p = 0.55\) at the α = 0.05 level. The two-sided p-value is greater than 0.05.

Part C: From P-value to CI Information

Given: A two-sided hypothesis test of \(H_0: p = 0.30\) yields a p-value of 0.03.

Question: Would a 95% confidence interval for \(p\) contain the value 0.30? Explain.

Reasoning:

  • The p-value of 0.03 is less than α = 0.05
  • This means we reject \(H_0\) at the α = 0.05 level
  • If we reject, then the null value must be outside the corresponding CI

Answer: No, a 95% confidence interval would not contain 0.30. Since the p-value (0.03) is less than 0.05, we reject the null hypothesis, which means the null value of 0.30 falls outside the 95% CI.

Part D: From P-value to CI Information

Given: A two-sided hypothesis test of \(H_0: p = 0.65\) yields a p-value of 0.12.

Question: Would a 95% confidence interval for \(p\) contain the value 0.65? Explain.

Reasoning:

  • The p-value of 0.12 is greater than α = 0.05
  • This means we fail to reject \(H_0\) at the α = 0.05 level
  • If we fail to reject, then the null value must be inside the corresponding CI

Answer: Yes, a 95% confidence interval would contain 0.65. Since the p-value (0.12) is greater than 0.05, we fail to reject the null hypothesis, which means the null value of 0.65 falls inside the 95% CI.

Part E: Interpreting Results

Scenario: A researcher conducts a two-sided test of whether a new teaching method affects exam pass rates compared to the historical rate of 70%. The sample proportion is \(\hat{p} = 0.78\) with a 95% CI of (0.71, 0.85).

Questions:

  1. What is the decision at α = 0.05?
  2. Is the p-value greater or less than 0.05?
  3. What does this tell us about the teaching method?

1. Decision:

  • Check: Does the 95% CI (0.71, 0.85) contain \(p_0 = 0.70\)?
  • Is 0.70 inside (0.71, 0.85)? NO (0.70 < 0.71)
  • Decision: Reject \(H_0\) at α = 0.05

2. P-value:

Since we reject at α = 0.05, the p-value must be less than 0.05.

3. Interpretation:

There is convincing evidence that the new teaching method produces a different pass rate than the historical 70%. The 95% CI of (71%, 85%) suggests the new method likely increases pass rates, since all plausible values are above 70%.

Example 4: Power — The Chance of Catching a Real Effect

Power is the probability that a test rejects \(H_0\) when \(H_A\) is really true — the chance of catching a real effect. It is the complement of a Type II error:

\[\text{power} = 1 - P(\text{Type II error})\]

Scenario: A lab will test whether a water-treatment step changes a contaminant concentration, at \(\alpha = 0.05\). Three possible study designs would use the same test:

Design Sample size True effect α
1 n = 30 small 0.05
2 n = 120 small 0.05
3 n = 120 large 0.05

Question: Rank the three designs from lowest to highest power, and explain what changes the power between each pair.

Ranking (lowest to highest power): Design 1 < Design 2 < Design 3.

From Design 1 to Design 2, only the sample size changes. More data shrinks the standard error, so the same true effect stands out more clearly against sampling variability — larger n raises power, with no error-rate trade-off.

From Design 2 to Design 3, only the true effect changes. A bigger real effect is simply easier to detect — larger effects mean higher power. This is not a knob the lab controls, but it determines how much data is needed: small effects require large samples.

What about α? No design here varies it, but it matters too: lowering α lowers power. Demanding stronger evidence before rejecting makes real effects harder to catch. Shrinking the Type I error rate raises the Type II error rate, and power falls with it.

Back to top