Recap: One-Sided Tests

Until now, we used one-sided hypothesis tests. Alternative hypothesis looked like one of the following:

• \(H_A: p > p_0\)
• \(H_A: p < p_0\)

P-value: Area in ONE tail of the normal distribution

We had a specific direction in mind for the alternative.

Two-Sided Hypotheses

Two-sided hypothesis test:

• \(H_0: p = p_0\) (parameter equals null value)
• \(H_A: p \neq p_0\) (parameter is different from null value)

The \(\neq\) symbol means “not equal to”—the true value could be greater OR less than \(p_0\).

We don’t specify a direction in the alternative hypothesis.

When to Use Two-Sided Tests

Use two-sided when:

• No prior expectation about direction
• Both directions would be scientifically interesting
• You want to detect ANY difference from the null

Default to two-sided unless you have a strong reason for one-sided.

The Problem with One Tail

If we used a one-sided test (\(H_A: p > 0.5\)):

• P-value = area to the RIGHT of our Z-score
• Only counts evidence in ONE direction

But wait… what if people were actually WORSE than chance?

• If only 32 students got it right (40%), that would also be evidence against \(H_0\)!
• A one-sided test would miss this

Both Tails Count

For \(H_A: p \neq 0.5\), evidence against \(H_0\) comes from either direction.

If \(\hat{p} = 0.60\) (Z = 1.79) OR \(\hat{p} = 0.40\) (Z = -1.79), we’d have equal evidence against \(H_0\).

Calculating Two-Sided P-values

Method: Double the smaller tail

1. Calculate \(Z = \frac{\hat{p} - p_0}{SE}\)

2. Find area to the LEFT of Z and area to the RIGHT of Z

3. Double the SMALLER of these two areas

• The two-sided p-value is twice as large as a possible one-sided one. So it is more conservative, it is harder to reject the null hypothesis

• This method works for both simulation-based and model-based inference, and for both symmetric and asymmetric distributions.

Confidence Interval

Check conditions:

• Independence: Each student responds independently ✓

• Success-Failure (Expected successes and failures):

  • \(80 \times 0.6 = 48\) ≥ 10 ✓
  • \(80 \times (1-0.6) = 32\) ≥ 10 ✓

Compute:

• \(SE_{ci} = \sqrt{\frac{0.6 \times(1-0.6)}{80}} = 0.0548\)
• General confidence Interval: \(\hat{p}\pm z*SE_{ci}\)
• 95% CI = \(\hat{p}\pm 1.96*SE_{ci} = (0.493, 0.707)\)← 0.5 is inside
• 90% CI = \(\hat{p}\pm 1.645*SE_{ci} = (0.51, 0.69)\)← 0.5 is NOT inside

Decision Errors: Four Scenarios

When we make a decision about null hypothesis (\(H_0\)), there are four possible outcomes:

Two ways to be right, two ways to be wrong.

Type I Error (False Positive)

Definition: Rejecting \(H_0\) when it’s actually true.

“False alarm” — claiming an effect that doesn’t exist.

In AI study context:

• Concluding people can detect AI images differently than chance…
• …when they actually can’t (detection rate really is 50%)

Consequence: Overestimating human detection abilities.

Type II Error (False Negative)

Definition: Failing to reject \(H_0\) when \(H_A\) is actually true.

“Missed detection” — failing to find a real effect.

In AI study context:

• Failing to conclude that people can detect AI images…
• …when they actually can (or can’t) at a rate different from 50%

Consequence: Missing important findings about human-AI interaction.

Significance Level Controls Type I Errors

The significance level α is the probability of making a Type I error when \(H_0\) is true. \[\boxed{\text{Probability of Type I Error = Significance Level} (\alpha)}\] - If we always use α = 0.05, we make Type I errors about 5% of the time when \(H_0\) is true - Smaller α → fewer Type I errors

But there’s a trade-off…

The Trade-off

We can’t minimize both error types simultaneously.

Choose α based on which error is more costly in your context.

What is Power?

• Power is the probability of correctly rejecting \(H_0\) when \(H_A\) is actually true.
• In other words: Power is he probability of detecting a real effect when it exists.
• We want high power (typically 80% or higher).
• If power is known, then we can find the probability of type II error:

\[\boxed{\text{Probabilty of Type II Error} = 1 - \text{Power}}\]

Illustration of Power

The null hypothesis distribution (H₀).

The alternative hypothesis distribution (Hₐ).

Factors Affecting Power

Three main factors increase power:

1. Larger sample size → more precision → easier to detect effects

2. Larger true effect → bigger differences are easier to spot

3. Larger α → easier to reject \(H_0\) (but more Type I errors!)

For our AI study: With only 80 students and an observed 10% difference from chance, we may not have had enough power to detect a real (but small) effect.

Which Error Matters More?

It depends on context!

Prioritize avoiding Type I errors when:

• False positives are dangerous or costly
• Example: Approving an ineffective medical treatment

Prioritize avoiding Type II errors when:

• Missing a real effect is dangerous
• Example: Failing to detect a disease in screening

Choose your significance level α accordingly.

CI and Two-Sided Test Connection

A \((1-\alpha)\) confidence interval and a two-sided test at level α give consistent results:

• If the CI contains \(p_0\) → Don’t reject \(H_0\) (p-value > α)
• If the CI excludes \(p_0\) → Reject \(H_0\) (p-value ≤ α)

Visualizing the Connection

Our AI study: 95% CI = (0.493, 0.707) contains 0.5 → consistent with failing to reject \(H_0\).

CIs Give More Information

The confidence interval tells us more than just the hypothesis test:

• Direction: Is the effect positive or negative?
• Magnitude: How big might the effect be?
• Plausible values: What’s the range of reasonable estimates?

Recommendation: Always report the CI, not just the p-value!

Our AI study: We’re 95% confident the true detection rate is between 49.3% and 70.7%.

Summary

References

• Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
• Chapter 14

AI Image Detection Research:

• How good are humans at detecting AI-generated images? - arXiv, July 2025
• As Good as a Coin Toss - Communications of the ACM, September 2025