7. Two-Sided Tests and Decision Errors

Recap: One-Sided Tests

Until now, we used one-sided hypothesis tests. Alternative hypothesis looked like one of the following:

• \(H_A: p > p_0\)
• \(H_A: p < p_0\)

P-value: Area in ONE tail of the normal distribution

We had a specific direction in mind for the alternative.

Two-Sided Hypotheses

Two-sided hypothesis test:

• \(H_0: p = p_0\) (parameter equals null value)
• \(H_A: p \neq p_0\) (parameter is different from null value)

The \(\neq\) symbol means “not equal to”—the true value could be greater OR less than \(p_0\).

We don’t specify a direction in the alternative hypothesis.

When to Use Two-Sided Tests

Use two-sided when:

• No prior expectation about direction
• Both directions would be scientifically interesting
• You want to detect ANY difference from the null

Default to two-sided unless you have a strong reason for one-sided.

The Problem with One Tail

If we used a one-sided test (\(H_A: p > 0.5\)):

• P-value = area to the RIGHT of our Z-score
• Only counts evidence in ONE direction

But wait… what if people were actually WORSE than chance?

• If only 32 students got it right (40%), that would also be evidence against \(H_0\)!
• A one-sided test would miss this

Both Tails Count

For \(H_A: p \neq 0.5\), evidence against \(H_0\) comes from either direction.

If \(\hat{p} = 0.60\) (Z = 1.79) OR \(\hat{p} = 0.40\) (Z = -1.79), we’d have equal evidence against \(H_0\).

Calculating Two-Sided P-values

Method: Double the smaller tail

1. Calculate \(Z = \frac{\hat{p} - p_0}{SE}\)

2. Find area to the LEFT of Z and area to the RIGHT of Z

3. Double the SMALLER of these two areas

This method works for both simulation-based and model-based inference, and for both symmetric and asymmetric distributions.

Decision Errors: Four Scenarios

When we make a decision about \(H_0\), there are four possible outcomes:

Two ways to be right, two ways to be wrong.

Type I Error (False Positive)

Definition: Rejecting \(H_0\) when it’s actually true.

“False alarm” — claiming an effect that doesn’t exist.

In AI study context:

• Concluding people can detect AI images differently than chance…
• …when they actually can’t (detection rate really is 50%)

Consequence: Overestimating human detection abilities.

Type II Error (False Negative)

Definition: Failing to reject \(H_0\) when \(H_A\) is actually true.

“Missed detection” — failing to find a real effect.

In AI study context:

• Failing to conclude that people can detect AI images…
• …when they actually can (or can’t) at a rate different from 50%

Consequence: Missing important findings about human-AI interaction.

Significance Level Controls Type I Errors

The significance level α is the probability of making a Type I error when \(H_0\) is true.

• If we always use α = 0.05, we make Type I errors about 5% of the time when \(H_0\) is true
• Smaller α → fewer Type I errors

But there’s a trade-off…

The Trade-off

We can’t minimize both error types simultaneously.

Choose α based on which error is more costly in your context.

What is Power?

Power is the probability of correctly rejecting \(H_0\) when \(H_A\) is actually true.

\[\text{Power} = 1 - \text{Probability of Type II Error}\]

In other words: The probability of detecting a real effect when it exists.

We want high power (typically 80% or higher).

Factors Affecting Power

Three main factors increase power:

1. Larger sample size → more precision → easier to detect effects

2. Larger true effect → bigger differences are easier to spot

3. Larger α → easier to reject \(H_0\) (but more Type I errors!)

For our AI study: With only 80 students and an observed 10% difference from chance, we may not have had enough power to detect a real (but small) effect.

Which Error Matters More?

It depends on context!

Prioritize avoiding Type I errors when:

• False positives are dangerous or costly
• Example: Approving an ineffective medical treatment

Prioritize avoiding Type II errors when:

• Missing a real effect is dangerous
• Example: Failing to detect a disease in screening

Choose your significance level α accordingly.

Key Assumptions: Model vs. Reality

Model Prediction

Model Prediction Gone Wrong

CI and Two-Sided Test Connection

A \((1-\alpha)\) confidence interval and a two-sided test at level α give consistent results:

• If the CI contains \(p_0\) → Don’t reject \(H_0\) (p-value > α)
• If the CI excludes \(p_0\) → Reject \(H_0\) (p-value ≤ α)

Visualizing the Connection

Our AI study: 95% CI = (0.493, 0.707) contains 0.5 → consistent with failing to reject \(H_0\).

CIs Give More Information

The confidence interval tells us more than just the hypothesis test:

• Direction: Is the effect positive or negative?
• Magnitude: How big might the effect be?
• Plausible values: What’s the range of reasonable estimates?

Recommendation: Always report the CI, not just the p-value!

Our AI study: We’re 95% confident the true detection rate is between 49.3% and 70.7%.

Summary

References