7. Two-Sided Tests and Decision Errors

b. Decision Errors

Yurk

Where We’ve Been

We’ve built a complete hypothesis-testing toolkit:

  • One-sided (H_A: p > p_0 or p < p_0) — Topics 3, 6
  • Two-sided (H_A: p \neq p_0) — Topic 7a

And we decide using a cutoff \alpha (usually 0.05): reject H_0 if p-value < \alpha.

Today’s question: what does that decision cost when we get it wrong — and what’s the connection between a two-sided test and a confidence interval?

Two Ways to Be Wrong

When we make a decision about H_0, there are four possible outcomes:

Reject H_0 Fail to Reject H_0
H_0 is true Type I Error Correct
H_0 is false Correct Type II Error

Two ways to be right, two ways to be wrong.

We do know which column we’re in — we made the decision ourselves. We never know which row we’re in — we don’t know whether H_0 is actually true. The point is that both kinds of mistake are possible, and they have different consequences.

Type I Error — False Positive

Type I: rejecting H_0 when it’s actually true.

“False alarm” — we claimed an effect that isn’t really there.

Courtroom (Topic 3): convicting an innocent defendant — H_0 (“not guilty”) is true, but the jury rejected it.

AI image study (7a): concluding people can detect AI images at a rate different from chance, when actually they can’t (true rate really is 50%).

Real-world cost: treating a useless intervention as effective; publishing a finding that won’t replicate.

Type II Error — False Negative

Type II: failing to reject H_0 when H_A is actually true.

“Missed detection” — there really is an effect, and we missed it.

Courtroom (Topic 3): letting a guilty defendant walk free — H_A (“guilty”) is true, but the jury failed to reject H_0.

AI image study (7a): failing to detect that students can identify AI images differently from chance, when in fact they really can.

Real-world cost: ignoring a real effect; abandoning an intervention that actually works.

Identify the Error Type

For each scenario, which error has been made — Type I or Type II?

  1. A medical screening test concludes a patient has a disease, but the patient is actually healthy. Type I. Rejecting H_0 (“patient is healthy”) when H_0 is true.

  2. A jury finds a guilty defendant “not guilty.” Type II. Failing to reject H_0 (“defendant is innocent”) when H_A is true.

  3. A spam filter lets a spam email through to your inbox. Type II. Failing to flag spam when it really is spam.

  4. A spam filter sends an important email from your boss to the spam folder. Type I. Flagging “spam” when the email isn’t spam — false alarm.

α Controls Type I Errors

The significance level \alpha is exactly the probability of a Type I error when H_0 is true.

  • Use \alpha = 0.05 → about 5% false-alarm rate when H_0 is true
  • Use \alpha = 0.01 → about 1% false-alarm rate when H_0 is true

Smaller \alpha → fewer Type I errors. That sounds great…

…but there’s a trade-off.

The Trade-Off

Type I errors Type II errors
Smaller α ↓ ↑
Larger α ↑ ↓

You can’t minimize both at once. Tightening the standard for “rejection” makes you less likely to reject when you shouldn’t — and also less likely to reject when you should.

→ Choose \alpha based on which mistake is more costly in your context.

Power: The Other Side of Type II

Power is the probability of correctly rejecting H_0 when H_A is actually true:

\text{Power} = 1 - P(\text{Type II error})

In words: if there really is an effect, what’s the chance our test detects it?

We want high power — typically aimed at 80% or higher.

Type II error is what we don’t want; power is the same idea phrased positively. Designing a study with low power is setting yourself up to miss a real effect.

What Boosts Power?

Three factors that increase power:

  1. Larger sample size (n) → smaller SE → easier to detect effects
  2. Larger true effect → bigger differences are easier to spot
  3. Larger \alpha → easier to reject (but more Type I errors)

Reality check: the AI image study used only 80 students. With a true detection rate near 60% — only a 10-point shift from guessing — the test had limited power to spot the effect.

Rank by Power

In each pair, which study has higher power to detect a true effect?

  1. Sample size. Two surveys ask whether students would support a proposed $200 annual fee to fund expanded mental health services. Survey A samples n = 50; Survey B samples n = 500. Same population, same \alpha. Survey B (n = 500). Larger sample → smaller SE → easier to detect any given effect.

  2. True effect size. Two clinical trials test whether their drug raises the recovery rate above the standard 60%. Drug A truly raises it to 65%; Drug B truly raises it to 80%. Same n, same \alpha. Drug B. Bigger true effects sit further out in the tails of the null distribution — easier to detect.

  3. Significance level. A pollster tests whether 50% of US adults can name their state’s senator, with n = 300. Should they use \alpha = 0.01 or \alpha = 0.10 for higher power? \alpha = 0.10. Looser cutoff → easier to reject — but you pay for it with more Type I errors when H_0 is actually true.

Bigger n: No Error Trade-Off

The \alpha knob trades errors:

  • Tighten \alpha → fewer Type I, more Type II (lower power)
  • Loosen \alpha → fewer Type II, more Type I

Sample size is different. Larger n:

  • Raises power directly — fewer Type II errors
  • Holds the Type I rate at \alpha — no error trade-off

→ When you can afford it, bigger n is the knob to turn first

Which Error Matters More?

It depends on context.

Prioritize avoiding Type I when:

  • A false alarm is costly or harmful
  • Example: approving an ineffective drug, jailing an innocent person

Prioritize avoiding Type II when:

  • Missing a real effect is costly or harmful
  • Example: failing to detect a disease in a screening test

→ Pick \alpha to match the stakes. The default \alpha = 0.05 is a convention, not a law.

Choose α

For each context, would you prefer a smaller \alpha (e.g., 0.01 — fewer false alarms) or a larger \alpha (e.g., 0.10 — fewer missed detections)? Explain.

  1. Approving a new chemotherapy drug for general use. Smaller \alpha. Approving an ineffective (or harmful) drug — a Type I error — has serious consequences for patients and the healthcare system.

  2. An initial cancer screening test that triggers further (safe, low-cost) follow-up testing. Larger \alpha. A Type II error here — missing a cancer case — is far worse than a Type I error, since false positives just lead to a follow-up test.

  3. Deciding whether to roll out a new study-skills program campus-wide based on a pilot study. Reasonable people differ. Lean smaller \alpha if rollout is expensive and disruptive; lean larger \alpha if the program is cheap and the cost of missing a real benefit is high. The point is to think about it, not default to 0.05.

Now: The Second Question

Slide 1 promised two threads today:

  1. What does our decision cost when we’re wrong? ✓ (errors, α, power)
  2. What’s the connection between a two-sided test and a confidence interval?

We’ve been hinting at thread 2 since 4b — but the rule only works for two-sided tests, so we had to wait until 7a built that machinery. Now we have it.

CI ↔︎ Two-Sided Test Connection

A confidence interval and a two-sided test at the matching level almost always give consistent results:

  • If the CI contains p_0 → fail to reject H_0 (p-value \geq \alpha)
  • If the CI excludes p_0 → reject H_0 (p-value < \alpha)

So what does matching level mean?

Matching the Level

Confidence level and \alpha (as a percent) add to 100%:

Confidence level Matched \alpha
90% CI \alpha = 0.10
95% CI \alpha = 0.05
99% CI \alpha = 0.01

Use the wrong pairing — say, a 90% CI with an \alpha = 0.05 test — and the rule doesn’t apply.

Visualizing the Connection

Left: the Payday 95% CI from 6b — (0.479,\; 0.547) — contains 0.5 → two-sided test fails to reject. Right: a hypothetical CI that excludes 0.5 → two-sided test rejects.

CI ↔︎ Decision

Each row gives a 95% CI for a proportion and a value of p_0 to test against (two-sided, \alpha = 0.05). Decide all four before clicking — then reveal one at a time.

95% CI p_0 Decision
(0.42,\; 0.58) 0.50 Fail to reject — CI contains 0.50.
(0.61,\; 0.74) 0.50 Reject — CI excludes 0.50.
(0.18,\; 0.31) 0.25 Fail to reject — CI contains 0.25.
(0.31,\; 0.39) 0.25 Reject — CI excludes 0.25.

CIs Tell You More Than P-Values

A hypothesis test gives you a yes/no (reject or not).

A confidence interval gives you:

  • Direction — is the effect above or below p_0?
  • Magnitude — how big could it plausibly be?
  • Plausible range — what values are consistent with our data?

Payday: What the CI Tells Us

95% CI = (0.479,\; 0.547).

That tells us much more than “fail to reject H_0: p = 0.5.” Now we know:

  • A substantial majority (say 60%+) is not consistent with these data
  • The plausible range straddles 50/50 by only a few percentage points
  • The data don’t rule out a slim majority — or a slim minority

Summary

Two kinds of error:

  • Type I — false alarm (controlled by \alpha)
  • Type II — missed detection (reduced by larger n, larger true effect, or larger \alpha)
  • Power = 1 - P(\text{Type II})
  • The two error rates trade off — pick \alpha based on the stakes

CI ↔︎ two-sided test (at the matching level):

  • CI contains p_0 → fail to reject
  • CI excludes p_0 → reject

CIs give you direction, magnitude, and plausible values — almost always more useful than a p-value alone.

End of the Single-Proportion Unit

You’ve now done everything we’ll do with a single proportion:

  • Hypothesis test — simulation (Topic 3) or normal model (6a)
  • Confidence interval — bootstrap (Topic 4) or normal model (6b)
  • Two-sided tests + decision errors / power (7a, 7b)
  • Scope of inference — when results generalize (Topic 5)

Topics 8+ reuse this same machinery — null distributions, p-values, CIs, conditions — for two proportions (8–10), means (11–13), and beyond. The mechanics will be familiar; only the settings change.

References

Course Home