Math 115: Introduction to Statistics – 7. Two-Sided Tests and Decision Errors

Where We’ve Been

We’ve built a complete hypothesis-testing toolkit:

One-sided ($H_A: p > p_0$ or $p < p_0$) — Topics 3, 6
Two-sided ($H_A: p \neq p_0$) — Topic 7a

And we decide using a cutoff $\alpha$ (usually 0.05): reject $H_0$ if p-value $< \alpha$.

Today’s question: what does that decision cost when we get it wrong — and what’s the connection between a two-sided test and a confidence interval?

Two Ways to Be Wrong

When we make a decision about $H_0$, there are four possible outcomes:

	Reject $H_0$	Fail to Reject $H_0$
$H_0$ is true	Type I Error	Correct
$H_0$ is false	Correct	Type II Error

Two ways to be right, two ways to be wrong.

We do know which column we’re in — we made the decision ourselves. We never know which row we’re in — we don’t know whether $H_0$ is actually true. The point is that both kinds of mistake are possible, and they have different consequences.

Type I Error — False Positive

Type I: rejecting $H_0$ when it’s actually true.

“False alarm” — we claimed an effect that isn’t really there.

Courtroom (Topic 3): convicting an innocent defendant — $H_0$ (“not guilty”) is true, but the jury rejected it.

AI image study (7a): concluding people can detect AI images at a rate different from chance, when actually they can’t (true rate really is 50%).

Real-world cost: treating a useless intervention as effective; publishing a finding that won’t replicate.

Type II Error — False Negative

Type II: failing to reject $H_0$ when $H_A$ is actually true.

“Missed detection” — there really is an effect, and we missed it.

Courtroom (Topic 3): letting a guilty defendant walk free — $H_A$ (“guilty”) is true, but the jury failed to reject $H_0$.

AI image study (7a): failing to detect that students can identify AI images differently from chance, when in fact they really can.

Real-world cost: ignoring a real effect; abandoning an intervention that actually works.

Identify the Error Type

For each scenario, which error has been made — Type I or Type II?

A medical screening test concludes a patient has a disease, but the patient is actually healthy. Type I. Rejecting $H_0$ (“patient is healthy”) when $H_0$ is true.
A jury finds a guilty defendant “not guilty.” Type II. Failing to reject $H_0$ (“defendant is innocent”) when $H_A$ is true.
A spam filter lets a spam email through to your inbox. Type II. Failing to flag spam when it really is spam.
A spam filter sends an important email from your boss to the spam folder. Type I. Flagging “spam” when the email isn’t spam — false alarm.

Your turn. Four scenarios — for each one, decide which kind of error has been made, Type One or Type Two. Pause and think about it before clicking through. The first scenario is a medical screening test that concludes a patient has a disease, when the patient is actually healthy. The null here is patient is healthy, and the test rejected that null when it was true — so this one is a Type One Error, the false-alarm flavor. The second scenario is a jury finding a guilty defendant not guilty. The null in court is innocent until proven guilty, so guilt makes the alternative true. The jury failed to reject when the alternative was true, which makes this a Type Two Error — the missed-detection flavor. The third scenario is a spam filter letting a spam email through. The filter’s null is this email is fine, and spam in the inbox means the alternative was true. The filter failed to flag it, so this is another Type Two Error. The fourth scenario is a spam filter sending an important email from your boss to spam. Same null — this email is fine — but this time the null really is true, and the filter rejected it anyway. That makes it a Type One Error, the false-alarm flavor again.

α Controls Type I Errors

The significance level $\alpha$ is exactly the probability of a Type I error when $H_0$ is true.

Use $\alpha = 0.05$ → about 5% false-alarm rate when $H_0$ is true
Use $\alpha = 0.01$ → about 1% false-alarm rate when $H_0$ is true

Smaller $\alpha$ → fewer Type I errors. That sounds great…

…but there’s a trade-off.

The Trade-Off

	Type I errors	Type II errors
Smaller α	↓	↑
Larger α	↑	↓

You can’t minimize both at once. Tightening the standard for “rejection” makes you less likely to reject when you shouldn’t — and also less likely to reject when you should.

→ Choose $\alpha$ based on which mistake is more costly in your context.

Power: The Other Side of Type II

Power is the probability of correctly rejecting $H_0$ when $H_A$ is actually true:

\[\text{Power} = 1 - P(\text{Type II error})\]

In words: if there really is an effect, what’s the chance our test detects it?

We want high power — typically aimed at 80% or higher.

Type II error is what we don’t want; power is the same idea phrased positively. Designing a study with low power is setting yourself up to miss a real effect.

What Boosts Power?

Three factors that increase power:

Larger sample size ($n$) → smaller SE → easier to detect effects
Larger true effect → bigger differences are easier to spot
Larger $\alpha$ → easier to reject (but more Type I errors)

Reality check: the AI image study used only 80 students. With a true detection rate near 60% — only a 10-point shift from guessing — the test had limited power to spot the effect.

Rank by Power

In each pair, which study has higher power to detect a true effect?

Sample size. Two surveys ask whether students would support a proposed $200 annual fee to fund expanded mental health services. Survey A samples $n = 50$; Survey B samples $n = 500$. Same population, same $\alpha$. Survey B ($n = 500$). Larger sample → smaller SE → easier to detect any given effect.
True effect size. Two clinical trials test whether their drug raises the recovery rate above the standard 60%. Drug A truly raises it to 65%; Drug B truly raises it to 80%. Same $n$, same $\alpha$. Drug B. Bigger true effects sit further out in the tails of the null distribution — easier to detect.
Significance level. A pollster tests whether 50% of US adults can name their state’s senator, with $n = 300$. Should they use $\alpha = 0.01$ or $\alpha = 0.10$ for higher power? $\alpha = 0.10$. Looser cutoff → easier to reject — but you pay for it with more Type I errors when $H_0$ is actually true.

Your turn. Three pairs — in each pair, which study has higher power to detect a true effect, holding everything else constant? Try all three on your own first. First pair — the first survey samples fifty students about a proposed two-hundred-dollar annual fee for expanded mental health services, the second survey samples five hundred. Larger sample shrinks SE, so the second survey has higher power. Second pair — both clinical trials test whether their drug raises recovery above the standard sixty percent. Drug A’s true effect is a five-point bump to sixty-five percent. Drug B’s is a twenty-point bump to eighty. Bigger true effects sit further out in the tails of the null distribution, so Drug B has higher power. Third pair — a pollster tests whether half of US adults can name their state’s senator, with n equal to three hundred. The choice — use alpha at zero point zero one, or alpha at zero point one zero. The looser cutoff makes it easier to reject, so alpha at zero point one zero has higher power. The trade-off — more Type One errors when the null actually holds.

Bigger n: No Error Trade-Off

The $\alpha$ knob trades errors:

Tighten $\alpha$ → fewer Type I, more Type II (lower power)
Loosen $\alpha$ → fewer Type II, more Type I

Sample size is different. Larger $n$:

Raises power directly — fewer Type II errors
Holds the Type I rate at $\alpha$ — no error trade-off

→ When you can afford it, bigger $n$ is the knob to turn first

Which Error Matters More?

It depends on context.

Prioritize avoiding Type I when:

A false alarm is costly or harmful
Example: approving an ineffective drug, jailing an innocent person

Prioritize avoiding Type II when:

Missing a real effect is costly or harmful
Example: failing to detect a disease in a screening test

→ Pick $\alpha$ to match the stakes. The default $\alpha = 0.05$ is a convention, not a law.

Choose α

For each context, would you prefer a smaller $\alpha$ (e.g., 0.01 — fewer false alarms) or a larger $\alpha$ (e.g., 0.10 — fewer missed detections)? Explain.

Approving a new chemotherapy drug for general use. Smaller $\alpha$. Approving an ineffective (or harmful) drug — a Type I error — has serious consequences for patients and the healthcare system.
An initial cancer screening test that triggers further (safe, low-cost) follow-up testing. Larger $\alpha$. A Type II error here — missing a cancer case — is far worse than a Type I error, since false positives just lead to a follow-up test.
Deciding whether to roll out a new study-skills program campus-wide based on a pilot study. Reasonable people differ. Lean smaller $\alpha$ if rollout is expensive and disruptive; lean larger $\alpha$ if the program is cheap and the cost of missing a real benefit is high. The point is to think about it, not default to 0.05.

Your turn. For each context, would you prefer a smaller alpha — fewer false alarms — or a larger alpha — fewer missed detections? And why? Pause on each one. The first scenario is approving a new chemotherapy drug for general use. The right call here is a smaller alpha — approving a drug that doesn’t actually work is a Type One Error, and that’s serious for patients and the healthcare system. We want strong evidence before we say yes. The second scenario is an initial cancer screening test that triggers safe, low-cost follow-up. Here a larger alpha makes sense — missing a real cancer case, a Type Two Error, is far worse than a false positive that just routes the patient to one more test. When the downside of a missed detection is severe and the downside of a false alarm is mild, the larger alpha is worth the trade. The third scenario is rolling out a study-skills program campus-wide based on a pilot. Reasonable people land in different places here. Lean smaller alpha if rollout is expensive and disruptive. Lean larger if the program is cheap and the cost of missing a real benefit is high. The point isn’t to land on one answer — the point is to think about which error costs more, rather than defaulting to zero point zero five.

Now: The Second Question

Slide 1 promised two threads today:

What does our decision cost when we’re wrong? ✓ (errors, α, power)
What’s the connection between a two-sided test and a confidence interval?

We’ve been hinting at thread 2 since 4b — but the rule only works for two-sided tests, so we had to wait until 7a built that machinery. Now we have it.

CI ↔︎ Two-Sided Test Connection

A confidence interval and a two-sided test at the matching level almost always give consistent results:

If the CI contains $p_0$ → fail to reject $H_0$ (p-value $\geq \alpha$)
If the CI excludes $p_0$ → reject $H_0$ (p-value $< \alpha$)

So what does matching level mean?

Matching the Level

Confidence level and $\alpha$ (as a percent) add to 100%:

Confidence level	Matched $\alpha$
90% CI	$\alpha = 0.10$
95% CI	$\alpha = 0.05$
99% CI	$\alpha = 0.01$

Use the wrong pairing — say, a 90% CI with an $\alpha = 0.05$ test — and the rule doesn’t apply.

Visualizing the Connection

Left: the Payday 95% CI from 6b — $(0.479,\; 0.547)$ — contains 0.5 → two-sided test fails to reject. Right: a hypothetical CI that excludes 0.5 → two-sided test rejects.

CI ↔︎ Decision

Each row gives a 95% CI for a proportion and a value of $p_0$ to test against (two-sided, $\alpha = 0.05$). Decide all four before clicking — then reveal one at a time.

95% CI	$p_0$	Decision
$(0.42,\; 0.58)$	$0.50$	Fail to reject — CI contains 0.50.
$(0.61,\; 0.74)$	$0.50$	Reject — CI excludes 0.50.
$(0.18,\; 0.31)$	$0.25$	Fail to reject — CI contains 0.25.
$(0.31,\; 0.39)$	$0.25$	Reject — CI excludes 0.25.

Your turn. Four rows — each gives a ninety-five percent confidence interval, a value of p naught to test against, and you decide whether the two-sided test rejects or fails to reject at alpha equal to zero point zero five. Decide all four before clicking through. First row — confidence interval from zero point four two to zero point five eight, testing p naught equal to zero point five. Zero point five is inside the interval, so fail to reject. Second row — confidence interval from zero point six one to zero point seven four, testing p naught equal to zero point five again. Zero point five is well outside the interval — so reject. Third row — confidence interval from zero point one eight to zero point three one, testing p naught equal to zero point two five. Zero point two five is inside — fail to reject. Fourth row — confidence interval from zero point three one to zero point three nine, testing p naught equal to zero point two five. Zero point two five sits below the interval — outside it — so reject.

CIs Tell You More Than P-Values

A hypothesis test gives you a yes/no (reject or not).

A confidence interval gives you:

Direction — is the effect above or below $p_0$?
Magnitude — how big could it plausibly be?
Plausible range — what values are consistent with our data?

Payday: What the CI Tells Us

95% CI $= (0.479,\; 0.547)$.

That tells us much more than “fail to reject $H_0: p = 0.5$.” Now we know:

A substantial majority (say 60%+) is not consistent with these data
The plausible range straddles 50/50 by only a few percentage points
The data don’t rule out a slim majority — or a slim minority

Summary

Two kinds of error:

Type I — false alarm (controlled by $\alpha$)
Type II — missed detection (reduced by larger $n$, larger true effect, or larger $\alpha$)
Power = $1 - P(\text{Type II})$
The two error rates trade off — pick $\alpha$ based on the stakes

CI ↔︎ two-sided test (at the matching level):

CI contains $p_0$ → fail to reject
CI excludes $p_0$ → reject

CIs give you direction, magnitude, and plausible values — almost always more useful than a p-value alone.

Wrapping up seven B with the two threads we worked through today. The first thread was decision errors. A Type One Error is a false alarm — rejecting a true null — and alpha is the dial that controls its rate. A Type Two Error is a missed detection, and it’s reduced by larger sample size, a larger true effect, or a looser alpha. Power is one minus the Type Two rate. The two error rates trade off against each other, so the choice of alpha is a judgment about which mistake costs more in context. The second thread was the link between a confidence interval and a two-sided test at the matching level. When the confidence interval contains p naught, the test fails to reject; when it excludes p naught, the test rejects. And the confidence interval gives you direction, magnitude, and plausible values — almost always more useful than the p-value alone.

End of the Single-Proportion Unit

You’ve now done everything we’ll do with a single proportion:

Hypothesis test — simulation (Topic 3) or normal model (6a)
Confidence interval — bootstrap (Topic 4) or normal model (6b)
Two-sided tests + decision errors / power (7a, 7b)
Scope of inference — when results generalize (Topic 5)

Topics 8+ reuse this same machinery — null distributions, p-values, CIs, conditions — for two proportions (8–10), means (11–13), and beyond. The mechanics will be familiar; only the settings change.

References

Introduction to Modern Statistics (2e) textbook by Mine Çetinkaya-Rundel and Johanna Hardin
Chapter 14

95% CI	\(p_0\)	Decision
\((0.42,\; 0.58)\)	\(0.50\)	Fail to reject — CI contains 0.50.
\((0.61,\; 0.74)\)	\(0.50\)	Reject — CI excludes 0.50.
\((0.18,\; 0.31)\)	\(0.25\)	Fail to reject — CI contains 0.25.
\((0.31,\; 0.39)\)	\(0.25\)	Reject — CI excludes 0.25.

	Reject \(H_0\)	Fail to Reject \(H_0\)
\(H_0\) is true	Type I Error	Correct
\(H_0\) is false	Correct	Type II Error

Confidence level	Matched \(\alpha\)
90% CI	\(\alpha = 0.10\)
95% CI	\(\alpha = 0.05\)
99% CI	\(\alpha = 0.01\)