3. Introduction to Hypothesis Testing

Putting It All Together

We have all the tools — time to apply them.

Today’s plan:

1. Worked example — Lady Tasting Tea, walking through all the steps introduced in 3a and 3b
2. What strengthens evidence — beyond the example, look at how sample size and effect size shape our conclusions

Priority isn’t just the steps (you’ve seen those) — it’s the why behind each one.

From Question to Hypothesis Test

The question — “could she be guessing and just got lucky?” — is hard to answer with intuition alone.

A hypothesis test gives us a structured way to ask:

How often would pure guessing produce a result this good or better?

• If guessing rarely produces 9+ correct → strong evidence she really can tell.
• If guessing often produces 9+ correct → her result could easily be luck.

State the Hypotheses

Let p = the lady’s true probability of correctly identifying a cup.

• H_0: p = 0.5 — “she’s just guessing”
• H_A: p > 0.5 — “she’s better than guessing”

H_0 is the skeptical default — what we assume unless the data force us otherwise.

H_A is the claim Fisher set out to test. One-sided because he only cared whether she’s better than chance, not worse.

If She’s Just Guessing…

If H_0 is true (lady is guessing), each cup is essentially a coin flip — half a chance she’s right.

But how many of 12 would she get right? Depends on luck:

• Sometimes 5 or 6 (close to half)
• Occasionally 8 or 9 (a lucky run)
• Once in a blue moon, all 12

Simulation will give us the full distribution.

Simulating Guessing

We’ve simulated with spinners and random integers. For a 50/50 outcome, any of these would work:

The most natural tool is a coin. Let’s dig into how that works.

From Coin to Null Distribution

For the lady tasting tea:

• Each flip = one cup. Heads = “correct”; tails = “incorrect.”
• Why a fair coin? If H_0 is true (she’s guessing), each cup is a 50% chance — exactly what a fair coin produces.
• 12 flips = one simulated trial of her 12-cup experiment — one alternate-universe Muriel guessing through all 12.
• \hat{p} from those 12 flips = (number of heads) / 12 = her simulated proportion correct on that trial.

Repeat many times → many simulated \hat{p} values → the null distribution.

Simulate the Null Distribution

We pretend H_0 is true and run 1000 simulated trials of 12 cups each (1000 sets of 12 coin flips), calculating \hat{p} for each.

This gives us the null distribution. Notice it clusters around \hat{p} = 0.5.

Where Does 9 of 12 Land?

Her observed \hat{p} = 9/12 = 0.75. (red dashed line). Pink bars = guessing-trials that did as well or better. Now we can calculate how rare her result would be if she is guessing.

Is 8% Surprising Enough?

Not extremely rare — but not common either.

To decide, we need a standard — we’ll use \alpha = 0.05.

At this significance level we fail to reject H_0.

The data don’t provide convincing evidence that the lady can tell the difference.

But careful — that does not mean she’s guessing. Failing to reject just means we lack strong evidence either way.

Postscript: What Actually Happened

The story goes that Muriel Bristol identified all 8 cups correctly.

Fisher’s real design tied the cups to one another in a way ours doesn’t, so his p-value is computed differently — but it comes out to about 0.014.

Below \alpha = 0.05, so that result would have been convincing — unlike the borderline 9-of-12 we just worked through.

α = 0.05 Is Just a Convention

There’s nothing magical about the 0.05 cutoff — Fisher picked it as a rough heuristic in the 1920s, and we’ve used it since.

Modern thinking moves away from a single threshold toward describing the strength of evidence as a continuum:

• Report the p-value itself — not just “significant” or “not”
• Smaller = stronger evidence against H_0; larger = weaker
• Always interpret in context: practical importance, study quality, what’s at stake

A p-value of 0.049 vs. 0.051 isn’t meaningfully different — but a strict cutoff treats them as opposites.

What Else Affects Evidence?

In 3b we saw: an observed \hat{p} further from the value H_0 predicts → smaller p-value → stronger evidence.

But there’s another factor — sample size. Going back to our 9-of-12 case (75% accuracy): what if she’d been tested on more cups instead of 12?

Same observed proportion — much stronger evidence. Why?

Why Larger n Means Stronger Evidence

Intuition: larger samples are less noisy.

With only 20 flips, 60% heads is plausible even from a fair coin — there’s a lot of random wobble in small samples.

With 500 flips, 60% from a fair coin would be extraordinary — large samples settle close to the true rate, so an extreme reading carries real signal.

Same observed proportion — very different evidence.

Two Factors That Strengthen Evidence

The strength of evidence against H_0 depends on:

1. How far the observed \hat{p} is from the value H_0 predicts — the size of the apparent effect.
2. The sample size n — more data sharpens the picture.

With the same observed \hat{p}, a larger sample produces a narrower null distribution — putting the observed value further into the tail and shrinking the p-value.

References

• Introduction to Modern Statistics (2e) textbook by Mine Cetinkaya-Rundel and Johanna Hardin
• Chapter 11, Section 16.1
• Fisher, R. A. (1935). The Design of Experiments. (Lady Tasting Tea, Chapter 2.)