Math 115: Introduction to Statistics – 3. Introduction to Hypothesis Testing

Where We Left Off

Topic 3 so far:

3a — set up a hypothesis test for the medical-consultant data: \(H_0: p = 0.10\), \(H_A: p < 0.10\), observed \(\hat{p} = 3/62 \approx 0.048\). Simulated the null distribution.
3b — measured surprise with the p-value (\(\approx 0.111\) for the consultant), set the significance level \(\alpha = 0.05\), and used the reject / fail-to-reject decision rule to fail to reject \(H_0\).

Putting It All Together

We have all the tools — time to apply them.

Today’s plan:

Worked example — Lady Tasting Tea, walking through all the steps introduced in 3a and 3b
What strengthens evidence — beyond the example, look at how sample size and effect size shape our conclusions

Priority isn’t just the steps (you’ve seen those) — it’s the why behind each one.

Lady Tasting Tea

Rothamsted Experimental Station, England, 1920s. Over afternoon tea, Muriel Bristol claims she can tell whether the milk was poured into the cup before or after the tea.

Sitting across from her, Ronald Fisher designs an experiment on the spot: brew 8 cups — some milk-first, some tea-first — and ask her to identify which is which.

We’ll work a slightly simpler version: flip a coin for each of 12 cups to decide milk-first or tea-first.

Result: she gets 9 of 12 correct.

The question: Is she really better than guessing — or just a lucky run?

From Question to Hypothesis Test

The question — “could she be guessing and just got lucky?” — is hard to answer with intuition alone.

A hypothesis test gives us a structured way to ask:

How often would pure guessing produce a result this good or better?

If guessing rarely produces 9+ correct → strong evidence she really can tell.
If guessing often produces 9+ correct → her result could easily be luck.

State the Hypotheses

Let \(p\) = the lady’s true probability of correctly identifying a cup.

\(H_0: p = 0.5\) — “she’s just guessing”
\(H_A: p > 0.5\) — “she’s better than guessing”

\(H_0\) is the skeptical default — what we assume unless the data force us otherwise.

\(H_A\) is the claim Fisher set out to test. One-sided because he only cared whether she’s better than chance, not worse.

If She’s Just Guessing…

If \(H_0\) is true (lady is guessing), each cup is essentially a coin flip — half a chance she’s right.

But how many of 12 would she get right? Depends on luck:

Sometimes 5 or 6 (close to half)
Occasionally 8 or 9 (a lucky run)
Once in a blue moon, all 12

Simulation will give us the full distribution.

Simulating Guessing

We’ve simulated with spinners and random integers. For a 50/50 outcome, any of these would work:

50/50 spinner

fair coin

random 1 or 2

The most natural tool is a coin. Let’s dig into how that works.

From Coin to Null Distribution

For the lady tasting tea:

Each flip = one cup. Heads = “correct”; tails = “incorrect.”
Why a fair coin? If \(H_0\) is true (she’s guessing), each cup is a 50% chance — exactly what a fair coin produces.
12 flips = one simulated trial of her 12-cup experiment — one alternate-universe Muriel guessing through all 12.
\(\hat{p}\) from those 12 flips = (number of heads) / 12 = her simulated proportion correct on that trial.

Repeat many times → many simulated \(\hat{p}\) values → the null distribution.

This is the conceptual heart of the simulation, so it’s worth slowing down. Start with the mapping: each coin flip stands in for one cup, with heads meaning she got it right and tails meaning she got it wrong. Why a fair coin specifically? Because under H naught her chance of being right on each cup is fifty percent, which is exactly what a fair coin produces — so the coin is H naught, mechanically. Twelve flips together then represent one full run of her twelve-cup experiment — one alternate-universe Muriel guessing her way through all twelve cups. From those twelve flips we compute p hat the same way she would: the number of heads divided by twelve. That’s one simulated p hat — what her result could have looked like if H naught were true. Repeat many times, and the collection of simulated p hat values is the null distribution.

Simulate the Null Distribution

We pretend \(H_0\) is true and run 1000 simulated trials of 12 cups each (1000 sets of 12 coin flips), calculating \(\hat{p}\) for each.

This gives us the null distribution. Notice it clusters around \(\hat{p} = 0.5\).

Where Does 9 of 12 Land?

Her observed \(\hat{p} = 9/12 = 0.75\). (red dashed line). Pink bars = guessing-trials that did as well or better. Now we can calculate how rare her result would be if she is guessing.

Computing the P-Value

The p-value is the fraction of simulations at or above her observed \(\hat{p} = 0.75\) — a direct measure of how rare.

From our 1000 simulations: 76 of 1000 were that high.

\[\text{P-value} \approx \frac{76}{1000} = 0.076\]

In words: if she’s truly guessing, she’d get 9 or more right about 8% of the time.

Is 8% Surprising Enough?

Not extremely rare — but not common either.

To decide, we need a standard — we’ll use \(\alpha = 0.05\).

At this significance level we fail to reject \(H_0\).

The data don’t provide convincing evidence that the lady can tell the difference.

But careful — that does not mean she’s guessing. Failing to reject just means we lack strong evidence either way.

Postscript: What Actually Happened

The story goes that Muriel Bristol identified all 8 cups correctly.

Fisher’s real design tied the cups to one another in a way ours doesn’t, so his p-value is computed differently — but it comes out to about 0.014.

Below \(\alpha = 0.05\), so that result would have been convincing — unlike the borderline 9-of-12 we just worked through.

α = 0.05 Is Just a Convention

There’s nothing magical about the 0.05 cutoff — Fisher picked it as a rough heuristic in the 1920s, and we’ve used it since.

Modern thinking moves away from a single threshold toward describing the strength of evidence as a continuum:

Report the p-value itself — not just “significant” or “not”
Smaller = stronger evidence against \(H_0\); larger = weaker
Always interpret in context: practical importance, study quality, what’s at stake

A p-value of 0.049 vs. 0.051 isn’t meaningfully different — but a strict cutoff treats them as opposites.

What Else Affects Evidence?

In 3b we saw: an observed \(\hat{p}\) further from the value \(H_0\) predicts → smaller p-value → stronger evidence.

But there’s another factor — sample size. Going back to our 9-of-12 case (75% accuracy): what if she’d been tested on more cups instead of 12?

\(n\)	\(\hat{p}\)	p-value
12	0.75	~0.076 (borderline)
60	0.75	~0.0001 (decisive)

Same observed proportion — much stronger evidence. Why?

Same p̂, Different n

Suppose the observed \(\hat{p} = 0.60\), testing \(H_0: p = 0.50\) vs \(H_A: p > 0.50\). Click through to see the null distribution change as \(n\) grows.

Suppose the lady got sixty percent of her cups right — a smaller effect than her actual seventy-five percent, but the smaller gap will make the sample-size effect easier to see across the animation. So our observed p hat is now zero point six, still testing whether p is greater than zero point five. We’ll watch the null distribution change as the sample size grows. One thing to keep straight: we’re still running one thousand simulated trials at each sample size. The only thing changing is how many cups are in each trial — n. With n equals twenty cups, the distribution is wide — zero point six is well within the bulk of guessing trials, so the p-value is large. Now n equals fifty. The distribution narrows, and zero point six moves a little further into the tail. At n equals one hundred, the distribution is tighter still, and the p-value shrinks again. And at n equals five hundred, the distribution is very narrow and zero point six is far out in the tail — the p-value is essentially zero. Same observed proportion, but a vastly different verdict.

Why Larger n Means Stronger Evidence

Intuition: larger samples are less noisy.

With only 20 flips, 60% heads is plausible even from a fair coin — there’s a lot of random wobble in small samples.

With 500 flips, 60% from a fair coin would be extraordinary — large samples settle close to the true rate, so an extreme reading carries real signal.

Same observed proportion — very different evidence.

Two Factors That Strengthen Evidence

The strength of evidence against \(H_0\) depends on:

How far the observed \(\hat{p}\) is from the value \(H_0\) predicts — the size of the apparent effect.
The sample size \(n\) — more data sharpens the picture.

With the same observed \(\hat{p}\), a larger sample produces a narrower null distribution — putting the observed value further into the tail and shrinking the p-value.

Which Result Is Stronger Evidence?

Three pairs of researchers, each testing \(H_0: p = 0.5\) vs \(H_A: p > 0.5\) on a possibly-biased coin. For each pair, decide whose result is stronger evidence — then advance to check.

Pair 1. Avery (\(n = 20\), \(\hat{p} = 0.65\)) vs. Cam (\(n = 100\), \(\hat{p} = 0.65\))

Cam. Same effect, larger \(n\) — narrower null distribution puts \(\hat{p} = 0.65\) further into the tail.

Pair 2. Dana (\(n = 100\), \(\hat{p} = 0.55\)) vs. Ellis (\(n = 100\), \(\hat{p} = 0.65\))

Ellis. Same \(n\), larger effect — \(\hat{p} = 0.65\) is further from \(0.5\) than \(\hat{p} = 0.55\).

Pair 3. Avery (\(n = 20\), \(\hat{p} = 0.65\)) vs. Blake (\(n = 200\), \(\hat{p} = 0.55\))

Not obvious! Avery’s effect is bigger but \(n\) is small; Blake’s effect is smaller but \(n\) is large. The two factors push opposite ways — we need to see the null distributions to decide. (Next slide.)

Time to put both factors to work. Three pairs of researchers are testing whether a coin is biased toward heads — null hypothesis says p equals zero point five, alternative says p is greater. For each pair, decide whose result is stronger evidence, and why. Advance the slide when you are ready to check. Pair one: Avery and Cam both observed sixty-five percent heads — the same p hat. The difference is sample size. Stronger evidence is Cam’s, because a larger n means a narrower null distribution — sixty-five percent sits further into the tail when n is one hundred than when n is twenty. Pair two: Dana and Ellis both ran one hundred flips, but Ellis got sixty-five percent heads to Dana’s fifty-five percent. Stronger evidence is Ellis’s, because at the same sample size a larger effect lands further from zero point five and further into the tail. Pair three is the interesting one — Avery has the bigger effect, sixty-five percent versus the null’s fifty percent, but only twenty flips. Blake has a smaller effect, fifty-five percent, but two hundred flips. The two factors push in opposite directions, and intuition alone won’t settle it. We need to see the null distributions side by side to decide — that’s the next slide.

Pair 3 Resolved: Let the Distributions Decide

Avery (\(n = 20\), \(\hat{p} = 0.65\))

Blake (\(n = 200\), \(\hat{p} = 0.55\))

Blake has stronger evidence — even with the smaller effect. The larger \(n\) narrowed his null distribution enough that \(\hat{p} = 0.55\) reaches further into the tail than Avery’s \(\hat{p} = 0.65\).

For pair three, intuition can’t settle it — we need to actually compute. Here are the two null distributions side by side, both assuming H naught is true and using one thousand simulated trials. Avery’s distribution on the left uses n equals twenty, and her observed p hat of zero point six five gives a p-value of about thirteen percent. Blake’s distribution on the right uses n equals two hundred — much narrower — and his observed p hat of zero point five five gives a p-value of about ten percent. So Blake wins: even with the smaller effect, the larger sample size narrowed his null distribution enough that zero point five five reaches further into the tail than Avery’s zero point six five. The takeaway is that effect size and sample size both matter, and either one can outweigh the other depending on the values involved.

References

Introduction to Modern Statistics (2e) textbook by Mine Cetinkaya-Rundel and Johanna Hardin
Chapter 11, Section 16.1
Fisher, R. A. (1935). The Design of Experiments. (Lady Tasting Tea, Chapter 2.)