Math 115: Introduction to Statistics – 3. Introduction to Hypothesis Testing

Recap: The Medical Consultant

A medical consultant claims her liver-donor patients have a lower complication rate than the 10% national average.

Hypotheses:

\(H_0: p = 0.10\) — “her rate equals the national average”
\(H_A: p < 0.10\) — “her rate is lower”

Her data: 3 complications in 62 surgeries, so \[\hat{p} = \frac{3}{62}=0.048\]

Recap: The Null Distribution from 3a

The histogram shows 1000 simulated \(\hat{p}\) values, assuming \(H_0\) is true — the values we would see if the consultant’s rate is actually 10%. Our observed \(\hat{p} = 0.048\) sits in the left tail. Values highlighted in pink were the same as hers or even smaller.

Measuring Surprise: The P-Value

We need a number that captures how surprising the consultant’s \(\hat{p} = 0.048\) really is. That number is the p-value.

P-value = Probability of seeing our result (or one more extreme) if \(H_0\) is true

Note

“More extreme” = in the direction of \(H_A\) — here that means lower (\(\hat{p} \leq 0.048\)), the shaded tail we just saw.

Courtroom parallel: P-value = strength of prosecution’s evidence. Smaller values = stronger evidence.

Three “P” Terms — Don’t Confuse Them!

Important

\(p\) = population parameter (the true proportion)
\(\hat{p}\) = sample statistic (our calculated estimate)
p-value = probability of seeing our result (or one more extreme) if \(H_0\) is true

They all use the letter “p” but mean different things.

Computing the P-Value

The p-value is the probability of seeing \(\hat{p} \leq 0.048\) if \(H_0\) is true
Each of our 1000 simulations shows one possible outcome if \(H_0\) is true (rate = 10%) — so the fraction of simulations at or below 0.048 is a direct estimate of that probability:

\[\text{P-value} \approx \frac{111}{1000} = 0.111\]

This is the fraction of the null distribution shaded pink

What If the Data Differed?

Suppose the consultant still had 62 surgeries, but a different number of complications. Click through to see how the p-value (shaded tail) changes:

To build intuition, let’s see how the p-value would change if the consultant had observed different counts in her sixty-two surgeries. Click through each scenario. First, with just one complication. P hat drops to zero point zero one six — far below the ten percent national average. The shaded tail is tiny — p-value about one percent. Next, three complications — our actual result. P hat is zero point zero four eight and the p-value is zero point one one one. Now six complications. P hat is just under ten percent, putting the observed value near the center of the distribution. The shaded tail covers more than half the histogram — p-value about fifty-eight percent. And finally, seven complications. P hat is slightly above ten percent, and the shaded tail covers most of the distribution — p-value about seventy-three percent.

So there’s a clear pattern across the four scenarios — as the observed proportion moves away from the direction H A predicted, the shaded tail grows and the p-value gets larger. The next slide says explicitly why that’s evidence against H naught.

The Pattern

Lower \(\hat{p}\) → smaller p-value → stronger evidence against \(H_0\)

Why? Because a lower \(\hat{p}\) is further from what \(H_0\) predicts (\(p = 0.10\)). The further your data fall from \(H_0\)’s prediction, the more surprising they’d be if \(H_0\) is true — and that’s exactly what we mean by “evidence against \(H_0\).”

Interpreting the P-Value

P-value \(\approx\) 0.111 means:

“If the consultant truly has the same 10% complication rate as the national average, there is about an 11% chance of observing 3 or fewer complications in 62 surgeries.”

Is this surprising?

Not extremely rare, but not common either
We need a standard to make a decision

Interpret a P-Value

Scenario: A college claims its 4-year graduation rate is 70%. A consumer-research org follows 80 students and finds 48 graduated → \(\hat{p} = 0.60\).

They test \(H_0: p = 0.70\) vs \(H_A: p < 0.70\) and get a p-value of 0.024.

Write a sentence interpreting this p-value, in the form:

“If \(H_0\) is true (i.e., the true rate is __ ), the chance of seeing __ or fewer is about __.”

Answer: “If the college’s true 4-year graduation rate is 70%, the chance of seeing 48 or fewer graduates in a sample of 80 is about 2.4%.”

The Significance Level (α)

We have the p-value. How do we use it to decide?

In 3a we said we’d reject \(H_0\) when our data “look very unusual” — but never said how unusual. The significance level (\(\alpha\)) is that cutoff, set in advance.

Decision rule (common choice: \(\alpha = 0.05\)):

If p-value \(< \alpha\): Reject \(H_0\) (“statistically significant”)
If p-value \(\geq \alpha\): Fail to reject \(H_0\) (not statistically significant)

Courtroom parallel: \(\alpha\) = “beyond reasonable doubt”

Making a Decision

Our results: p-value \(\approx\) 0.111, \(\alpha = 0.05\)

Conclusion:

Since \(0.111 \geq 0.05\), we fail to reject \(H_0\) using \(\alpha = 0.05\) as our cutoff.

The data do not provide convincing evidence that the consultant has a lower complication rate than the national average.

Courtroom parallel: the jury returns “not guilty” — the prosecution didn’t meet the burden of proof.

What Does “Fail to Reject” Mean?

We have our verdict. But we need to be careful about what it actually means.

What we CAN say	What we CANNOT say
Data don’t provide strong evidence against \(H_0\)	\(H_0\) is true
10% is a plausible value for the consultant’s rate	Consultant’s rate is exactly 10%
We can’t rule out that the consultant is average	Consultant is definitely not better

“Fail to reject” ≠ “accept”: the consultant might have a lower rate — we just don’t have enough evidence to conclude that.

Courtroom parallel: “Not guilty” ≠ “innocent”

Spot the Mistake

A streaming service claims 60% of users binge-watch. An analyst surveys 200 users and finds 130 do (65%). Testing \(H_0: p = 0.60\) vs \(H_A: p > 0.60\) gives a p-value of 0.075 → fail to reject \(H_0\).

For each statement, decide: valid or invalid conclusion?

Statement	Valid?
There’s a 7.5% chance that \(H_0\) is true (the rate really is 60%).	Invalid
If the true rate is 60%, we’d see 65% or more in a sample of 200 about 7.5% of the time.	Valid
We’ve proven that 60% of users binge-watch.	Invalid
Our data don’t give strong evidence that more than 60% of users binge-watch.	Valid

Your turn. Consider a different scenario: a streaming service claims sixty percent of users binge-watch. An analyst surveys two hundred users and finds one hundred thirty do — about sixty-five percent. Testing H naught: p equals zero point six against H A: p greater than zero point six gives a p-value of zero point zero seven five — failing to reject H naught. Pause and decide whether each of the four statements is a valid or invalid conclusion. Advance when ready. Two of the statements are invalid — they overstate what failing to reject actually means. The first claims a “seven point five percent chance H naught is true,” but that’s a misreading — the p-value does not measure how likely H naught is. The third claims we’ve “proven” sixty percent — we never prove H naught. The other two are valid: the formal interpretation, and the careful “don’t have strong evidence” framing.

Guided Practice (GP)

Scenario: Researchers want to know if MOST payday borrowers support new regulations.

Data: In a sample of 500 borrowers:

260 support new regulations

Question: Is this evidence that MORE than half of all payday borrowers support new regulations?

We’ll work through each step together.

Set Up (GP)

Step 1: Identify the parameter

\(p\) = proportion of ALL payday borrowers who support regulations

Step 2: State the hypotheses

\(H_0: p = 0.50\) (exactly half support)
\(H_A: p > 0.50\) (more than half support)

Step 3: Calculate sample statistic

\(\hat{p} = \frac{260}{500} = 0.52\)

Simulate (GP)

Step 4: Simulate 1000 samples, assuming \(H_0\) is true.

We’ll do this step together — the null distribution was prepared in advance using Randomize in Jamovi.

Advance the slide to see it.

Null Distribution (GP)

The histogram shows 1000 simulated \(\hat{p}\) values when the true rate is 50%. Our observed \(\hat{p} = 0.52\) is the red dashed line. The shaded tail is everything at or above it (since \(H_A: p > 0.50\), “more extreme” = higher).

Estimate the P-Value (GP)

Step 5: Calculate the p-value using the null distribution.

Estimate the bar heights in the shaded tail and use your estimate to calculate the p-value. Advance to the next slide for the answer.

Counting the Tail (GP)

Sum = 58 + …

Sum = 58 + …

Sum = 58 + 49 + …

Sum = 58 + 49 + 27 + …

Sum = 58 + 49 + 27 + 19 + 11 + 11 + 4 + 2 + 1 = 182

Calculate P-Value (GP)

\[\text{P-value} = \frac{182}{1000} = 0.182\]

If the true rate really is 50%, there’s about a 18% chance of seeing \(\hat{p} \geq 0.52\) in a sample this size.

Decide and Conclude (GP)

Step 6: Make a decision about the hypothesis test (use \(\alpha = 0.05\)).

Since \(0.182 \geq 0.05\), we fail to reject \(H_0\).

Step 7: State the conclusion in context.

The data do not provide convincing evidence that more than half of payday borrowers support the new regulations.

Writing It Up

For a formal report, we’d combine all the pieces — sample details, hypotheses, p-value, decision, and contextual interpretation:

In a sample of 500 payday borrowers, 260 (52.0%) supported the new regulations. A randomization test of \(H_0: p = 0.50\) vs \(H_A: p > 0.50\) produced a p-value of 0.182. We fail to reject \(H_0\) at \(\alpha = 0.05\); the data do not provide convincing evidence that more than half of all payday borrowers support the new regulations.

The Hypothesis Testing Framework

Step	Hypothesis test	Courtroom
1.	State \(H_0\) and \(H_A\)	File charges
2.	Collect data; compute \(\hat{p}\)	Gather evidence
3.	Simulate the null distribution	Imagine an innocent defendant
4.	Calculate the p-value	Weigh the evidence
5.	Compare to \(\alpha\)	Apply the burden of proof
6.	State conclusion in context	Return the verdict

Next class: we’ll walk through a complete worked example from start to finish — Lady Tasting Tea.

References

Introduction to Modern Statistics (2e) textbook by Mine Cetinkaya-Rundel and Johanna Hardin
Chapter 11, Section 16.1