3. Introduction to Hypothesis Testing

b. P-Values and Decisions

Yurk

Recap: The Medical Consultant

A medical consultant claims her liver-donor patients have a lower complication rate than the 10% national average.

Hypotheses:

  • H_0: p = 0.10 — “her rate equals the national average”
  • H_A: p < 0.10 — “her rate is lower”

Her data: 3 complications in 62 surgeries, so \hat{p} = \frac{3}{62}=0.048

Recap: The Null Distribution from 3a

The histogram shows 1000 simulated \hat{p} values, assuming H_0 is true — the values we would see if the consultant’s rate is actually 10%. Our observed \hat{p} = 0.048 sits in the left tail. Values highlighted in pink were the same as hers or even smaller.

Measuring Surprise: The P-Value

We need a number that captures how surprising the consultant’s \hat{p} = 0.048 really is. That number is the p-value.

P-value = Probability of seeing our result (or one more extreme) if H_0 is true

Note

“More extreme” = in the direction of H_A — here that means lower (\hat{p} \leq 0.048), the shaded tail we just saw.

Courtroom parallel: P-value = strength of prosecution’s evidence. Smaller values = stronger evidence.

Three “P” Terms — Don’t Confuse Them!

Important

  • p = population parameter (the true proportion)
  • \hat{p} = sample statistic (our calculated estimate)
  • p-value = probability of seeing our result (or one more extreme) if H_0 is true

They all use the letter “p” but mean different things.

Computing the P-Value

  • The p-value is the probability of seeing \hat{p} \leq 0.048 if H_0 is true

  • Each of our 1000 simulations shows one possible outcome if H_0 is true (rate = 10%) — so the fraction of simulations at or below 0.048 is a direct estimate of that probability:

\text{P-value} \approx \frac{111}{1000} = 0.111

  • This is the fraction of the null distribution shaded pink

What If the Data Differed?

Suppose the consultant still had 62 surgeries, but a different number of complications. Click through to see how the p-value (shaded tail) changes:

The Pattern

Lower \hat{p} → smaller p-value → stronger evidence against H_0

Why? Because a lower \hat{p} is further from what H_0 predicts (p = 0.10). The further your data fall from H_0’s prediction, the more surprising they’d be if H_0 is true — and that’s exactly what we mean by “evidence against H_0.”

Interpreting the P-Value

P-value \approx 0.111 means:

“If the consultant truly has the same 10% complication rate as the national average, there is about an 11% chance of observing 3 or fewer complications in 62 surgeries.”

Is this surprising?

  • Not extremely rare, but not common either
  • We need a standard to make a decision

Interpret a P-Value

Scenario: A college claims its 4-year graduation rate is 70%. A consumer-research org follows 80 students and finds 48 graduated → \hat{p} = 0.60.

They test H_0: p = 0.70 vs H_A: p < 0.70 and get a p-value of 0.024.

Write a sentence interpreting this p-value, in the form:

“If H_0 is true (i.e., the true rate is __ ), the chance of seeing __ or fewer is about __.”

Answer: “If the college’s true 4-year graduation rate is 70%, the chance of seeing 48 or fewer graduates in a sample of 80 is about 2.4%.”

The Significance Level (α)

We have the p-value. How do we use it to decide?

In 3a we said we’d reject H_0 when our data “look very unusual” — but never said how unusual. The significance level (\alpha) is that cutoff, set in advance.

Decision rule (common choice: \alpha = 0.05):

  • If p-value < \alpha: Reject H_0 (“statistically significant”)
  • If p-value \geq \alpha: Fail to reject H_0 (not statistically significant)

Courtroom parallel: \alpha = “beyond reasonable doubt”

Making a Decision

Our results: p-value \approx 0.111, \alpha = 0.05

Conclusion:

Since 0.111 \geq 0.05, we fail to reject H_0 using \alpha = 0.05 as our cutoff.

The data do not provide convincing evidence that the consultant has a lower complication rate than the national average.

Courtroom parallel: the jury returns “not guilty” — the prosecution didn’t meet the burden of proof.

What Does “Fail to Reject” Mean?

We have our verdict. But we need to be careful about what it actually means.

What we CAN say What we CANNOT say
Data don’t provide strong evidence against H_0 H_0 is true
10% is a plausible value for the consultant’s rate Consultant’s rate is exactly 10%
We can’t rule out that the consultant is average Consultant is definitely not better

“Fail to reject” ≠ “accept”: the consultant might have a lower rate — we just don’t have enough evidence to conclude that.

Courtroom parallel: “Not guilty” ≠ “innocent”

Spot the Mistake

A streaming service claims 60% of users binge-watch. An analyst surveys 200 users and finds 130 do (65%). Testing H_0: p = 0.60 vs H_A: p > 0.60 gives a p-value of 0.075 → fail to reject H_0.

For each statement, decide: valid or invalid conclusion?

Statement Valid?
There’s a 7.5% chance that H_0 is true (the rate really is 60%). Invalid
If the true rate is 60%, we’d see 65% or more in a sample of 200 about 7.5% of the time. Valid
We’ve proven that 60% of users binge-watch. Invalid
Our data don’t give strong evidence that more than 60% of users binge-watch. Valid

Guided Practice (GP)

Scenario: Researchers want to know if MOST payday borrowers support new regulations.

Data: In a sample of 500 borrowers:

  • 260 support new regulations

Question: Is this evidence that MORE than half of all payday borrowers support new regulations?

We’ll work through each step together.

Set Up (GP)

Step 1: Identify the parameter

  • p = proportion of ALL payday borrowers who support regulations

Step 2: State the hypotheses

  • H_0: p = 0.50 (exactly half support)
  • H_A: p > 0.50 (more than half support)

Step 3: Calculate sample statistic

  • \hat{p} = \frac{260}{500} = 0.52

Simulate (GP)

Step 4: Simulate 1000 samples, assuming H_0 is true.

We’ll do this step together — the null distribution was prepared in advance using Randomize in Jamovi.

Advance the slide to see it.

Null Distribution (GP)

The histogram shows 1000 simulated \hat{p} values when the true rate is 50%. Our observed \hat{p} = 0.52 is the red dashed line. The shaded tail is everything at or above it (since H_A: p > 0.50, “more extreme” = higher).

Estimate the P-Value (GP)

Step 5: Calculate the p-value using the null distribution.

Estimate the bar heights in the shaded tail and use your estimate to calculate the p-value. Advance to the next slide for the answer.

Counting the Tail (GP)

Sum = 58 + …

Sum = 58 + …

Sum = 58 + 49 + …

Sum = 58 + 49 + 27 + …

Sum = 58 + 49 + 27 + 19 + 11 + 11 + 4 + 2 + 1 = 182

Calculate P-Value (GP)

\text{P-value} = \frac{182}{1000} = 0.182

If the true rate really is 50%, there’s about a 18% chance of seeing \hat{p} \geq 0.52 in a sample this size.

Decide and Conclude (GP)

Step 6: Make a decision about the hypothesis test (use \alpha = 0.05).

Since 0.182 \geq 0.05, we fail to reject H_0.

Step 7: State the conclusion in context.

The data do not provide convincing evidence that more than half of payday borrowers support the new regulations.

Writing It Up

For a formal report, we’d combine all the pieces — sample details, hypotheses, p-value, decision, and contextual interpretation:

In a sample of 500 payday borrowers, 260 (52.0%) supported the new regulations. A randomization test of H_0: p = 0.50 vs H_A: p > 0.50 produced a p-value of 0.182. We fail to reject H_0 at \alpha = 0.05; the data do not provide convincing evidence that more than half of all payday borrowers support the new regulations.

The Hypothesis Testing Framework

Step Hypothesis test Courtroom
1. State H_0 and H_A File charges
2. Collect data; compute \hat{p} Gather evidence
3. Simulate the null distribution Imagine an innocent defendant
4. Calculate the p-value Weigh the evidence
5. Compare to \alpha Apply the burden of proof
6. State conclusion in context Return the verdict

Next class: we’ll walk through a complete worked example from start to finish — Lady Tasting Tea.

References

Course Home