15. Inference for Multiple Means

Math 115

Yurk and Martin

Extending Our Framework

Previously, we compared two groups using the t-test.

But what if we have 3 or more groups?

  • Compare test scores across 4 teaching methods?
  • Compare salaries across 5 job categories?
  • Compare vocabulary scores across 4 social classes?

We need a new approach!

Why Not Multiple T-Tests?

With 4 groups, we could do pairwise t-tests:

  • Groups 1 vs 2, 1 vs 3, 1 vs 4, 2 vs 3, 2 vs 4, 3 vs 4
  • That’s 6 separate tests!

The problem: Each test has a 5% chance of Type I error (if α = 0.05)

Overall probability of at least one Type I error:

\[1 - 0.95^6 = 0.265 = 26.5\%\]

This inflated error rate is unacceptable!

The Solution: One Overall Test

Instead of many pairwise tests, do one holistic test:

Hypotheses:

  • \(H_0: \mu_1 = \mu_2 = \mu_3 = \cdots = \mu_k\) (all means are equal)
  • \(H_A:\) At least one mean is different

If we reject \(H_0\), THEN we can explore which groups differ (with appropriate adjustments).

This approach is called Analysis of Variance (ANOVA).

Vocabulary Scores

Research question: Do vocabulary test scores differ by self-identified social class?

  • Groups: Lower, Middle, Upper, Working (k = 4)
  • Response: Score on 10-question vocabulary test
  • Data: General Social Survey (n = 795)

EDA: Vocabulary Scores

Class n Mean SD
Lower 41 5.07 2.24
Middle 331 6.76 1.89
Upper 16 6.19 2.34
Working 407 5.75 1.87

The Key Idea: Between vs. Within

Two sources of variability in the data:

Between groups: How different are the group means from each other?

Within groups: How much do individuals vary within each group?

If \(H_0\) is true (all means equal), between-group variability should be small relative to within-group variability.

Variability Between vs. Within (Sums of Squares)

Between groups: How far are group means from the overall mean?

\[SSG = \sum_{i=1}^{k} n_i(\bar{x}_i - \bar{x})^2\]

Within groups: How far are individual values from their group means?

\[SSE = \sum_{i=1}^{k} \sum_{j=1}^{n_i} (x_{ij} - \bar{x}_i)^2\]

Mean Squares (MSG and MSE)

To compare variability, we convert sums of squares to mean squares by dividing by degrees of freedom:

Between Groups:

\[MSG = \frac{SSG}{df_G}\]

\[df_G = k - 1\]

(number of groups minus 1)

Within Groups:

\[MSE = \frac{SSE}{df_E}\]

\[df_E = n - k\]

(total observations minus number of groups)

Why divide by df? This accounts for how many groups and observations we have, making the variability measures comparable.

The F Statistic

\[F = \frac{\text{Variability Between Groups}}{\text{Variability Within Groups}} = \frac{MSG}{MSE}\]

  • MSG = Mean Square between Groups
  • MSE = Mean Square Error (within groups)

Interpretation:

  • If \(H_0\) is true: F should be close to 0 (similar variability)
  • Large F → group means are more spread out than expected → evidence against \(H_0\)

Computing F

For the vocabulary example:

Between Groups Within Groups
Degrees of freedom \(df_G = 4 - 1 = 3\) \(df_E = 795 - 4 = 791\)
Sum of Squares \(SSG = 236.56\) \(SSE = 2869.8\)
Mean Square \(MSG = \frac{236.56}{3} = 78.85\) \(MSE = \frac{2869.8}{791} = 3.63\)

F statistic: \(F = \frac{MSG}{MSE} = \frac{78.85}{3.63} = 21.73\)

This F is much larger than 0!

The ANOVA Table

Jamovi reports the full ANOVA table:

Sum of Squares df Mean Square F p
class 236.56 3 78.85 21.73 < 0.001
Residuals 2869.80 791 3.63
  • F = 21.73 (test statistic)

The Null Distribution

When conditions are met, the F-statistic follows an F-distribution if \(H_0\) is true:

  • F-distribution depends on degrees of freedom: \(df_1 = k - 1\) and \(df_2 = n - k\)
  • This distribution describes what \(F\) values we’d expect if all group means are equal

Computing the p-value:

  • P-value = area to the right of observed \(F\) under the F-distribution

Conditions for ANOVA

Independence:

  • Random sample ✓
  • Independent observations within and between groups ✓

Normality:

  • Reasonable sample sizes in each group ✓
  • No extreme outliers visible ✓

Equal variance:

  • SDs are similar across groups (1.87 to 2.34) ✓

Conditions are met for using the F-distribution.

The F-Distribution

  • F-distribution has two df parameters: \(df_1 = k-1\), \(df_2 = n-k\)
  • P-value is always the right tail (area beyond observed F)

Conclusion

Results:

  • F = 21.73
  • df₁ = 3, df₂ = 791
  • P-value < 0.001

Decision: Reject \(H_0\)

Conclusion: There is convincing evidence that mean vocabulary scores differ across self-identified social classes. At least one group has a different mean.

Randomization Test for Independence

  • We can randomly permute the response (vocabulary score) between the four explanatory groups (social class)
    • To simulate the null hypothesis being true
  • For each permuted sample, we calculate value of the F statistic
  • In this course we won’t cover randomization for multiple means using ANOVA
    • But you should know that you can use randomization for these problems

Randomized Null Distribution

  • Here is a null distribution and F-statistic for the vocabulary score based on social class

What ANOVA Does and Doesn’t Tell Us

ANOVA tells us:

  • At least one group mean is different from the others

ANOVA does NOT tell us:

  • WHICH specific groups differ
  • The direction or magnitude of difference

Summary

ANOVA compares means across 3+ groups:

Component Formula/Value
F statistic \(F = \frac{MSG}{MSE}\)
Degrees of freedom \(df_1 = k - 1\), \(df_2 = n - k\)
P-value Right tail of F-distribution

Intuition: Large F means between-group variability is large relative to within-group variability → evidence that means differ

Conditions: Independence + Normality + Equal variance

Connection to Independent Samples T-Test

Two Means Multiple Means
Groups 2 3 or more
Test statistic T F
Null distribution t-distribution F-distribution
Hypothesis \(\mu_1 = \mu_2\) \(\mu_1 = \mu_2 = \cdots = \mu_k\)
Alternative \(\mu_1 \neq \mu_2\) At least one differs

For k = 2, ANOVA and two-sample t-test give equivalent results!

(In fact, \(F = T^2\) when comparing two groups)

References