Comparing Many Means

Vocabulary Score

• Do scores on a vocabulary test vary between self-identified social classes?
• You can try answering questions from the Wordsum test here
• Wordsum test has 10 questions, scores can range from 0-10

Inference

• Self-identified social classes: “Lower” (L), “Middle” (M), “Upper” (U), “Working” (W)
• Let \(\mu_C\) be the mean score on the Wordsum test for social class \(C\)
• We will conduct a hypothesis test with hypotheses
  • \(H_0: \mu_L=\mu_M=\mu_U=\mu_W\)
  • \(H_A:\) at least one of the means is different

Data

• gss ¹ dataset
• Sample of 795 individual responses from General Social Survey (GSS)
• wordsum is score on Wordsum test
• class is self-identified social class

EDA

A Holistic Approach to Comparing Means

• One way to approach this problem would be to make 6 pairwise comparisons (comparing each group to every other group) using two-sample t-tests
• However, if the null hypothesis is true, there is a 5% chace of making a type 1 error with each test (if \(\alpha=0.05\))
• The probability of making at least 1 type 1 error would be \(1-0.95^6=0.265\)
• Instead we take a holistic view and test whether at least one of the means is different from the others

• If there is convincing evidence that at least one of the means is different we can follow up with post-hoc pairwise tests to see which groups are different
• We will take steps to control the type 1 error given the multiple hypothesis tests
• This is a topic we will discuss in more detail later

F Statistic

• The test statistic for three or more means is an \(F\) statistic
• \(F\) is a ratio that compares variability between groups to variability within groups
• E.g., if variability in word scores between social classes is large relative the the variability within social classes, then \(F\) will be large
• Larger values of \(F\) give stronger evidence against the null hypothesis

Variability Between Groups

• \(MSG\) is the mean square between groups, a measure of variability between groups \[MSG=\frac{1}{df_{G}}SSG\]
• The degrees of freedom for \(k\) groups is \(df_{G}=k-1\)

• \(SSG\) is the sum of squares between groups \[SSG = \sum_{i=1}^kn_i(\bar{x}_i-\bar{x})^2\]
• \(\bar{x}_i\) is the mean for group \(i\)
• \(\bar{x}\) is the overall mean, which we can compute directly from the data or from the group means \[\bar{x}=\frac{n_1\bar{x}_1 +n_2\bar{x}_2+\cdots +n_k\bar{x}_k}{n_1+n_2 + \cdots + n_k}\]

• For the vocabulary test data the overall mean score is \[\begin{array}{rcl}\bar{x} &=& \frac{n_L\bar{x}_L +n_M\bar{x}_M+n_U\bar{x}_U+ n_W\bar{x}_W}{n_L+n_M + n_U + n_W}\\ &=&\frac{41\cdot5.07 + 331\cdot6.76+ 16 \cdot 6.19 + 407\cdot 5.75}{41+331+16+407}\\ &=& 6.14\end{array}\]
• The sum of squares between groups is

• The degrees of freedom are \(df_{G}=k-1=4-1=3\)
• Thus, the mean square between groups is \[MSG=\frac{1}{df_{G}}SSG=\frac{1}{3}\cdot236.56=78.85\]

Variability Within Groups

• MSE is the means square error, a measure of variability within groups \[MSE=\frac{1}{df_E}SSE\]
• The degrees of freedom for a sample of size \(n\) with \(k\) groups is \(df_E=n-k\)

• SSE is the sum of squared errors, which can be computed two ways
• The first way requires us to computed \(SSG\) first. \(SSE = SST - SSG\), where \(SST\) is the sum os squares total \[SST=\sum_{i=1}^n(x_i-\bar{x})^2\]
• The second way uses the sample variances \[SSE = (n_1-1)s_1^2+(n_2-1)s_2^2+\cdots+(n_k-1)s_k^2\]

• For the vocabulary test data the SSE is

• The degrees of freedom are \(df_{E}=n-k=795-4=791\)
• Thus, the mean square error is \[MSE=\frac{1}{df_{E}}SSE=\frac{1}{791}\cdot2869.80=3.628\]

Calculating F

• \(F\) is computed as \[F=\frac{Variability\,Between\,Groups}{Variability\,Within\,Groups}=\frac{MSG}{MSE}\]
• For the vocabulary test data \(F\) is \[F=\frac{78.85}{3.628}=21.73\]

• Usually we won’t compute the sums of squares, mean squares, or \(F\) statistic ourselves
• They are displayed in Analysis of Variance (ANOVA) tables that are printed by statistical software
• The ANOVA analysis in Jamovi displays the full ANOVA table, but the One-Way ANOVA does not

ANOVA table

Key for understanding the ANOVA table

Hypothesis Test Using Random Permutation

• We can use randomization to simulate variability in the \(F\) statistic under a true null hypothesis
• To simulate independence between word score and social class, we randomly permute the values of the response (word score)
• We can use the Randomize module in Jamovi to do this

Histogram of F scores (null distribution) for 1,000 random permutations of word scores. Dashed vertical line indicates observed F score.

• There are 0 randomized \(F\) statistics that are at least as large as the observed value (21.73)
• The p-value is approximately \(0/1000 = 0\)

Hypothesis Test Using a Mathematical Model

• Like \(X^2\), the \(F\) statistic is always non-negative
• To compute a p-value we find the area in the right tail of the appropriate \(F\)-distribution that is beyond the observed value of \(F\)

Checking Condtions

• For the word score example, the distributions are approximately normal, and the variances are roughly equal (this is particularly important because the group sizes are so different)
• The observations are independent, because the GSS uses random sampling

Calculate P-Value Using F-distribution

The \(F\)-distribution with \(df_G=3\) and \(df_E=791\) degrees of freedom. The observed \(F\)-statistic is 21.73. The p-value is the total area to the right of 21.73 (too small to see).

• We can also read p-value from the ANOVA table

Conclusions

• With \(F=21.73\) we reject the null hypothesis (p-value < 0.001). There is convincing evidence that at least one of the mean word scores is different between the self-identified social classes.
• We are unable to conclude which social classes are different based on this analysis.
• However, if we take care to control the type 1 error we can follow up with post-hoc tests to explore the pairwise differences in means
• We will consider such post-hoc analyses later