Post-Hoc Tests, Multiple Comparisons

Addition to Chapter 22
Math 115

Wordsum Score

• Do Wordsum test scores vary between self-identified social classes?
• Data set available here
• Self-identified social classes: “Lower” (L), “Middle” (M), “Upper” (U), “Working” (W)
• Let \(\mu_C\) = mean score for social class \(C\)
• Hypothesis test
  • \(H_0: \mu_L=\mu_M=\mu_U=\mu_W\)
  • \(H_A:\) at least one of the means is different

ANOVA

• We conducted a hypothesis test based on a randomized null distribution of \(F\) statistics
• And a test (ANOVA) using a model (\(F\)-distribution)
• We found convincing evidence (\(F_{3,791}=21.73\), p-value < 0.001) that at least one mean score is different

Follow-Up Tests

• So far we have taken a holistic view, considering all of the groups at the same time to determine if at least one of the means is different
• If we conclude that there is convincing evidence that there is a difference, we can make pairwise group comparisons
• If there are \(k\) groups, then there are \(m=k\cdot(k-1)/2\) pairwise comparisons
• In the Wordsum example there are 4 groups, resulting in \(m = (4\cdot 3)/2=6\) possible pairwise comparisons

Multiple comparisons

We could perform 6 separate hypothesis tests (e.g., \(t\)-tests):

• \(H_0:\mu_L-\mu_M=0\), \(\hspace{2ex} H_A:\mu_L-\mu_M\neq0\)
• \(H_0:\mu_L-\mu_U=0\), \(\hspace{2ex} H_A:\mu_L-\mu_U\neq0\)
• \(H_0:\mu_L-\mu_W=0\), \(\hspace{2ex} H_A:\mu_L-\mu_W\neq0\)
• \(H_0:\mu_M-\mu_U=0\), \(\hspace{2ex} H_A:\mu_M-\mu_U\neq0\)
• \(H_0:\mu_M-\mu_W=0\), \(\hspace{2ex} H_A:\mu_M-\mu_W\neq0\)
• \(H_0:\mu_U-\mu_W=0\), \(\hspace{2ex} H_A:\mu_U-\mu_W\neq0\)

Data

• Ridge Plot
• Boxplot
• Summary Statistics
• T-Tests

Ridge plot showing distribution of word scores for each self-identified social class

Ridge plot showing distribution of word scores for each self-identified social class

class	n	mean	sd
LOWER	41	5.07	2.24
MIDDLE	331	6.76	1.89
UPPER	16	6.19	2.34
WORKING	407	5.75	1.87

• Pairwise t-tests using pooled SD (pooled across all groups)
• No adjustment for multiple comparison

 POST HOC TESTS

 Post Hoc Comparisons - class                                                      
 ───────────────────────────────────────────────────────────────────────────────── 
   class          class      Mean Difference    SE       df     t         p        
 ───────────────────────────────────────────────────────────────────────────────── 
   LOWER     -    MIDDLE              -1.688    0.315    791    -5.353    < .001   
             -    UPPER               -1.114    0.561    791    -1.985     0.048   
             -    WORKING             -0.676    0.312    791    -2.167     0.031   
   MIDDLE    -    UPPER                0.574    0.488    791     1.177     0.240   
             -    WORKING              1.012    0.141    791     7.178    < .001   
   UPPER     -    WORKING              0.438    0.485    791     0.902     0.367   
 ───────────────────────────────────────────────────────────────────────────────── 
   Note. Comparisons are based on estimated marginal means

• It seems like there are significant differences between MIDDLE and LOWER, LOWER and WORKING, UPPER and LOWER, WORKING and MIDDLE groups
• There are no significant differences between UPPER and MIDDLE, UPPER and WORKING groups
• HOWEVER, unadjusted p-value do not account for a possibility of increased Type 1 Error

The Problem with Multiple Comparisons

• With \(m\) pairwise comparisons using a significance level of \(\alpha\), for each test, the probability of making at least one Type 1 error if there are no difference between groups is \(1-(1-\alpha)^m\)
• If each \(H_0\) is true, probability of at least one Type 1 error in 6 tests with \(\alpha = 0.05\): \[1-0.95^6=0.265\]

Familywise Error Rate

• The Familywise Error rate (FWE) is the probability of making at least one Type 1 error when performing multiple hypothesis tests
• We can control the FWE using multiple comparison methods
• These methods use a reduced significance level for each hypothesis test to ensure \(FWE\leq\alpha\)

Bonferroni Method

• The Bonferroni method is the simplest multiple comparison method
• Let \(E_i\) be the event of making a Type 1 error with test \(i\)
• For \(m\) tests, \[P(E_1 \text{ or } E_2 \text{ or }\cdots\text{ or } E_m) \leq P(E_1) + P(E_2)+\ldots + P(E_m)\]
• If each test conducted at significance level \(\alpha/m\) \[FWE\leq\frac{\alpha}{m}+\frac{\alpha}{m}+\ldots+\frac{\alpha}{m}=m\cdot\frac{\alpha}{m}=\alpha\]

• For \(m\) tests, the Bonferroni method tests each one at a level of \(\alpha/m\)
• For the Wordscore example each test would use a level of \(0.05/6=0.00833\)
• Equivalently, the p-value from each test is adjusted by multiplying by the number of tests
• The adjusted p-values are compared to the original significance level

Adjusted P-Values

Using the Bonferroni method

 POST HOC TESTS

 Post Hoc Comparisons - class                                                                      
 ───────────────────────────────────────────────────────────────────────────────────────────────── 
   class          class      Mean Difference    SE       df     t         p         p-bonferroni   
 ───────────────────────────────────────────────────────────────────────────────────────────────── 
   LOWER     -    MIDDLE              -1.688    0.315    791    -5.353    < .001    < .001   
             -    UPPER               -1.114    0.561    791    -1.985     0.048           0.285   
             -    WORKING             -0.676    0.312    791    -2.167     0.031           0.183   
   MIDDLE    -    UPPER                0.574    0.488    791     1.177     0.240           1.000   
             -    WORKING              1.012    0.141    791     7.178    < .001    < .001   
   UPPER     -    WORKING              0.438    0.485    791     0.902     0.367           1.000   
 ───────────────────────────────────────────────────────────────────────────────────────────────── 
   Note. Comparisons are based on estimated marginal means

• Using unadjusted p-values, there are quite a few significant differences
• After the Bonferroni adjustment there are significant Differences between MIDDLE and LOWER, MIDDLE and WORKING groups

• Bonferroni method is very conserviative, resulting in FWE that is usually much smaller than \(\alpha\) (loss of power)
• There are alternative methods that are less conservative and have higher power while still controlling FWE
• E.g. Tukey’s and Holm’s method (also available in Jamovi)

Tukey Procedure

• Less conservative than Bonferroni method
• Only for pairwise comparisons of means

 POST HOC TESTS

 Post Hoc Comparisons - class                                                                                 
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   class          class      Mean Difference    SE       df     t         p         p-tukey    p-bonferroni   
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   LOWER     -    MIDDLE              -1.688    0.315    791    -5.353    < .001    < .001    < .001   
             -    UPPER               -1.114    0.561    791    -1.985     0.048      0.195           0.285   
             -    WORKING             -0.676    0.312    791    -2.167     0.031      0.134           0.183   
   MIDDLE    -    UPPER                0.574    0.488    791     1.177     0.240      0.642           1.000   
             -    WORKING              1.012    0.141    791     7.178    < .001    < .001    < .001   
   UPPER     -    WORKING              0.438    0.485    791     0.902     0.367      0.804           1.000   
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   Note. Comparisons are based on estimated marginal means

• Based on Tukey’s adjusted p-values, there are significant differences between MIDDLE and LOWER, WORKING and MIDDLE groups
• We could also construct the confidence intervals for the differences between the means
• If the difference of means is significant, then the corresponding confidence interval will not contain value zero

Conclusions

• We come to the same conclusions using the Tukey procedure or the Bonferroni method (but not the unadjusted p-values!)
• Based on the results of the ANOVA, we concluded that there is convincing evidence that at least one of the mean scores is different
• We followed this with post-hoc pairwise tests for differences between group means

• Based on the pairwise tests, we conclude that there is convincing evidence of differences between mean scores for the “Middle” and “Lower” social classes and and between mean scores for the “Working and Middle” social classes.
• We are unable to reject the other null hypotheses
• For example, it is plausible that the mean scores are the same for “Upper” and “Lower” social classes

Brain Vloume Change Example

• Brain size typically shrinks as people age past adulthood, and such shrinkage may be linked to dementia.
• Any intervention that can protect against brain shrinkage could help to protect the elderly against dementia and Alzheimer’s disease.
• Researchers in China investigated whether different kinds of exercise/activity might help to prevent brain shrinkage or perhaps even lead to an increase in brain volume (Mortimer et al., 2012) ¹.

1. “Changes in Brain Volume and Cognition in a Randomized Trial of Exercise and Social Interaction in a Community-Based Sample of Non-Demented Chinese Elders”, (by James A. Mortimer Ding Ding, Amy R. Borenstein, Charles DeCarli, Qihao Guo, Yougui Wu, Qianhua Zhao and Shugang Chu), J. of Alzheimer’s Disease 30 (2012) 757-766

• The researchers randomly assigned elderly adult volunteers into four activity groups: Tai Chi, Walking, Social interaction, and No intervention.
• Except for the group with no intervention, each group met for about an hour three times a week for 40 weeks to participate in their assigned activity.
• The tai chi group was led by a tai chi master and an assistant, the walking group walked around a track, the social interaction group met at a community center and discussed topics that interested them, and the no- intervention group just received four phone calls during the study period.
• A total of 120 participants started the study, and 13 dropped out along the way, so 107 completed the study.

• Each participant had an MRI to determine brain volume before the study began and again at its end.
• The researchers measured the percentage change in brain volume in each participant’s brain during that time.
• The researchers thought that physical activity would help increase brain volume; hence they anticipated that the tai chi and walking groups would tend to show larger increases in brain volume during the study than the control group and the social interaction group.
• Dataset available here

EDA

• Ridge Plot
• Boxplot
• Summary Statistics

Ridge plot showing distribution of the percentage brain change for each treatment

Ridge plot showing distribution of word scores for each self-identified social class

# A tibble: 4 × 4
  Treatment     n   mean    sd
  <chr>     <int>  <dbl> <dbl>
1 None         24 -0.198 1.21 
2 Social       27  0.520 0.611
3 TaiChi       29  0.673 0.802
4 Walking      27  0.450 1.05 

Inference

• Treatment: “TaiChi”, “Social”, “Walking”, “None”

• Let \(\mu_C\) be the mean percentage brain volume change for each type of activity

• We will conduct a hypothesis test with hypotheses

  • \(H_0: \mu_{TaiChi}=\mu_{Social}=\mu_{Walking}=\mu_{None}\)
  • \(H_A:\) at least one of the means is different

• Equivalently, we can state the hypotheses as

  • \(H_0\): There is no association between the type of activity and the changes in the brain volume
  • \(H_A:\) There is an association between the type of activity and the changes in the brain volume

Random Permutation

• To simulate independence between brain volume change and activity, we randomly permute the values of the explanatory variable

• After each permutation we calculate F-statistics

• Below are five such permutations

     id BrainChange Treatment randPerm1 randPerm2 randPerm3 randPerm4 randPerm5
1     1       1.987    TaiChi   Walking   Walking    Social    TaiChi    TaiChi
2     2       1.960    TaiChi   Walking    Social      None    TaiChi   Walking
3     3       0.304    TaiChi    TaiChi    TaiChi    TaiChi      None    TaiChi
4     4       0.005    TaiChi    TaiChi    TaiChi   Walking    Social      None
5     5      -0.829    TaiChi    TaiChi   Walking    Social    TaiChi   Walking
6     6       1.227    TaiChi   Walking   Walking    Social    TaiChi      None
7     7       1.179    TaiChi      None      None      None      None    Social
8     8       0.541    TaiChi   Walking    Social    Social   Walking    TaiChi
9     9       0.388    TaiChi    Social    Social    TaiChi    Social   Walking
10   10       0.610    TaiChi    Social    TaiChi   Walking    Social   Walking
11   11       0.049    TaiChi    Social    TaiChi    TaiChi      None    TaiChi
12   12       0.492    TaiChi      None    TaiChi   Walking    TaiChi   Walking
13   13       0.179    TaiChi   Walking      None   Walking   Walking      None
14   14       1.383    TaiChi    TaiChi      None    Social   Walking      None
15   15      -0.623    TaiChi   Walking      None    Social    TaiChi    Social
16   16       1.777    TaiChi   Walking   Walking   Walking   Walking   Walking
17   17       0.356    TaiChi    Social    Social   Walking      None    Social
18   18      -0.217    TaiChi   Walking    Social   Walking    TaiChi    TaiChi
19   19       0.449    TaiChi    Social   Walking    Social    TaiChi    Social
20   20      -0.728    TaiChi    Social      None    TaiChi    TaiChi      None
21   21       1.040    TaiChi    TaiChi   Walking      None    TaiChi   Walking
22   22       0.614    TaiChi      None    Social    TaiChi    Social   Walking
23   23       1.482    TaiChi    Social   Walking    TaiChi      None    TaiChi
24   24       0.386    TaiChi      None      None    TaiChi      None   Walking
25   25       0.435    TaiChi   Walking    Social   Walking      None      None
26   26       1.618    TaiChi      None      None    Social    TaiChi      None
27   27       0.576    TaiChi    TaiChi    TaiChi    TaiChi   Walking      None
28   28       0.678    TaiChi   Walking    TaiChi    TaiChi      None    Social
29   29       2.201    TaiChi      None      None   Walking      None   Walking
30   30       1.123   Walking    Social    TaiChi    TaiChi      None    Social
31   31       0.990   Walking   Walking   Walking      None   Walking    TaiChi
32   32       0.839   Walking   Walking   Walking    TaiChi   Walking    Social
33   33      -0.427   Walking    Social   Walking    Social      None    TaiChi
34   34      -0.579   Walking   Walking   Walking    TaiChi      None    TaiChi
35   35       0.617   Walking    TaiChi    TaiChi    Social    Social      None
36   36       1.833   Walking   Walking      None      None   Walking    TaiChi
37   37      -1.632   Walking    TaiChi    Social   Walking   Walking   Walking
38   38       2.762   Walking    TaiChi    TaiChi   Walking      None    Social
39   39      -0.377   Walking      None   Walking   Walking    Social    TaiChi
40   40      -1.343   Walking      None      None      None    TaiChi    Social
41   41      -0.652   Walking    TaiChi   Walking      None   Walking    Social
42   42      -0.994   Walking    Social   Walking   Walking    Social   Walking
43   43      -0.026   Walking      None    TaiChi   Walking    TaiChi    TaiChi
44   44       0.411   Walking    TaiChi    Social    TaiChi    TaiChi      None
45   45       0.364   Walking    TaiChi   Walking   Walking    Social    Social
46   46       0.952   Walking    Social      None    Social   Walking      None
47   47       0.470   Walking    TaiChi    Social    Social    Social      None
48   48       1.145   Walking      None    TaiChi    Social    Social    Social
49   49       1.338   Walking   Walking    TaiChi   Walking    Social    Social
50   50       1.492   Walking   Walking   Walking    TaiChi    Social    TaiChi
51   51       1.105   Walking   Walking   Walking    TaiChi      None    TaiChi
52   52      -1.061   Walking    TaiChi   Walking      None   Walking    Social
53   53       0.694   Walking    TaiChi    Social   Walking    TaiChi    TaiChi
54   54       1.210   Walking      None      None   Walking    TaiChi    Social
55   55       1.484   Walking    Social      None    TaiChi      None      None
56   56       0.411   Walking      None    Social      None    TaiChi   Walking
57   57       1.001    Social    Social    TaiChi    Social      None    TaiChi
58   58       0.130    Social      None    TaiChi      None    TaiChi   Walking
59   59       0.276    Social    TaiChi      None    TaiChi    TaiChi    TaiChi
60   60       0.708    Social    Social    TaiChi    TaiChi    TaiChi    Social
61   61       0.672    Social      None      None    Social   Walking   Walking
62   62       0.490    Social   Walking    Social    Social    Social      None
63   63       0.822    Social    Social   Walking    Social    Social    TaiChi
64   64      -1.179    Social    Social   Walking      None      None    TaiChi
65   65       0.776    Social    Social    TaiChi   Walking      None      None
66   66       1.796    Social      None    Social      None   Walking    Social
67   67       0.165    Social      None    TaiChi    Social    TaiChi    TaiChi
68   68       0.412    Social    TaiChi   Walking    TaiChi    Social   Walking
69   69       0.805    Social    Social    Social      None    TaiChi    Social
70   70       0.529    Social    TaiChi    Social    Social      None   Walking
71   71      -0.050    Social    Social    Social   Walking    Social      None
72   72       0.559    Social    TaiChi   Walking      None   Walking      None
73   73       0.807    Social      None   Walking      None   Walking    TaiChi
74   74       0.596    Social   Walking    TaiChi    TaiChi   Walking   Walking
75   75       0.813    Social    TaiChi      None   Walking   Walking    TaiChi
76   76       0.803    Social    TaiChi    Social    TaiChi    Social   Walking
77   77       1.701    Social    TaiChi    TaiChi      None    TaiChi   Walking
78   78      -0.513    Social    TaiChi   Walking   Walking    Social      None
79   79       0.065    Social   Walking    Social      None    Social   Walking
80   80      -0.359    Social      None      None    TaiChi    TaiChi      None
81   81       0.613    Social    TaiChi    Social      None   Walking      None
82   82       0.555    Social    Social    Social      None   Walking    TaiChi
83   83       1.059    Social      None    Social      None      None   Walking
84   84      -1.347      None    TaiChi      None      None   Walking    Social
85   85       1.665      None   Walking      None    Social      None    TaiChi
86   86      -1.673      None    Social    Social      None    Social    Social
87   87       1.052      None    Social    Social      None    TaiChi    TaiChi
88   88      -0.956      None    TaiChi      None    TaiChi    Social    Social
89   89      -0.563      None   Walking    TaiChi      None      None    TaiChi
90   90       0.611      None    TaiChi   Walking    Social    Social    TaiChi
91   91      -1.540      None      None    Social   Walking    Social    Social
92   92       1.272      None   Walking    TaiChi   Walking    Social      None
93   93      -1.195      None    TaiChi      None   Walking      None    Social
94   94      -0.811      None      None    Social   Walking    TaiChi    Social
95   95      -1.138      None      None    TaiChi    Social   Walking   Walking
96   96       0.946      None   Walking   Walking    Social    TaiChi   Walking
97   97      -0.093      None   Walking    TaiChi    TaiChi   Walking   Walking
98   98      -0.887      None      None    TaiChi    TaiChi   Walking      None
99   99       1.762      None    Social      None    TaiChi   Walking    TaiChi
100 100       2.011      None    TaiChi    TaiChi    Social    TaiChi   Walking
101 101      -0.333      None    Social   Walking    TaiChi    Social    Social
102 102      -0.607      None    Social    TaiChi    Social   Walking      None
103 103       1.198      None   Walking    TaiChi    Social    Social    TaiChi
104 104      -1.083      None   Walking      None   Walking      None   Walking
105 105      -1.160      None    Social    Social    TaiChi    Social      None
106 106      -2.034      None      None      None    Social    TaiChi    Social
107 107       0.140      None    Social    TaiChi    TaiChi   Walking    Social

• And the simulated values of the F-statistics

Rows: 5
Columns: 2
$ replicate <int> 1, 2, 3, 4, 5
$ stat      <dbl> 0.3301483, 0.6679932, 0.7408042, 1.0869489, 0.6699737

• The observed value of the F-statistic from the data is \(F=4.24\)

 ANOVA

 ANOVA - BrainChange                                                            
 ────────────────────────────────────────────────────────────────────────────── 
                Sum of Squares    df     Mean Square    F           p           
 ────────────────────────────────────────────────────────────────────────────── 
   Treatment          11.18543      3      3.7284771    4.243280    0.0071804   
   Residuals          90.50384    103      0.8786780                            
 ────────────────────────────────────────────────────────────────────────────── 

Finding p-value

• Permutations
• Mathematical Model
• ANOVA Table

• There are 7 randomized \(F\) statistics that are at least as large as the observed value (4.24)
• The p-value is \(7/1000 = 0.007\)

• Technical conditions are satisfied (look at EDA)
  • Normality: Dist’n of each group is not extremely skewed
  • Equal variance: \(1.21\le 2 \cdot 0.611\)

• Jamovi output

 ANOVA

 ANOVA - BrainChange                                                            
 ────────────────────────────────────────────────────────────────────────────── 
                Sum of Squares    df     Mean Square    F           p           
 ────────────────────────────────────────────────────────────────────────────── 
   Treatment          11.18543      3      3.7284771    4.243280    0.0071804   
   Residuals          90.50384    103      0.8786780                            
 ────────────────────────────────────────────────────────────────────────────── 

Follow-up Analysis

• Since the p-value of the ANOVA test was less that \(\alpha=0.05\) we now can proceed with a follow-up analysis
• We will consider both Bonferroni and Tukey methods
• Bonferroni Method:
  • For 4 groups there will be 6 tests, so the Bonferroni method tests each one at a level of \(\alpha/6=0.05/6=0.0083\)
• Tukey procedure will calculate 95% confidence intervals for pairwise difference of means.
  • If difference is statistically significant then corresponding CI will not contain value \(0\).

Bonferroni Method

• Adjusted p-values

 POST HOC TESTS

 Post Hoc Comparisons - Treatment                                                                      
 ───────────────────────────────────────────────────────────────────────────────────────────────────── 
   Treatment         Treatment    Mean Difference    SE       df     t         p        p-bonferroni   
 ───────────────────────────────────────────────────────────────────────────────────────────────────── 
   None         -    Social               -0.7189    0.263    103    -2.734    0.007           0.044   
                -    TaiChi               -0.8715    0.259    103    -3.369    0.001           0.006   
                -    Walking              -0.6484    0.263    103    -2.466    0.015           0.092   
   Social       -    TaiChi               -0.1526    0.251    103    -0.609    0.544           1.000   
                -    Walking               0.0705    0.255    103     0.276    0.783           1.000   
   TaiChi       -    Walking               0.2231    0.251    103     0.890    0.376           1.000   
 ───────────────────────────────────────────────────────────────────────────────────────────────────── 
   Note. Comparisons are based on estimated marginal means

Using significance level 0.05 on adjusted p-values, there are significant differences between TaiChi and None and Social and None

Tukey Procedure

 POST HOC TESTS

 Post Hoc Comparisons - Treatment                                                                                 
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   Treatment         Treatment    Mean Difference    SE       df     t         p        p-tukey    p-bonferroni   
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   None         -    Social               -0.7189    0.263    103    -2.734    0.007      0.036           0.044   
                -    TaiChi               -0.8715    0.259    103    -3.369    0.001      0.006           0.006   
                -    Walking              -0.6484    0.263    103    -2.466    0.015      0.071           0.092   
   Social       -    TaiChi               -0.1526    0.251    103    -0.609    0.544      0.929           1.000   
                -    Walking               0.0705    0.255    103     0.276    0.783      0.993           1.000   
   TaiChi       -    Walking               0.2231    0.251    103     0.890    0.376      0.810           1.000   
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
   Note. Comparisons are based on estimated marginal means

Based on the Tukey’s procedure, there are significant differences between TaiChi and None and Social and None

Conclusion

• We have strong evidence against the null hypothesis and in support of an association between activities and change in brain volume.

• In other words, there is significant difference in the brain volume changes between the groups

• We cannot generalize to a larger population since it was not a random sample (the participants were volunteers)

• We can draw cause-and-effect conclusion since it was a randomized experiment

• Based on the follow-up analysis, there are significant differences in the average brain volume change between groups TaiChi and None and Social and None

Additional Thoughts

• We can also perform post-hoc tests after we perform a hypothesis test for multiple proportions (e.g., a chi-squared test)
• We can calculate confidence intervals for pairwise differences (means or proportions) using the same ideas
• Bonferroni correction can be applied to the confidence level
• Use \(100\cdot(1-\alpha/m)\%\) for \(m\) comparisons
• For a 95% confidence level, we would compute CI for the 6 pairwise differences with confidence level \(100\cdot(1-0.05/6)=99.17\%\)

Multiple Proportions test (Optional)

• Dataset attendance available here

• In a Midwestern college students from three majors (“STEM”,“Business” and “Art”) were surveyed about how often they attend the college’s Help Center.

• Each respondent chooses one of three opinions: “Never”,“Sometimes” and “Often”.

• The goal is to determine whether there is an association between students’ major and the attendance.

Data summary

• Data
• Expected
• \(\chi^2\) test

Major	Never	Sometimes	Often	Total
STEM	20	10	5	35
Business	20	10	5	35
Arts	20	19	19	58
Total	60	39	29	128

Major	Never	Sometimes	Often	Total
STEM	20(16.41)	10(10.67)	5(7.92)	35
Business	20(16.41)	10(10.67)	5(7.92)	35
Arts	20(27.18)	19(17.67)	19(13.14)	58
Total	60	39	29	128

 χ² Tests                       
 ────────────────────────────── 
         Value    df    p       
 ────────────────────────────── 
   χ²     8.43     4    0.077   
   N       128                  
 ────────────────────────────── 

Attendance frequency does not differ significantly across majors when all three levels are used.

Combine response categories

• The categories Sometimes and Often were not clearly defined

• Let us combine them into new Attend category.
  Now the responses are:

• Never

• Attend (= Sometimes + Often)

Comparison of multiple proportions

• Since our response variable is now binary we can define “successes” and “failures”
• “Success” is “Attend”
• So we can compare proportions of successes in all three majors
• Then the null hypothesis and the alternative hypotheses could be stated as \[H_0: p_{STEM}=p_{Business}=p_{Art}\\H_A: \ At\ least\ one\ proportion\ is\ different\]

Test results (3×2)

• Data
• Visualization
• Test
• Post-Hoc

Major	Never	Attend	Total
STEM	20	15	35
Business	20	15	35
Arts	20	38	58
Total	60	68	128

 χ² Tests                       
 ────────────────────────────── 
         Value    df    p       
 ────────────────────────────── 
   χ²     6.54     2    0.038   
   N       128                  
 ────────────────────────────── 

There is a significant difference among majors in the likelihood of attending the Academic Support Center.

    Pairwise comparisons using Pairwise comparison of proportions 

data:  x out of n 

         Stem Business
Business 1.00 -       
Art      0.16 0.16    

P value adjustment method: holm 

Holm correction

The Holm procedure is a sequential adjustment that controls the family-wise error rate more efficiently than Bonferroni.

It works by:
1. Ordering the raw p-values from smallest to largest.
2. Comparing each to a sequentially larger α threshold (α/m, α/(m−1), …).
3. Once a test is not significant, larger p-values are automatically considered non-significant.

Final conclusion

• When attendance was categorized into Never, Sometimes, and Often, no significant association was found.
  
• When simplified into Attend (Sometimes + Often) vs. Never, the result became significant.
  
• This suggests that Art students attend the Help Center more frequently than other majors.