Post Hoc Tests: Tukey Honestly Significant Difference Test

There are various types of tests that can be used in experimental designs. If an analysis indicates there may be interesting information that was not targeted for analysis in the preplanned tests, then the Tukey honestly significant difference (HSD) test is very valuable. Using this type of post hoc test, a researcher can further analyze the results after the data have been collected and initial analyses run.

The Tukey HSD test is a post hoc test used when there are equal numbers of subjects contained in each group for which pairwise comparisons of the data are being made. Post hoc tests, like this one, literally mean after the fact. They are used to determine whether any group or set of treatment conditions significantly differs from one or more others. The Tukey HSD test is more likely to identify statistically significant differences than [Page 1306]other post hoc tests. This entry discusses the utility of the Tukey HSD post hoc test, gives a thorough developmental overview, and then provides further elaboration.

The Tukey HSD test is used after a significant F ratio is found via an analysis of variance (ANOVA) test. Running an ANOVA will simply not provide information about the specific differences between groups or treatments; therefore, it is important to use tests like the Tukey HSD. For example, suppose that a researcher conducted a one-way ANOVA comparing four groups (A, B, C, D). A significant F value would not specify which group or combination of groups is significantly different from any other group or groups. Group A might differ from Group B, but not Group C or D, whereas Group D might differ from both Groups B and C. A significant F value is only the first step in understanding and explaining the data. Further testing, in this case through post hoc tests, is necessary to discern the specific pattern of differences between treatment conditions or groups. This test is conducted when there are an equal number of cases per group, but it can be done with unequal group sizes, so long as there is not a great difference in number of subjects per group.

Performing multiple t-tests leads to an increased risk of committing a type I error; avoiding this is part of the motivation for conducting post hoc tests. A type I error occurs when a researcher incorrectly rejects a true null hypothesis. In other words, a researcher erroneously reports finding a significant difference between groups that in fact do not differ from one another. This is one of the most intolerable errors that researchers can make. To avoid committing a type I error, post hoc tests like the Tukey HSD increase the critical value needed to reject the null hypothesis.

The Tukey HSD test is designed to examine all possible pairs of means while maintaining type I error rates for making comparisons at alpha (significance level). There are two different types of type I error, comparison-wise and experiment-wise. Comparison-wise error rate is the probability of making a type I error for any of the possible individual comparisons in an experiment. Conversely, the probability of making type I errors for the full set of possible comparisons in an experiment is called experiment-wise error. The Tukey HSD test maintains an experiment-wise type I error rate, which is really a way of saying it maintains a type I error rate at alpha for the entire set of comparisons, not just each individual comparison. For example, instead of calculating a critical t-value for each individual comparison, the Tukey HSD calculates only one critical value to use for all post hoc comparisons. Of the countless post hoc tests available, many cite the Tukey HSD as their favorite because of the power it exerts over alpha.

...