Type I Error

Understanding Type I Errors

In statistical analysis, hypothesis testing is used to determine whether the variations among different groups can be attributed to a manipulation or random chance. In educational contexts, hypothesis testing generally includes two types of hypotheses: a null hypothesis and alternative hypothesis. A null hypothesis states that the phenomenon or manipulation under investigation produces no effect (i.e., or makes no difference). An alternative hypothesis (also referred to as the research hypothesis) states the opposite of the null hypothesis; that is, an alternative hypothesis states that the phenomenon or manipulation under investigation does produce an effect (i.e., it does make a difference). An example of a null hypothesis is “The new teaching strategy does not positively influence students’ learning outcomes.” An example of an alternative hypothesis is “The new teaching strategy does positively influence students’ learning outcomes.”

Because researchers want to correctly conclude that variations among different groups are attributed to a manipulation rather than random chance, researchers take precautions to avoid making a false claim. That is, researchers take precautions against making a Type I error. This precaution is setting a level of significance, which is really just a “rejection of the null hypothesis” decision threshold that is based on probability. For example, a researcher may set a level of significance to .05, meaning that the researcher is willing to say 5 times out of 100 that the variation among the groups is attributed to the manipulation when, in fact, the variation among the groups is attributed to random chance. In other words, a level of significance of .05 means that 5 times out of 100, the researcher will commit a Type I error (i.e., claiming that the variations among the different groups is attributed to a manipulation when, in fact, the variation is attributed to random chance).

Consider the following hypotheses:

[Page 1741]If the level of significance is set to .01 and the researchers reject the null hypothesis based on their significant statistical results, then the researcher has a 99 in 100 chance of correctly saying that the new measure is a better predictor of academic outcomes than is the old measure and a 1 in 100 chance of incorrectly saying that the new measure is not a better predictor of academic outcomes than is the old measure. In other words, the researcher has a 1 in 100 chance of making a Type I error (i.e., claiming that the new measure is indeed a better predictor of academic outcomes than the old measure when, in reality, the new measure is not a better predictor of academic outcomes than is the old measure).

...