Statistical Power

In using sample data to test hypotheses about population values, one possibility is that the data will lead you to retain the null hypothesis even though it is false. This is Type II error (β). To accentuate the positive rather than the negative, however, the term power has been used to define the converse of Type II error: The power of a statistical test is the probability of correctly rejecting the null hypothesis when it is false, or P = 1 − β. Statistical power is an extremely important concept because most social science research is designed to demonstrate that a particular variable does, in fact, influence behavior. We set out to find evidence that the “hypothesis of no effect” (the null hypothesis) is indeed false. We want to demonstrate that a particular form of therapy is effective, that an educational program works, that world events do influence political opinion, and so forth. To do these things, we desire high levels of statistical power.

Although statistical power will decrease when α, the probability of a Type I error (rejecting a true null hypothesis), is set low, there are some principles of good experimental design that can be used to increase power: use large samples, develop reliable response measures, select homogeneous subject populations, use repeated measures on the same subjects, and use standardized and unambiguous tasks and instructions. One factor that we cannot control, however, is the true value of the population parameter being estimated by our statistical test. The greater the discrepancy between the value stated in the null hypothesis and the true situation, the greater the statistical power. Logically (and fortunately), we will be more apt to detect the effect of a strong (powerful) treatment than a weak one. Some researchers plot power functions or power curves to show how the power of their statistical test varies depending on the degree of falsity of the null hypothesis (Cohen, 1988). Conversely, some researchers will consult previous studies or conduct their own pilot work to estimate the size of their experimental effect and then use this to determine the sample size they would need in their main study to attain a predetermined level of statistical power, say .90. (See Cohen, 1988, for how this is done.)

Hays (1994) likens statistical power to the power of a microscope. Just as a high-powered microscope increases our chances of detecting something not readily visible with a low-powered instrument, so does a high-powered test increase our chances of detecting when the null hypothesis is false. However, even a high-powered microscope is limited by the size of the object it is attempting to detect. So it is that, all other things being equal, the larger the effect that we are trying to detect in our research, the greater will be the power of our tests for detecting it. Good research techniques will also increase the likelihood of detecting the effect.