Power Curves

Problems of Null Hypothesis Significance Testing

In most social-scientific investigation, the exact treatment effect remains difficult to quantify. A researcher may be able to deduce a prediction from theory that, for example, threat messages are more prone to incur persuasive boomerang (creating attitude change in the opposite of the intention of the message sender) compared to no-threat messages. The approximate effect of the manipulation (e.g., expected mean difference between the threat and the no-threat condition on a 5-point scale) is still hard to specify a priori. Theories in social science, lacking in mathematical rigor, would only tell the direction of the impact, but its magnitude almost always remains unknown.

This imprecision in theoretical knowledge precludes a direct testing of the research hypothesis. Alternatively, under the current paradigm of social-scientific research, one would establish a null prediction claiming no systematic treatment effect, and by statistically demonstrating that the data contradict the nil null, the researcher suggests accepting the research prediction as true (i.e., the principle of proof by contradiction).

The decision rule to reject the null depends on three quantities: the size of the effect (e.g., mean difference between the treatment and the control condition), the amount error (e.g., unpredictable and uncontrollable individual variations), and the sample size. Noting that most parametric statistics are expressed by the ratio of the effect to the error, a statistic must increase in magnitude with increased effect and/or with decreased error. And, with all else equal, larger statistics raise the chance to reject the null and accept the alternative prediction as true.

Importantly, in parametric statistics, the error term is standardized by adjusting it with the sample size such that the overall standard error declines in half when the sample size quadruples.[Page 1311] This means one can lower the size of the error by increasing the sample size, and in turn, inflate the chance to reject the null artificially. That is, with exactly the same effect and the same amount of error term (i.e., standard deviation), the magnitude of the statistic—hence the likelihood of rejecting the null—may vary depending on the size of the sample; a large effect may appear statistically meaningless when the sample size was small, whereas a trivial effect can be rendered “statistically significant” when the test involved an enormous sample.

Statistical Power as Remedy: An Example

The concept of statistical power helps prevent such abusive practice of statistics by requiring the practical effect (cf. statistical significance) to be specified prior to executing the experiment. Practical effect is the effect that has a minimum practical implication. For the owner of a factory, the decision to adopt a new assembly line depends on whether or not it would bring about the least amount of increase in productivity that is necessary to see a substantive improvement for the corporate management (e.g., increased revenue, financial power to hire more employees). Suppose the company is currently producing 138 units per day. The owner knows from past experience that a production increase of at least four more units is needed to create a practically meaningful increase in revenue. That distance between 142 (H1) and 138 (H0) represents a practical effect (or practical difference).

...