Power Analysis

Test Statistic

The empirical data for a research study are first collected from a representative sample and then summarized by a test statistic. The type of statistic used to test the research hypothesis is determined by the research design. Depending on random assignment, a study can be classified as either an observational study or a true experiment. An observational study does not involve any randomization. In this type of study, a t statistic can be used to test the correlation between two continuous variables. However, a true experiment involves random assignment of subjects into treatment and control conditions. A two-group randomized experiment (one treatment and one control group) can use a Z test to compare the two population means if the population standard deviation is known; when the standard deviation is unknown, a t statistic is used to compare the two population means. If there are two treatment groups and one control group in the randomized experiment, an F statistic will be computed to detect any mean differences among the three groups.

Hypothesis testing uses the probability behavior of the test statistic to assess the plausibility of the null and alternative hypotheses. The test statistic can deviate from its most expected value assuming the null hypothesis is true, but it can do so with predictably decreasing probability. The more the test statistic differs from the most expected value, the less likely such a test statistic can occur by chance. If the test statistic is highly discrepant from the value expected under the null hypothesis, it is construed as significantly contradicting the null hypothesis. In this case, the null hypothesis is deemed implausible and shall be rejected.

How deviant must the test statistic be from its expected value under the null hypothesis before it can be rejected? It depends on the p value of the test statistic or the probability of obtaining a statistic at least as deviant as the observed one. A small p value suggests that the statistic is atypical of its probabilistic occurrence when the null hypothesis is true. The null hypothesis therefore is rejected, [Page 1283]and the alternative hypothesis (i.e., the research hypothesis) is plausible. On the contrary, a large p value indicates that such a statistic occurs often and does not contradict the null hypothesis. In this case, the null hypothesis shall be retained—the research hypothesis is not supported.

Significance Level

A p value less than or equal to 5% (i.e., p ≤ .05) is conventionally considered small enough to reject the null hypothesis, and the 5% is the significance level (α). In practice, the exact p value of the computed test statistic does not have to be calculated. The computed test statistic is often compared with a critical value, whose p value is known to exactly equal the significance level of 5%. If the computed test statistic is further away from its expected value than the critical value, it can be inferred that the actual p value of the computed test statistic is smaller than the significance level of 5%, and the null hypothesis shall be rejected.

...