Cochran Q Test

The Q Statistic

In the main paper in which Q appears, Cochran was concerned with combining estimates from k separate experiments. The types of experiments included determinations of a physical or astronomical constant, [Page 300]bioassays, and agricultural field experiments; typical estimates were a simple mean, a difference between two means, a median lethal dose, and a regression coefficient. Each experiment provided an estimate, yi, and an estimate, $s_i^2$, of the variance of yi. Also, each $s_i^2$ had a number of degrees of freedom, ni, which would ordinarily come from a mean square (e.g., the sample variance or the residual mean square of a regression). Thus, the setting differed from most meta-analyses. Ideally, all experiments were estimating the same quantity, μ, but it could vary among experiments (i.e., the quantities could be μi instead of a single μ). Thus, Q summarized the variation among the estimates in the form of a weighted sum of the squared deviations of the yi from the weighted mean ${\overline{y}}_w=\Sigma w_i{y}_i/\Sigma w_i$ with weights $w_i=1/s_i^2$,

$$Q={\displaystyle \sum _{i=1}^k{w}_i{(y_i-{\overline{y}}_w)}^2}.$$

A large value of Q indicates heterogeneity among the yi. The degrees of freedom, ni, are not used in calculating Q; but they do appear, for example, in an approximate formula for the standard error of ${\overline{y}}_w$.

When the degrees of freedom (ni) are “large” and the μi are equal, the distribution of Q approaches the chi-square distribution on k−1 degrees of freedom. The literature, however, provides little information on a quantitative definition of “large.”

In a meta-analysis, yi is usually the estimate of the effect in Study i (e.g., the standardized mean difference or a difference of proportions), and $s_i^2$ is an estimate of its (within study) variance. The test of homogeneity refers Q to the chi-square distribution on k−1 degrees of freedom and rejects the null hypothesis if p < .10 (the criterion p < .05 is less common because the test is considered to have low power). Many authors routinely use chi-square on k−1 df as the null distribution. This procedure is understandable because the $s_i^2$ in a meta-analysis are rarely, if ever, accompanied by numbers of degrees of freedom. Indeed, some common measures of effect, such as a difference in proportions, do not have a natural way to define a number of df. When the ni are available, they are often closely related to the total sample size of the two groups in the study. In many meta-analyses, the total sample size is not “large,” so the test of homogeneity is unreliable.

...