Meta-Analysis

The term meta-analysis refers to the statistical analysis of results from individual studies for purposes of integrating the findings. The person who coined the term in 1976, Gene Glass, felt that the accumulated findings of studies should be regarded as complex data points that were no more comprehensible without statistical analysis than the data points found in individual studies.

Procedures for performing meta-analysis had appeared in statistics texts and articles long before 1976, but instances of their application were rare. It took the expanding evidence in the social sciences and the growing need for research syntheses to provide impetus for the general use of meta-analysis. The explosion in social science research focused considerable attention on the lack of standardization in how scholars wishing to integrate literatures arrived at general conclusions from series of related studies. For many topic areas, a separate verbal description of each relevant study was no longer possible. One traditional strategy was to focus on one or two studies chosen from dozens or hundreds. This strategy failed to portray accurately the accumulated state of knowledge. First, this type of selective attention is open to confirmatory bias: A particular synthesist may highlight only studies that support his or her initial position. Second, selective attention to only a portion of all studies places little or imprecise weight on the volume of available tests. Finally, selectively attending to evidence cannot give a good estimate of the strength of a relationship. As evidence on a topic accumulates, researchers become more interested in “how much” rather than simply “yes or no.”

Traditional synthesists also faced problems when they considered the variation between the results of different studies. Synthesists would find distributions of results for studies sharing a particular procedural characteristic but varying on many other characteristics. They found it difficult to conclude accurately whether a procedural variation affected study outcomes because the variability in results obtained by any single method meant that the distributions of results with different methods would overlap.

It seemed, then, that there were many situations in which synthesists had to turn to quantitative synthesizing techniques, or meta-analysis. The application of quantitative inference procedures to research synthesis [Page 636]was a necessary response to the expanding literature. If statistics are applied appropriately, they should enhance the validity of synthesis conclusions. Metaanalysis is an extension of the same rules of inference required for rigorous data analysis in primary research.