File Drawer Problem

A meta-analysis is a quantitative summary or synthesis of findings of studies that focus on a common question; one example is a quantitative synthesis of results of studies that focus on the efficacy of psychotherapy. Unfortunately, studies that are included in a meta-analysis can be unrepresentative of all the methodologically sound studies that address this common question, so the combined results of the studies in the meta-analysis can be misleading. Included studies may be unrepresentative because of the well-documented “publication bias,” which refers to a bias against publication of results of studies that do not yield statistically significant results. Because of this bias, results of studies that are not statistically significant often (a) do not appear in print (either as journal articles or as published abstracts of presentations), (b) wind up tucked away in researchers' “file drawers,” and (c) remain undetected or inaccessible to meta-analysts. In the most extreme case of the “file drawer problem,” the collection of studies included in a meta-analysis consists exclusively of those that yielded results significant at the conventional .05 level.

The most popular method of dealing with the file drawer problem involves calculation of Robert Rosenthal's Fail-Safe-N (FSN). The FSN—which was derived under the (questionable) assumptions that (a) the studies targeted by meta-analyses use two-tailed (nondirectional) tests and (b) the studies in the file drawers average null results—is an estimate of the minimum number of unpublished studies (tucked away in file drawers) that would threaten the validity of significant combined results of a meta-analysis. For example, for a well-known 1982 meta-analysis (by Landman and Dawes) focusing on a set of 42 studies of efficacy of psychotherapy that were considered (by the meta-analysts) to be appropriately controlled, the FSN was 461. Since the combined results of this meta-analysis indicated statistically significant beneficial effects of psychotherapy, it was inferred (by FSN users) that there would have to exist at least 461 unpublished file drawer studies (averaging null results) to threaten the validity of this conclusion. Although there are no firm guidelines for interpretation of FSNs, Rosenthal suggested using FSNc = K(5) + 10 as a critical value or rule of thumb (where K = number of studies in the meta-analysis); thus only FSNs below FSNc would be considered to threaten significant combined results of a meta-analysis. For the Landman and Dawes meta-analysis, the FSN of 461 is well above the FSNc of 220 (i.e., 42(5) + 10 = 220), suggesting to users of the FSN that the file drawer problem was negligible in this meta-analysis.