Binomial Effect Size Display

The binomial effect size display (BESD) is an intuitively appealing display of the magnitude of an experimental effect. Communication of statistical information can often be improved by selecting an adequate representation through which the statistic is communicated. For example, risk information is commonly better understood when communicated in the form of natural frequencies (e.g., 2 out of 100 patients experience a side effect) than in the form of probabilities (e.g., the probability of a side effect is .02) or percentages (e.g., 2% of patients experience a side effect). When statistical information is not communicated in a readily understood format, the likelihood of misinterpretation is high. One piece of statistical information vulnerable to misinterpretation is the strength of an experimental effect or effect size.

An effect size quantifies the magnitude of an effect and is typically reported using the indices r, r2, d, or g. The interpretation of the magnitude of an effect is challenging, though, in part because the interpretation requires an anchor or standard of comparison and because it is challenging to understand the implications of a small effect. At what size is an effect small or large? What does it mean to observe a small effect? Following Jacob Cohen, effect sizes are commonly classified as small (d = .2), medium (d = .5), and large (d ≥ .8). This classification helps comparing effect sizes and interpreting the magnitude of effects. However, whereas the relevance of a medium or large effect is evident in that it indicates a substantial effect of an independent variable or an intervention, the relevance and practical implications of small effects are often less clear. What does it mean when a study reports an effect size of d = .2? Does it mean the effect is negligible? Robert Rosenthal and Donald B. Rubin noticed that researchers often underestimate the importance of an intervention if it is related to small effects. To illustrate that small effects can be practically relevant, the authors introduced the binomial effect size display (BESD). The BESD tries to answer the following question: What is the effect of an intervention or treatment on the success rate, improvement rate, or selection rate given a specific criterion? As Rosenthal and Rubin demonstrate, success rates can differ remarkably even if effect sizes are small.

Typically applied within an experimental research context, the BESD specifies changes in the experimental treatment’s success rate when compared to a control condition (see Table 1). The BESD illustrates an independent variable’s effect on a criterion’s success rate and is reckoned as the difference between the success rate in the treatment condition and the success rate in the control condition. The experimental success rate is computed as .50 + r/2 and the control condition success rate is computed as .50 − r/2, where r indicates a correlation coefficient. Accordingly, the magnitude of r is identical to the success rate’s improvement.

Table 1 Binomial Effect Size Display (BESD) for Small Effect Sizes

Behavioral interventions often rely on reporting effect sizes as the proportion of variance explained in an outcome variable (see r2 in Table 1). When associated with small correlation coefficients, these values are very low and can greatly mislead interpretation of the relevance and practical implications of an experimental effect. For example, as Table 1 illustrates, a substantial success rate increase from 40.2% to 59.8% may be misinterpreted as an irrelevant and negligible effect if the effect size is reported as only 4% (r2 = .04) of the variance explained. As even a 20% increase in success rate may be misrepresented as a practically not relevant effect, using the BESD may offer an appropriate option for researchers aiming to communicate the practical relevance of small effect sizes.

...