Phi Coefficient (in Generalizability Theory)

The ϕ coefficient in generalizability theory—not to be confused with the ϕ correlation coefficient used to estimate the degree of association between dichotomous categorical variables or the ϕ (λ) statistic used to estimate the dependability of test scores at various cut points—is one of two coefficients used in generalizability theory (or G theory) to estimate score dependability (which is analogous to score reliability in classical test theory). The first coefficient, which is often called the generalizability coefficient (or G coefficient), is used to estimate the dependability of scores for tests designed for relative decisions (also known as norm-referenced decisions like those typically made with standardized tests). The second, which is often called the ϕ coefficient, is used to estimate the dependability of scores on a test for absolute decisions (also known as criterion-referenced decisions like those typically made with classroom tests). This entry explains how the ϕ coefficient is calculated and how it can be used to improve the dependability of a test.

Both the G and ϕ coefficients provide an estimate of the overall dependability of the scores or as it is expressed in G theory: the proportion of universe score variance. Such dependability estimates are calculated using the following general equation:

$$\text{Dependability}=\frac{{\widehat{\sigma }}_p^2}{{\widehat{\sigma }}_p^2+{\widehat{\sigma }}_e^2}.$$

In this case, ${\widehat{\sigma }}_p^2$ is the estimated persons variance component, and ${\widehat{\sigma }}_e^2$ is the estimated error variance ([Page 1250]all variance components discussed in this entry are derived from specially adapted analysis of variance procedures—steps that are beyond the scope of this entry). Then, the dependability estimate is the ratio of estimated persons variance $\left({\widehat{\sigma }}_p^2\right)$ to the estimated persons variance plus estimated error variance $({\widehat{\sigma }}_p^2+{\widehat{\sigma }}_e^2)$.

The ϕ coefficient (also known as Φ, or the dependability coefficient for absolute decisions) in particular is used to estimate the overall dependability, or proportion of universe score variance, of a set of scores used for absolute (or criterion referenced) decisions. ϕ is interpreted on a .00 to 1.00 scale, where .00 indicates zero dependability (or zero universe score variance) and 1.00 represents 100% dependability (or 100% universe score variance).

ϕ coefficient is calculated using the G theory equation that follows:

$$\Phi (\Delta )=\frac{{\widehat{\sigma }}_p^2}{{\widehat{\sigma }}_p^2+{\widehat{\sigma }}_e^2(\Delta )}.$$

Here, Φ(Δ) is the ϕ dependability estimate for absolute error $(\Delta ),{\widehat{\sigma }}_p^2$ is the estimated persons variance component, and ${\widehat{\sigma }}_e^2(\Delta )$ is the estimated error variance for absolute (or criterion referenced) decisions. Then, the dependability estimate is the ratio of estimated persons variance $\left({\widehat{\sigma }}_p^2\right)$ to the estimated persons variance plus absolute error variance $({\widehat{\sigma }}_p^2+{\widehat{\sigma }}_e^2(\Delta ))$.

Consider a situation in which a tester wants to study the relative effects of three potential sources of error (called facets)—including persons (p), raters (r), and rating categories (c)—and the four possible interactions of those facets pr, pc, rc, and prc. Such a study could have included other facets like composition topics (e.g., two different topics), rating occasions (i.e., raters doing the scoring two different times), rater types (e.g., teachers vs. naive raters), and so forth. Based on variance components for each facet and their interactions, it is assumed in G theory that all facets (except persons) and their interactions with each other and persons can contribute to error in absolute decisions. Thus, absolute error for this example is defined

...