Generalizability Theory

Educational researchers are often interested in making inferences from what may be considered observable to that which is unobserved. Responses to items on a multiple-choice test, an argumentative essay, and other overt behaviors (e.g., number of times a child raises her hand in a classroom) are observable. Unobserved variables on the other hand are used by educational researchers to explain patterns in observations. Intelligence, personality, aptitude, and critical thinking cannot, strictly speaking, be directly observed. Such variables refer to theoretical attributes that are at best indirectly investigated. For example, a researcher may hypothesize that differences in critical thinking (i.e., unobserved) account for why some students have higher scores than others on an assignment (i.e., observable). The extent to which it is reasonable to conclude that observed scores reflect critical thinking is a validity issue. Measurement error constrains the validity of score-based interpretations.

Measurement may be defined as the systematic assignment of numerals according to a set of rules. Measurements may distinguish mutually exclusive categories (e.g., ethnic groups), rank-order observations (e.g., high school class rank), or indicate differences in magnitude (e.g., degrees in Fahrenheit). Reliability assessment, traditionally conceived, aims to quantify the consistency of scores in a population whereas measurement error reflects random inconsistencies. Generalizability theory—hereafter referred to as G theory—provides a framework for investigating the extent to which distinct sources of error influence the precision of scores obtained from a measurement procedure.

The basic concepts of G theory, such as variance decomposition, universe scores, and facets of measurement, are introduced in this entry using a hypothetical example. This is followed by discussing simple extensions in measurement design employed within G theory, such as whether a facet is treated as fixed or random. Finally, the entry concludes by summarizing the strengths and limitations of G theory when compared to traditional approaches for assessing reliability.