Classical Test Theory

Introduction

Psychological measurement is based on the responses of participants to stimuli that may have been presented by the psychologist or, when an observation study is done, may have occurred in a natural setting. In this chapter, we refer to process of obtaining the measurements as a test. Psychologists are well aware that, although responses to stimuli may reflect the processes she expected the stimuli to elicit, the responses often will also reflect processes she would have preferred not to elicit. Therefore individual differences in the observed measurements must be conceptualized as reflecting multiple sources of variance. A statistical theory of psychological measurement formalizes the idea that observed measurements reflect multiple sources of variance by using a statistical model for the observed measurements. Such models include the classical true score model (CTSM), extensions of the CTSM such as the parallel, tau-equivalent, and congeneric test models, the generalizability theory model, strong true models such as the binomial and compound binomial models (Lord, 1965) and the multinomial and compound multinomial models (Lee, 2005), the common factor model, and item response theory models.

The determination of which statistical theories of psychological measurement constitute classical test theory (CTT) is to some degree arbitrary. Perhaps most, if not all, experts in test theory would agree that the CTSM and its extensions are classical test theories and most, if not all, would agree that strong true score theories and item response theory are not classical test theories, but rather should be classified as modern test theories. No doubt arguments would arise about whether generalizability theory and common factor analysis are classical or modern test theories. In this chapter CTT will comprise the CTSM and its extensions and as well as generalizability theory. Also, because there are connections between the CTSM and common factor analysis, elements of common factor analysis will be introduced. In addition equating and validity will be covered, albeit briefly.

The Classical True Score Model

The CTSM is based on the common sense idea that any measurement is likely to entail some measurement error, the results of transitory processes that operate during the measurement session. For example, a student who [Page 94]takes a multiple choice test and is not confident of her answer to a question may guess as to the correct choice. On any test, momentary fluctuations in memory and attention can influence the results. Momentary fluctuation in strength of attitude may influence the response to an attitude item. It should be clear from these few examples that such processes can operate in any measurement situation. The CTSM proposes that the observed score variable (Xk) is composed of two components, the true score variable (Tk) and the error score variable (Ek):

The k subscript is used because subsequently we will consider more than one test.

Lord and Novick (1968: 28) described three different conceptions of the true score. One is the Platonic true score, ‘a naturally defined “true value” on the measurement being taken.’ A second concerns the average score a person would obtain over replications of the measurement process and is the limiting value of this average as the number of replications increases. Lord and Novick argue against each of these conceptions of the true score. In their preferred conception, for any individual being measured at any particular point in time there is a distribution of possible outcomes of the measurement, a distribution that reflects the operation of processes such as guessing and momentary fluctuations in memory and attention or in strength of an attitude. This distribution, which describes the possible outcomes of the measurement process for a particular person, is called a propensity distribution. Table 5.1 provides illustrations of propensity distributions for a hypothetical population of nine individuals measured on an 11-point scale. According to Table 5.1, person A's most likely scores are 0 or 1, but a score as high as a 7 is possible. Person I's most likely scores are 9 or 10, but a score as low as 3 is possible. Similar description can be made for the other seven individuals.

...