Computerized Adaptive Testing

Key Features of Computerized Adaptive Tests

Most paper-and-pencil test forms comprise a fixed number of test items. These fixed test forms are typically constructed by test developers to meet a set of specifications comprising a content blueprint that indicates the proportional representation of content on each test form and statistical requirements such as the target test difficulty and minimum reliability (e.g., minimum score precision per form). All examinees assigned to a particular test form see exactly the same test items—usually in the same position.

CAT replaces the paradigm of using fixed test forms with one using variable test forms that are uniquely customized for each examinee. [Page 345]Theoretically, under CAT, every individual can receive a test form uniquely designed to match his or her proficiency. One of the earliest adaptive testing paradigms was proposed in 1971 by Frederic Lord, who investigated flexilevel testing as a precursor to CAT. Fixed test booklets contained items arranged in order of difficulty. An examinee would start the flexilevel test in the midrange of difficulty and take easier items each time the examinee answered the current item incorrectly, or conversely, a more difficult item for each correct answer. This same principle underlies most CATs in operational use, today. Figure 1 provides an overview of a slightly more sophisticated (and modern) CAT algorithm.

Figure 1 A basic computerized adaptive testing (CAT) algorithm

As shown in Figure 1, the CAT is typically initiated by selecting and administering a small number of preselected test items to provide a preliminary estimate of the examinee’s proficiency score. That preliminary proficiency score is then used to select easier or more difficult test items, similar to the logic of a flexilevel test. (The actual maximum information criterion used for optimally selecting the next item is discussed later in this entry.) The selected item is then administered, a new provisional proficiency score is estimated, and another item is selected. The process continues until a fixed test length or some other designated stopping criterion has been reached—a criterion usually related to decision accuracy or to the precision of the estimated proficiency score.

Despite the seeming simplicity of the CAT algorithm shown in Figure 1, it is important to understand that the phrase computerized adaptive testing actually refers to many different types of computer-based test (CBT) delivery models and technologies. Two factors tend to distinguish most types of adaptive test: (1) the size and nature of the test units selected and (2) the actual test unit selection/test form assembly and scoring mechanisms employed. In addition, there are a number of subtle variations on the theme of item-level CAT, such as Wim van der Linden’s shadow testing, a method of ensuring extended control over the content balance and quality of each customized test form, and stratified CAT, a method that attempts to buffer the overexposure of “statistically popular” items in the item bank.

...