Angoff Method

This entry describes the Angoff method for setting standards on educational tests and how it can be used to set valid standards on educational tests. Standard setting refers to the process used to establish cut scores on educational tests that are used to classify test takers into categories such as “pass,” “fail,” “proficient,” “advanced,” and other categories generally referred to as achievement levels. Many educational tests, such as licensure tests professionals are required to pass to become licensed and high school graduation tests that students must pass to receive a high school diploma, require these standards.

Most people in modern society have taken tests based on which the standards are set. However, it is not widely known as to how those standards were set. The most popular method is the Angoff method and its variations.

In 1971, William Angoff wrote a seminal chapter called “Scales, Norms, and Equivalent Scores” in a book on educational measurement. In the chapter, he described how test developers transform students’ responses to test items into standardized scores and how they maintain equivalence of these score scales over time.

In describing how to incorporate meaning into the score scale by setting “pass/fail” standards on the scale, Angoff described a method suggested by his colleague Ledyard Tucker. This process involved having subject matter experts (SMEs) think about the “minimally competent” test taker; that is, the test taker who “just barely” has the sufficient knowledge and skills required to pass the exam (sometimes referred to as the “borderline” candidate). The task for the SMEs was to review each test item and judge whether the minimally competent test taker would answer the item correctly. The passing score suggested by each SME is calculated by simply summing the number of items the SME predicted would be correctly answered by the minimally competent candidate and then averaging that score across the SMEs.

[Page 93]Angoff added a footnote to his description of Tucker’s “yes/no” method and suggested instead of judging whether the minimally competent test taker would or would not correctly answer the item, the SMEs could estimate the probability the minimally competent test taker would correctly answer the item. Those probability ratings could then be summed, and averaged over SMEs, to derive the passing standard. The process he suggested in that footnote became known as the Angoff method and quickly became the most popular method for setting standards on educational tests.

Like all test-centered standard-setting methods (i.e., methods where SMEs review and rate test items), the Angoff method involves several steps. These steps include (a) discussing the knowledge and skills of the minimally competent test taker, (b) reviewing the test items, (c) providing a probability rating for each item, (d) discussing all or a subset of those ratings, and (e) revising the original ratings as the SMEs regard necessary. The final cut score is based on the revised ratings in Step (e).

As a simple illustration of the Angoff method, imagine a test with 100 items. If an SME reviewed each item and estimated the minimally competent test taker would have a 0.50 probability of answering each item correctly, the SME-suggested passing score would be 50 (i.e., 0.50 × 100 items). Of course, no SME would assign the same probability value to all items because items vary in their difficulty. Thus, our example is oversimplified to illustrate how the cut score is calculated for a single SME. The final cut score would be averaged over all SMEs.

...