Modified Angoff Method

Procedures

Two critical components are needed to conduct a modified standard setting procedure: (1) a completed test and (2) a panel of SMEs to judge the test. There are no strict guidelines as to how many SMEs are needed, but a general rule of thumb suggests somewhere between 10 and 20 SMEs should participant in a standard setting panel where they make judgments on the individual items on the test. The definition of SME can be considered subjective. Each organization should define a set of parameters to guide the selection of SMEs. For example, criteria may include a minimum number of years in practice, recognition of accomplishment, a postgraduate degree, or a leadership position.

The standard setting panel focuses on the SME providing ratings for each item, keeping in mind a subpopulation of examinees when providing the ratings. The subpopulation or referent group of interest are borderline/just passing examinees, or as originally described by Angoff, “minimally acceptable person,” now referred to as “minimally competent examinee” when applying any version of the Angoff procedure. Angoff’s original suggestion was for the SMEs to rate each item on the test, keeping in mind the minimally acceptable person. An item would receive a 1 if the minimally acceptable person would answer the item correctly, and a 0 if the respondent would answer the item incorrectly. The sum could represent the raw score of the minimally acceptable person and be used to represent the lowest acceptable passing score on the exam.

The most common variation, or the modified version of this, is to have SMEs state the probability that a minimally competent person would get each item correct. To conceptualize a probability, SMEs may be asked to consider how many out of 100 minimally competent examinees would get the item correct, and report this as a p value, or proportion. For example, if an SME felt 70 minimally competent examinees would get the item correct, the item would receive a score, now known as an “Angoff rating” of .70. Providing guidelines such as only using ratings that are multiples of 5 or 10 can be helpful. Often, multiple rounds or iterations (no more than three) are conducted in order to calibrate the SME’s ratings. In between rounds, real data may be presented, such as item difficulties (p values) for an entire sample of examinees to help SMEs to understand how the item has functioned in the past. If we consider the SME who provided an Angoff rating of .70 on an item and then reveal that in a previous examination, 50% of all examinees got the item correct (yielding a p value of .50), then that SME may want to consider lowering his or her Angoff rating. Additionally, impact information could also be provided, such as the percentage of previous examinees who would be classified as failing or passing using the first iteration of ratings.

...