Standard Setting

Although the specification of criteria in evaluation is well developed, there is much mystery associated with standard setting. The key question for standard setting is how to distinguish defensibly between evaluands that share a particular attribute or characteristic to varying degrees. Standard setting is socially desirable and is a necessity in credentialing professionals, certifying services, labeling safe products, and equitably distributing resources. Even in cases where a clear and exact standard is set, such as in setting cut scores on tests, the process is inexact, and even when sophisticated empirical strategies are used, standard setting is, at its root, a value judgment based on sociopolitical concerns. The quality of the judgment about what constitutes an appropriate standard or level of performance is determined as much by who is making the determination as by any particular method used.

Standard setting is the process used for delineating the differences among performance levels or standards. The most well-formulated methods for standard setting are in the domain of achievement testing and are a crucial part of evaluation in high-stakes situations, such as when the test score will be the determining factor regarding some reward or punishment. Examples are graduation from high school, admission to college, eligibility for scholarships, employment, and budgeting. There are a number of strategies used for standard setting in these cases. The most common are the modified Angoff, contrasting groups, and bookmark procedures.

The Angoff procedure is based on expert judgment about the probability of a minimally competent person answering a particular test question correctly. Judges are selected who are presumed to have appropriate expertise in the domain being tested to make a well-educated guess about these probabilities. The probabilities for all items on a test are summed to give a cut score (actually the average of a number of expert judges), which divides the group into those minimally competent and those not.

The contrasting groups procedure is based on a comparison of expected and actual performance among different ability groups. One begins with a group of individuals divided into ability groups based on some other factor (such as course grades, courses taken, teacher nomination, other test scores), and the relevant test is administered to these groups. Based on the differences in performance of the different ability groups (usually plotted on a graph), the score (number of correct items) that most clearly distinguishes the group becomes the cut score.

The bookmark procedure uses item-response theory and was developed by CTB/McGraw-Hill in 1996. This method orders test questions on a scale of difficulty, from easy to hard. Expert judges then determine where cut scores should be set along that scale. The judges, first individually and then in consultation with one another, bookmark the scores they believe separate levels (for example, advanced, proficient, and unsatisfactory) of performance.

The standard-setting process has come under close scrutiny in situations where distinguishing different levels of performance accurately or inaccurately makes a serious difference to individuals or groups of individuals. For example, many professions and occupations (teachers, lawyers, doctors, police officers) use examination scores to determine who will be licensed or certified to practice. Product safety is another area where standard setting is contested, for example when [Page 399]the level of polychlorinated biphenyls in farmed salmon is considered too high. The trustworthiness of the standard-setting procedure to clearly establish an appropriate cut score for the purpose at hand is critical. There have indeed been many legal challenges to the validity of cut scores.

...