Standard Setting

It is widely accepted that standard setting methods aim to distinguish between competent and incompetent examinees who sit for a test or an examination (these terms are used interchangeably). Hans Pant and colleagues describe standard setting as an umbrella term that incorporates consensual approaches of panels of experts to set discrete cut scores on continuous test performance scales.

This description regards standard setting as a consensual decision-making process, which translates the cumulative understanding of decision makers of what constitutes competence and incompetence to establish a discrete cut score on a continuous scale, aiming to separate the competent from the incompetent. However, others regard standard setting as a decision-making process aiming to establish a cut score that is the boundary between acceptable and nonacceptable performance; that is, they focus on acceptability rather than competence.

The standard setting process is required when the observed test score cannot identify whether the examinee has clearly passed or clearly failed the test (or between any other consecutive grades). Scores within that range can be regarded as borderline. For example, there is a need to determine the test score in the final examination of medical knowledge that would ensure the examinee has acquired enough knowledge to practice medicine. Similar decisions need to be made about drivers, pilots, and other students to determine whether they are eligible to progress in their program.

Although in practice the determination of cut scores is normally made by experts or other representatives of the professional communities, these panels are not the decision makers but rather standard recommenders. The standards are actually set by the authorized bodies (e.g., professional associations, academies, boards of education, state agencies) that consider the recommendations and make operative decisions. Setting standards in a systematic way is important, as it enhances the confidence of the public and other stakeholders that the decision is robust, fair, reliable, valid, and unbiased.

There is a plethora of standard setting methods. Many, but not all, methods are based on making a consensual decision. There are a few ways to classify standard setting methods; however, standard setting methods are typically based on either a panel’s judgment or on statistical techniques using test scores generated by examinees. Some of the most advanced methods utilize both. Methods can also be classified by the frame of reference used for the decision: norm-referenced or criterion-referenced methods. The norm-referenced method uses the examinees’ test scores as a reference for the decision making (normally determined proportion of passes and fails), whereas the criterion-referenced method utilizes an external reference and is independent of the examinee’s overall performance and does not generate a fixed proportion of passes or fails. Others classify standard setting methods by two categories: examinee-centered and test-centered methods. Test-centered methods determine cut scores based on the content of the test, whereas examinee-centered methods determine the cut score by perceived attributes of the examinees.

The categories of competence and incompetence may relate to any two subsequent categories (e.g., fail vs. pass; pass vs. distinction). Standard setting is based on the assumption that the test scores provide sufficient information that allows reasonable estimation whether an examinee is competent or not, in other words whether the examinee met the performance criteria to be classified within the higher category among the two consecutive categories considered.

...