Decision Consistency

What Is Decision Consistency?

By definition, decision consistency refers to the accuracy in the classification of individual examinees on multiple forms or administrations of an exam. If we look at a simple example, end-of-year exams required by students in elementary and secondary education, the exam is focused on measuring whether the student is proficient or not proficient in different subject areas. Over each form or administration of the exam, two options are possible: either the examinee will be classified as proficient or the examinee will be classified as not proficient. This will create a 2 × 2 decision table of possible outcomes across the two exams (see Table 1).

Of interest in decision consistency is to measure the probability of a consistent classification. The distinction between decision consistency and other reliability measures for an exam (e.g., test–retest reliability, internal consistency) is that decision consistency assumes that the reliability of the scores themselves are not as important as the final decision. That is, the consistency in the items and the resulting test score is not being measured; [Page 476]rather, the final decision of the model to classify examinees into their categories (e.g., proficient vs. not proficient) is. Although there may be variation in the final scores, the probability of consistency of the final decision is being calculated.

The relationship between decision consistency and decision accuracy can be seen by replacing the second exam outcomes by the expected decision of the examinee of the exam based on the true score of the examinee. By looking at the relationship between the expected and observed outcomes, the probability of a consistent classification will estimate the decision accuracy of the exam.

Factors Affecting Decision Consistency

Multiple test characteristics may affect measures of decision consistency. For example, exams with more items may lead to better discrimination, which will increase decision consistency. The location of the cut point for examinee decision categories will also greatly impact the consistency of decisions. Cut points that are located closer to the center of the examinee distribution tend to have lower decision consistency. The greater the generalizability of the test scores, the greater decision consistency the exam will generate. And, as a last example, the greater the similarity between the examinee distributions of the two exams that are being compared, the greater the measure of decision consistency.

...