Consequential Validity Evidence

Consequential validity evidence provides information about the social consequences that result from using a test for a particular purpose. Various types of evidence can be presented to provide information [Page 372]about a test’s consequential validity; these types of evidence include subgroup scores, results of test-based classification decisions (e.g., instructional or curricular differences, negative social consequences within a peer group, differences in opportunity), and errors in test use. Evidence supporting consequential validity is typically used to demonstrate how intended outcomes have been achieved, a lack of differential impact across subgroups, and the presence of positive and absence of negative systemic effects resulting from the testing program.

In 1989, Samuel Messick introduced the idea of a consequential basis for validity; since that time, there has been a great deal of scholarship produced regarding whether consequential validity evidence is needed to support the interpretation of a test’s results. Some scholars argue that since consequential evidence deals with ethical rather than measurement considerations, it should not be considered as part of the validity argument. Others argue that ethical issues should be included in the scope of validity, and consideration of the social consequences resulting from a test’s interpretation is an important ethical consideration when constructing a validity argument. The following sections of this entry outline some possible types of consequential validity evidence, then explore the main arguments forwarded by scholars in favor of including consequential validity evidence in the validity argument and by scholars in favor of considering consequences outside of the framework of validity.

Although scholarship continues to be published on both sides of this debate, in 2008, Gregory Cizek and colleagues found that just 2.5% of the 283 tests they reviewed provided this type of evidence in their validity arguments. Although consequential validity evidence has been indicated as a source of validity evidence in the Standards for Educational and Psychological Testing, these results clearly show that it has failed to gain traction as an important source of validity evidence.