Criterion Validity

Concurrent Validity

When a developer designs an instrument intended to measure any particular construct (say, intelligence), one of the most straightforward ways to begin establishing its validity is to conduct a concurrent validity study. The basic task is to identify an available instrument (say, the Stanford-Binet Intelligence Scale), the validity of which has already been established, that measures the same construct or set of constructs as the new instrument. The second job is for the developer to identify a large, ideally random sample of people who are appropriate for the purposes of the instrument. Next, data are collected for both instruments from every person in the sample, generally on the same day. In many circumstances, it is considered important to collect the data in a counterbalanced fashion in order to control for practice effects. Fourth, a correlational technique appropriate to the scale of measurement is applied to the pairs of scores (e.g., Pearson product-moment correlation, Spearman rank correlation). Finally, the developer compares the obtained correlation or correlations with those of the established instrument and other similar instruments collected under similar circumstances. To the extent that the results compare favorably with other available validity coefficients (as they are called), the developer has begun the lengthy process of establishing the validity of the new instrument.

In the case of norm-referenced instruments, a second analysis of the data is in order: the comparison of the mean scores for the two instruments. Although the validity coefficient, if sufficiently high, establishes the fact that the instruments measured similar constructs, at least in the current sample, it does not indicate that the new instrument is unbiased. If the standard score means are significantly different statistically, then the two instruments will lead to different decisions in practice. For instance, if the instruments both measure intelligence, and the IQ cutoff score for mental retardation is 70, a substantial number of individuals will be misdiagnosed by the new instrument. If the mean IQ is too high compared with the established intelligence test, then a considerable number of Type II errors will be made. If the opposite circumstance exists, then Type I errors are the problem. Commonly, the source of this problem is attributed to the unrepresentativeness of the new instrument's norm group.

...