Validity Generalization

Validation Studies and Situational Specificity

The higher the validity coefficient, the more accurate the decisions made as a result of scores on the measure. Therefore, it is in the best interest of practitioners and researchers to use measures that have sufficient validity evidence. In addition, legal and professional guidelines may suggest the necessity for conducting validation studies. It was previously believed that local validation studies were required to determine the validity of using a measure. This meant that validation studies had to be conducted at each separate location or situation in which a measure was used. This was necessary because the view at the time was that there was something unique about each location or situation in which a test was used that caused a measure to be valid in one location but not in another location. This is known as the situational specificity hypothesis. However, conducting validation studies is time-consuming and costly. Furthermore, depending on the location, the test user may only have a small sample size from which to collect data. This small sample size affects the variance of validity coefficients across situations due to random sampling error. In fact, researchers have documented that variations in validity coefficients across locations are primarily due to studies having small sample sizes and have nothing or little to do with the characteristics of, or differences across, locations.

Meta-Analysis as a Tool

To address these sampling, measurement error, and range restriction issues, researchers can employ meta-analysis. Meta-analysis is a set of research synthesis methods, which combines the results of several primary studies leading to a cumulated sample size that substantially mitigates the effects of random sampling error. Furthermore, psychometric meta-analysis allows for the correction of errors due to range restriction in predictors and measurement error. By increasing the sample size and correcting for error, meta-analysis provides an estimate of the mean population validity and the variance in population validity. If the results of the meta-analysis indicate that 90% or more of the validity coefficient estimates are above 0, the measure is said to demonstrate validity generalization. This eliminates the need for costly, time-consuming, and underpowered local validation studies.

Special Considerations

This section offers several considerations in validity generalization. First, if one is correcting for [Page 1783]measurement error when determining validity, one will likely only correct the validity coefficient for measurement error in the criterion. This is because the test, with its inherent measurement error, is being used in the screening. Second, in order to use validity generalization evidence, one should consider whether the test and criterion included in the meta-analysis are comparable to fit one’s needs. For instance, consider this scenario: A school principal wants to use a measure of conscientiousness to predict the likelihood that a teacher will turnover. In an effort to determine whether the measure is a valid predictor of turnover, the principal locates a meta-analysis concerning conscientiousness. The meta-analysis indicates that conscientiousness has a validity coefficient of .35; however, the criterion variable in the meta-analysis was job performance, not turnover. Therefore, this meta-analysis does not provide evidence for the principal’s objective. If the meta-analysis had used turnover as the criterion but had used a measure of personality (e.g., locus of control) that measured something meaningfully different than conscientiousness, then the evidence is also not useful. Lastly, one needs to determine whether the population in the meta-analysis suits one’s needs. For instance, if a meta-analysis includes only studies with individuals in primary school, but the test user wishes to use the measure with teens, it is not clear whether the meta-analytic evidence is applicable to the test user’s needs.

...