Utility Tests

Utility tests offer a set of criteria for judging evaluations that parallels the more traditional validity tests for experimental research. Just as validity tests look for and attempt to counteract threats to validity, utility tests look for and attempt to counteract threats to utility. In a classic article that introduced the idea of utility tests, Weiss and Bucuvalas examined the criteria that decision makers employ in judging evaluation and policy research. They found that decision makers employ both truth tests and utility tests. Truth tests are a layperson's equivalent of a researcher's validity tests. Utility tests involve additional criteria that are conceptualized from the point of view of intended users:

• Relevance. Does the evaluation relate to my needs and interests as a real-world decision maker?
• Understandability. Can I make sense of the findings, or are they presented with so much research jargon that an ordinary person cannot really understand them?
• Conclusiveness. Was some conclusion reached or judgment made that has program or policy implications, or are there so many qualifications attached to the findings that they are essentially inconclusive?
• Actionability. Can I do something with the findings? Do they provide concrete direction and reasonable recommendations?
• Political viability. Does the evaluation speak to real-world conditions and take into account political realities, or is it an “ivory tower” report?
• Fairness. Can I trust the findings? Is the evaluator credible, the report balanced?
• Utility. Overall, is the evaluation significant enough for me, as a busy decision maker, to spend time on? Is it useful to me?

Although evaluation users place a high premium on utility tests, some controversy exists within the profession about whether evaluations should be judged by utility. Those opposed argue that methodological rigor and validity should be primary. They worry that too much attention to utility will lead to weak designs. Tension can occur between “truth tests” and “utility tests,” but the evidence shows that those who commission evaluations want both. In response, the evaluation profession has adopted both utility and accuracy standards as criteria of excellence.

Involving intended users in the evaluation process has been one major approach to reducing threats to utility, as in utilization-focused evaluation. Some evaluators worry that methodological rigor may be sacrificed if nonscientists collaborate in making methods decisions. On the whole, the evidence supports the view that high-quality involvement of intended users will result in high-quality, useful evaluations. [Page 429]Decision makers want data that are useful and accurate. Validity and utility are interdependent. Threats to utility are as important to counter as threats to validity. Skilled evaluation facilitators can help nonscientists understand methodological issues so that they can judge for themselves the trade-offs involved in choosing among design options to guard against threats to both validity and utility.