Diagnostic Validity

Diagnostic validity applies to any test, measurement, or decision-making strategy that categorizes people. Also referred to as categorical validity or, more pragmatically, as the 2 × 2 table, diagnostic validity examines the relationship between how a test categorizes a subject and in which category the subject actually is Relevant categories might include, among others, HIV-positive individuals, top employment prospects, violent recidivists, child molesters, fit parents, suitable graduate students, or incompetent defendants. Validity information answers questions regarding the probability that a classification is correct, the utility of the [Page 255] test or strategy for different purposes, and how to interpret the classification. This information also solves Bayes' theorem: We often know the percentage of paranoids, say, who score positively on a test of paranoia (by administering the test to a large group of paranoids); Bayes' theorem computes the reasonableness of inferring paranoia from a positive test score. The answer requires knowledge about the incidence of paranoia and about how nonparanoids do on the test.

In this entry, test is used specially to mean any score, sign, symptom, or series of these used to categorize people. The Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV) is a manual of tests. Each diagnosis is accompanied by a test to determine if a particular subject has the diagnosis in question. For example, the test for paranoid personality is (a) the presence of a personality disorder, plus (b) the presence of at least four of seven behaviors, plus (c) the exclusion of some other diagnoses. The fact that this is a test for paranoid personality is disguised by the failure of the publishers to include the 2 × 2 table that would answer questions about diagnostic validity. The test is made to look like the definition of the disorder instead of a method of detecting who has the disorder and who does not. Before the test was codified, there must have been some other way to determine who had paranoid personality disorder and who did not, and that method was extremely unlikely to be a perfect fit with the current criteria, even if that other way was only in the imagination of the test writers.

As an example, let's assume that the DSM-IV test for paranoid personality was a very good test. (We can only assume because the actual data have not been published.) Assume that some expert clinicians carefully identified 100 people as having this disorder. Further assume that the test published in DSM-IV gave a positive result for all 100 individuals, and that of 100 randomly selected psychiatric admissions without paranoid personality disorder, only 5 tested positive with the DSM-IV criteria. That would certainly be impressive, 100 out of 100 correctly identified with the disorder, and 95 out of 100 correctly identified without the disorder.

Now for some terminology. In this case, the expert clinicians' original diagnoses constitute the gold standard, which is the method by which subjects were placed in their actual categories. The gold standard is crucial for interpreting test results, because even the best test predicts only the categories assigned by the gold standard. Thus, for example, a test of violent recidivism usually has a gold standard of rearrest as the indicator of recidivism, so the test can never be better at identifying recidivists than rearrest is, and it is obvious that there are some people who recidivate but are not caught. Furthermore, there are some people who are arrested, but incorrectly, and not because they recidivated. Understanding the gold standard is crucial to understanding what a test that categorizes people is able to achieve.

...