Inter-Rater Reliability

Inter-rater reliability, which is sometimes referred to as interobserver reliability (these terms can be used interchangeably), is the degree to which different raters or judges make consistent estimates of the same phenomenon. For example, medical diagnoses often require a second or third opinion. Competitions, such as judging of art or a figure skating performance, are based on the ratings provided by two or more raters. Researchers might have raters assigning scores for degree of pathology in an individual or type of verbal response in a study examining communication. In the area of psychometrics and statistics, reliability is the overall trustworthiness of a measure. Common terms to describe reliability include consistency, repeatability, dependability, and generalizability. High reliability is achieved if similar results are produced under consistent conditions. For example, measuring an adult’s height is often very reliable because the method of measuring height is consistent. This entry reviews the importance and types of reliability, details methods for calculating inter-rater reliability, and discusses how to choose a method of calculation.

Although there are many different types of reliability methods (including inter-rater reliability, which is described in detail in this entry), the [Page 863]goal of estimating any type of reliability is to determine how much of the variability in scores (or ratings) is due to errors in measurement and how much is due to the variability in true scores. A true score is the replicable feature of the concept or phenomenon being measured. Errors of measurement are components of the observed score that reflect uncertainty of the true score. Errors of measurement include systematic error as well as random error. This simple equation represents the conceptual breakdown of this relationship:

$$\text{Observed test score}=\text{true score}+\text{error score}$$

Values of reliability coefficients are generally reported as correlational indices and range from .00 (all errors) to 1.00 (no error with perfect reliability). Highly reliable scores (i.e., those closer to 1.00) are accurate and reproducible.

The type of reliability often used in various disciplines and professions such as anthropology, education, marketing, medicine, psychology, sports, and even the arts is inter-rater reliability. Although rating scales can take many forms, they typically require the rater to make a subjective judgment about some characteristic of an object by assigning it to some point on a scale defined in terms of that characteristic. Thus, reliability or the consistency of the rating is of utmost importance because the results should be generalizable and not be the idiosyncratic result of one person’s judgment. Stable characteristics that are clearly defined for the raters are the primary contributors to inter-rater reliability. Inconsistency of inter-rater reliability can happen due to many factors. First, the factors might be temporary, such as rater motivation, health issues (e.g., severe headache), or fatigue. Second, the factors contributing to inconsistency may be labeled as specific; examples include not comprehending the task or fluctuations in memory or attention. Third, some aspects of the situation may interfere with careful rating, such as environmental noise or some type of major disruption during the task. Finally, chance (e.g., luck, guessing) may play into the consistency of the ratings.

...