Observer Reliability

Features of Observer Reliability

Assessing observer reliability relies on three distinct properties: reproducibility, stability, and accuracy. Reproducibility is perhaps the most straightforward and easiest test of observer reliability. Put simply, it refers to how a set of trained coders produce identical data within an acceptable amount of error. For example, a longitudinal content analysis of major news coverage in the United States is considered to meet the reproducibility criteria when two different sets of trained coders produce similar data on identical samples. This characteristic of observer reliability is tested using the test-test procedure with two coders, and their agreement is also known as a test-retest.[Page 1119] Observer reliability is said to be stable when the data produced by independent coders do not change over the course of a study. For instance, in the aforementioned longitudinal content analysis of major news coverage, one set of trained coders’ data is stable when it does not drift over time from the original training sample. In other words, observer reliability is stable when there is an acceptable amount of change in coding within an independent judge. Observer reliability also requires a degree of accuracy, in that the margin of error for independent coders is minimized as much as possible. Returning to the discussion of the longitudinal content analysis, accuracy is said to be high when as the coded units increase, so too does the number of agreed-upon units. An accurate measure of observer reliability is high when an independent judge is consistent within a margin of acceptable error against a gold standard. Accuracy is considered the strongest feature of observer reliability; however, when designing a study, it is also important to consider reproducibility and stability.

...