Intercoder Reliability Standards: Stability

Reliability involves the extent to which an experiment, test, or any systematic procedure yields the same results across replicated trials under identical conditions. Mathematically, reliability is defined as the ratio of the variation between a true score and an observed score (notably, this definition is applicable to measurement variables essentially and does not apply to nonmeasurement variables). In many cases a true score cannot be directly observed and measured. In order to obtain a true score for many variables important to social scientists, one would have to remeasure the variable infinitely. While no single measurement can determine the true score exactly, the average of an infinite number of repeated measurements would yield the true score. Measurement, by definition, is not error free and ...