Intercoder Reliability Techniques: Krippendorff’s Alpha

Reliability Data

Following Klaus Krippendorff’s Content Analysis: An Introduction to Its Methodology, 3rd Edition (2013, pp. 277–301), the canonical form of reliability data for one variable consists of a table of the categories or values c assigned to a set of nu units u, each by at least m ≥ 2 observers, coders, judges, or measuring instruments i or j.

Table 1 Reliability Data for Predefined Units

The General Form of Alpha

Agreement coefficients that aspire to be interpretable as indicators of reliability have to have two numerically distinct reference points at which reliability is perfect or absent, respectively. They are embedded in α’s definition:

1

where Do is the disagreement observed within the columns of the reliability data in Table 1, and De is the disagreement that is expected when reliability is absent, defined as the total lack of any relationship between the data and the phenomena of interest. Algebraically, when all values within the colums of Table 1 are the same but vary across them, Do = 0 and α = 1. When the observed disagreement is indistinguishable from what can be expected in the absence of any relationship to the phenomena being coded, Do = De and α = 0. α may become negative when disagreement is systematic, when observers agree to disagree, or when observers use different coding instructions. This condition has little to do with reliability. Reliable data that are worth considering should measure close to α = 1. So, the practical range of α’s values is

2

Coincidence Matrix Representations of Reliability Data

Coincidence matrices represent reliability data in terms of pairs of values c and k in the columns of Table 1. They do not preserve the references to the observers who assigned these values to units nor to the units in which they were found. They provide conceptually convenient tabulations.

...