Intercoder Reliability Techniques: Percent Agreement

Calculating Percent Agreement

The formula for calculating percent agreement is simple and straightforward, and it is by far the easiest reliability technique available to researchers. Percent agreement (PA) is equal to the number of agreed upon ratings (NA) divided by the sum of the cases with agreements and the cases with disagreements (ND), multiplied by 100 to arrive at a percent,

$$PA=N_A{N}_A+N_D\times 100.$$

The following example of coded data between two independent judges is used to motivate the formula for calculating percent agreement. Two independent judges coded for the presence (“1”) or absence (“0”) of a given behavior:

In this present example, there are six cases where Coder A and Coder B are in agreement and four cases of disagreement. Thus, the percent agreement for Coder A and Coder B is 60%,

$$PA=66+4\times 100=60\%.$$

Using the formula and the current example, it is evident that calculating percent agreement among two or more independent judges is a straightforward and simple endeavor. Despite its ease, researchers should be also be aware of its other strengths and shortcomings.

Strengths and Weaknesses of Percent Agreement

Percent agreement is a valuable and convenient tool for assessing how well two independent judges consistently identify a variable or set of variables. It satisfies the criterion of an adequate measure of reliability by using two independent judges. In other words, percent agreement values are not biased by the judges used in the coding of variables. However, beyond satisfying the requirement of two independent judges, its use is rather limited.

Although percent agreement is a useful baseline assessment for independent judges’ coding, there are a number of weaknesses of which a young and seasoned research must be aware. First, percent agreement does not account for agreement by chance in its calculation. It is possible that a specific percent agreement between two independent judges is reached by chance alone, even if the two [Page 752]judges were blindfolded. Without parsing out agreement due to chance alone, a researcher’s claim that two independent judges reached an acceptable threshold is rendered meaningless, especially if one wishes to make comparisons to other empirical studies on a similar topic with a similar methodological approach.

...