Kappa Coefficient of Agreement

Development of κ

A simple, logical coefficient of agreement between two raters is the observed proportion of subjects the raters placed into the same category. This is called the observed proportion of agreement, denoted pa. Table 1 is a contingency table containing hypothetical data that depicts the proportion of subjects and the categories in which each rater placed them. For example, 4% of subjects were placed in Category C by Rater 1 and in Category A by Rater 2. Based on Table 1, the proportion of agreement, which is the sum of the proportions along the main diagonal given by $p_a={\displaystyle {\sum }_{i=1}^k{p}_{ii}}$, is .70. Therefore, the raters agreed and placed 70% of subjects into the same category.

Although the proportion of agreement is a simple and logical coefficient of agreement, it has been criticized by Cohen and others as being insufficient. The inadequacy stems from the idea that raters will have a certain level of agreement by chance alone, and pa does not take that into consideration. Others before Cohen developed their own corrections for chance agreement; however, Cohen’s correction is one of the few that involves the use of marginal proportions in its calculation with the assumption that each rater’s marginal proportions are specific to that rater and not common across raters. To be exact, the expected proportion of chance agreement, denoted pc, is given by $p_c={\displaystyle {\sum }_{i=1}^k{p}_{i.}\cdot p_{.i}}$, where κ is the number of categories, pi. is the proportion of subjects Rater 1 put into Category i, and p.i is the [Page 916]proportion of subjects Rater 2 put into Category i. Thus, Cohen’s expected proportion of agreement by chance is the sum of the product of marginal proportions for each category. Using the data in Table 1, pc = 0.2508. This is interpreted to mean that by chance alone, it is expected that the raters will agree on approximately 25% of the ratings. Cohen’s κ uses both pa and pc. More specifically, the formula for κ is given by $\kappa =\frac{p_a-p_c}{1-p_c}$. Proper use of κ requires the following assumptions set forth by Cohen: (a) independence of subjects; (b) nominal, independent, mutually exclusive, and exhaustive categories; and (c) independence of raters. Assuming that these assumptions have been satisfied for the data in Table 1, κ = 0.60. This value represents the proportion of agreement between the raters after the removal of chance agreement.

...