Intercoder Reliability Techniques: Holsti Method

Content analysis is a widely applied research method in multiple disciplines in social science such as media studies, communication, psychology, marketing, and sociology. Content analysis [Page 741]can be applied to analyze the manifest meaning of content of various documents in an objective, systematic, and quantitative way. With the increasing access to various online media content, content analysis is gaining popularity in mass communication research in particular.

Intercoder reliability (also referred to as intercoder or interrater agreement) is an important methodological issue in content analysis. Intercoder reliability refers to the level of agreement among two or more independent coders when they use the same coding scheme to evaluate characteristics of communication messages or artifacts. Intercoder reliability has been used as a critical criterion to evaluate the validity and reliability of a content analysis research report since it can infer the reliability or reproduction level of the coding scheme. Coding schemes with a high level of intercoder reliability have the practical benefit of being used by multiple coders to conduct parts of an overall coding work. Dividing a large amount of coding work into smaller parts can expedite the coding process and make large content analysis projects manageable. Low level of intercoder reliability is considered to be an important methodological limitation in content analysis because it may indicate poor operationalization of key concepts, unreasonable categories, and insufficient coder training.

Intercoder reliability is measured by having two or more coders use the same coding scheme or categories to code the content of a set of documents (e.g., news article, stories, online posts), and then calculate the level of agreement among the coders. A general practice of testing intercoder reliability is to test a 10% representative sample of the overall data. Coders typically receive training to understand the coding scheme and then code the data independently. Different indices have been used to calculate intercoder reliability; the commonly used and discussed ones in communication and media research include Holsti’s method, Scott’s pi (π), Cohen’s kappa (κ), and Krippendorff’s alpha (α).

This entry introduces the procedure and formula of Holsti’s method of intercoder reliability, discusses the strengths and limitations of this method, and how this method can be applied and reported. Examples are provided to demonstrate the application of this method.