The use of multiple coders to make evaluations and provide an assessment of some quality represents a common practice in research. For example, suppose a researcher is coding a discourse from a transcript between persons engaged in a conversation involving a conflict. A potential set of codes could be that one person’s turn in the conversation could be viewed as (a) an attack on the comments of the other person, (b) an attempt to bolster or defend one’s self, (c) a move to integrate, collaborate, or seek common ground, or (d) some other comment.

The challenge is that the person’s conversational move must be evaluated and a meaning assigned (using the coding system) by an outside observer. The coding of utterances (or any ...