Building the Plane in Flight: Establishing Post Hoc Inter-Rater Reliability Coefficients in an Educational Context

Abstract

The work presented in this case study results from a study conducted in 2012-2014 examining a newly created teacher evaluation system to determine the inter-rater reliability of the classroom observation instrument. The teacher evaluation system was the result of a partnership between the school district and the university in the same city designed to improve district operation, teacher quality, and student performance through various avenues of research and development. It was through this partnership that we gained entrée into the district to complete the research presented here. The system-wide professor-in-residence developed the teacher evaluation system and its various tools. The teacher evaluation system was field-tested, and the instruments, including the classroom observation instrument, went through content validity; however, this study was the first to assess the instrument’s reliability. The study included 41 principals and assistant principals who were asked to complete ratings on four videotaped classroom lessons from teachers within the district. Four videos were chosen based on several criteria, including subject matter, grade level, and teacher quality level. The data gathered enabled an assessment of inter-rater reliability and also enabled the investigation into various calculation methods to determine inter-rater reliability and their effectiveness in an educational context. This case study outlines one method to deal with what is a common occurrence in educational research; namely, investigating the effectiveness of an instrument after the instrument is in use. The challenges in designing studies post hoc are highlighted and various aspects of conducting research with educational leaders are discussed.

locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles