Essay Items

Human Scoring

Human raters are used to evaluate an essay’s quality by assigning a score associated with response characteristics that are outlined in a scoring rubric. A scoring rubric provides predefined descriptive scoring schemes that are developed by substantive experts to guide the analysis of students’ written responses. The assumption of employing predefined scoring schemes is that the evaluation of written responses becomes less subjective and provides greater consistency in ratings.

In general, there are two types of rubrics that are commonly used in human scoring: holistic and analytic. Holistic rubrics provide the descriptions of abilities, skills, and proficiencies that examinees are expected to demonstrate at a particular score level. Analytic rubrics are more specific and break down the characteristics of each score into several components, allowing raters to itemize and define the strengths and weaknesses of the responses. Scoring with holistic rubrics generally takes less time than scoring with analytic rubrics, but analytic rubrics provide more detailed individual-level scoring criteria and feedback. Although analytic rubrics tend to provide more fine-grained feedback to students, one concern is that the scoring dimensions tend to be highly correlated with one another. For examples of holistic and analytic rubrics, readers can refer to “Designing Scoring Rubrics for Your Classroom” by Craig A. Mertler and Educational Assessment of Students by Anthony J. Nitko.

In high-stakes testing programs, implementation of human scoring involves three major steps: training, calibration, and operational scoring. The first step requires raters to carefully read the rubric to gain familiarity and to examine sample essays that correspond to each score level. Each sample essay represents various aspects of the rubric for the assigned score, which assists raters with better familiarizing themselves with the basics of the scoring scheme. Then, raters are assigned a set of prescored responses, with the score withheld from the rater, to evaluate individually.

...