Bookmark Method

The bookmark method is a standard setting method used to establish one or more cut scores associated with interpretable levels of performance on an assessment. In 1995, Daniel Lewis and Howard Mitzel developed the bookmark method that became widely used in the 2000s, with a majority of states employing it to meet the requirements of the No Child Left Behind Act (in particular, requirements associated with reporting test results in terms of achievement levels). This entry describes the technical foundations of the bookmark method and how it uniquely executes activities common to many standard setting procedures.

Standard setting is necessary to systematically set one or more cut scores that separate a test scale into two or more categorical levels of achievement such as “failing” and “passing.” The bookmark method is differentiated from other standard setting methods by its use of item response theory and the ordered item booklet (OIB) as a foundation for key standard setting activities. The bookmark method continues to be widely used both internationally and for U.S. state summative assessment programs. Variations of the bookmark method, such as the Mapmark method, have also emerged in practice.

Most standard setting methods assemble a qualified panel of subject-matter experts to participate in a standardized process that includes the following three key activities:

• 1.
  the orientation of panelists to the testing program and test of interest,
• 2.
  the training of panelists to make ratings that support cut score estimation, and
• 3.
  discussion and consensus building among panelists over multiple rounds of ratings.

The bookmark method’s approach to these three standard setting activities uniquely defines the method. The primary bookmark method tool, the OIB, is assembled in print or digitally as an ordered set of scaled test items that is representative of the construct measured by the assessment of interest.

OIB items are displayed in ascending order, by difficulty, which is defined as scale location. The scale location of dichotomous item i is the score, Si, such that an examinee with ability Si has a specified probability of success, referred to as the response probability. The most common response probability employed in practice, and for discussion in this entry, is 2/3. Thus, a selected response item is located at the scale location where an examinee has a 2/3 probability of success.

Polytomous items are located at multiple scale locations, one for each positive score point. For example, a constructed response item with obtainable scores of 0, 1, and 2 is mapped on two locations—the scale scores where an examinee has a 2/3 probability of achieving at least a 1 and at least a 2, respectively. Thus, the OIB is a set of test items that represent the construct of interest, presented in order of difficulty, with dichotomous items intermingled with polytomous item score points. The OIB supports each of the three key standard setting activities listed earlier. For simplicity, let us assume panelists are setting a single “passing” cut score.

The first activity—the orientation of panelists to the testing program and test of interest—begins with training that is relatively undifferentiated [Page 217]among standard setting procedures including discussion of the nature and consequences of the testing program, the content standards, and scoring rubrics. However, training on the construct measured by the test is uniquely implemented under the bookmark method by a structured study of the items in the OIB. Panelists typically study the OIB in small groups by reviewing the test items in order, from easiest to hardest, and answering and discussing the following two questions for each

...