Instrumentation

A large range of instruments are used in educational research and practice. Instrumentation refers to the nature and use of such instruments. Consideration of instrumentation is imperative to [Page 839]ensure that evidence has desirable psychometric properties. Measurement invariance is among the properties most important for good assessment and has received attention in the scientific literature. This entry provides a brief overview of educational instruments, discusses measurement invariance, and reviews examples and key research.

Instrumentation varies depending on the phenomena being assessed and the purpose of assessment. Questionnaires may be deployed in surveys to assess practical or subjective things. Schedules may be used to structure interviews that generate qualitative data. Tests may be used in exams to assess knowledge or skill. Scenarios may be used in performance assessments to probe skill. Instruments may be deployed verbally, in paper, online, in person, or in a mixed range of modalities. Education is expansive and eclectic, and measurement in education may also draw on instruments from other fields such as medicine, engineering, or finance.

Part of the broader notion of validity is measurement invariance, which pertains to whether instruments are measuring the same phenomenon across place and time. Instruments that have variable measurement properties across place or time are a clear threat to the validity of measurement and hence to the assessment more broadly.

By way of example, evaluation of a mathematics intervention may be confounded if the test used to assess student competence before and after the intervention is biased against females (variation across place) or is somehow easier for people postintervention irrespective of their mathematics competence (variation across time)—perhaps due to prior exposure to test format or topics. In this instance, the intervention will appear more effective than it really is because of a larger difference between preintervention and postintervention scores arising from instrument invariance and irrespective of actual competence. What has happened is a change in the instrument’s measurement properties rather than a change in the participants.

A variety of validation procedures can be used to assure instrument invariance. In terms of validation design, multitrait/multimethod evaluation is the most comprehensive, though invariance can also be tested across people and time. Common psychometric procedures include item response modeling and a host of covariance analyses such as covariance modelling, structural equation modeling, reliability replication, and even exploratory factor analysis. With sufficient validation, it is feasible and even desirable to use different but psychometrically linked instruments across place and time within the same study. What is important is having known validation and calibration properties.

There is a tradition of research on instrumentation and invariance in particular. This work has focused on building methods ensuring that instruments measure the same constructs and sustain calibration across different episodes of application. Ensuring the constancy of instrument measurement properties—the “instrumentality”—is of fundamental importance to ensure internal and hence external and consequential assessment validity.

See also Analysis of Covariance; Exploratory Factor Analysis; Interviews; Measurement Invariance; Multitrait–Multimethod Matrix; Structural Equation Modeling; Surveys; Tests; Threats to Research Validity