Item Analysis

What Is Item Analysis?

Educational and psychological assessments are used to measure a broad range of knowledge, skills, abilities (KSA), and other constructs. These assessments are used for both high-stakes and low-stakes decisions. Examples include professional certifications, university admissions, psychological testing, and formative feedback, among others. The delivery of these assessments is just as broad. Common modes include paper-and-pencil delivery, electronic scoring, online or computer delivery, and computer-adaptive formats.

Regardless of use or delivery, making inferences about an individual’s KSA and other constructs involves examining an overall summed score, or percentage correct, and assessing the validity of those scores. Making valid inferences from assessments means that the use and interpretation of the assessment lead to the proper conclusions about the individual’s KSA and other constructs. Part of making valid inferences is related to how well the items are performing related to the item’s intended purpose. This is because an individual’s item-level responses serve as more refined pieces of information than a single, overall summed score. Subsequently, if the items are measuring the KSA and constructs as anticipated, then decisions related to the assessment scores can be made with a certain confidence. If the items are not performing as intended, then those decisions cannot be made with the same confidence level.

Investigating the performance of items, and whether the items are measuring the information in which they were intended to measure, is the purpose of item analysis. Results from these analyses are used to determine the items that are able to contribute to, in an acceptable manner, the measurement of the individual’s KSA and other constructs. Additionally, item analysis determines that items might need to be revised or removed from the assessment. There are two general frameworks that can aid in an item analysis: item response theory and classical test theory. The focus here will be on item analysis under the classical test theory framework.

Why Is Item Analysis Done?

Item analysis serves several purposes. The first, already briefly introduced, relates to identifying items that contribute to the making of valid inferences from educational and psychological assessments. This will be called the inferential stage. Part of making valid inferences is related to the reliability of the assessment. Reliability is a measure of how stable an individual’s assessment scores are. Another way to think of reliability is reproducibility. If an individual were to take the assessment again, how similar the individual’s new scores would be to the first administration is the degree of reproducibility. If they are identical, the scores are perfectly reproducible and reliable. Such scores are free from measurement error. If assessment scores are not highly reproducible, they are not considered reliable and the inferences made will not be valid because they provide a measure that contains a large amount of measurement error. Reliability is therefore a necessary, but not sufficient, condition for validity. The result is that many components of an item analysis are designed to assess the degree of reliability of the scores.

...