Systematic Error

Additive versus Correlational Systematic Error

One way of visualizing systematic error is in terms of a thermometer or a weighing machine that consistently deviates from the true value in a specific direction by a constant sum, say, as a result of an error in calibrating the zero point on the device. Such an error is a type of systematic error called additive systematic error. Additive systematic error inflates or deflates responses by a constant magnitude and might result in reduced correlations with other items. It should be noted that additive systematic error can be constant across responses and therefore has no effect on relationships, or it can be partial in the sense that the additive effect deflates or inflates responses to one end of the scale and restricts variance. Such additive error could be caused by several factors, such as leading questions, interviewer bias, unbalanced response categories, consistently lenient or stringent ratings resulting from wording or other factors, or a tendency to agree or disagree. Factors that cause responses to be consistently off in one direction across respondents lead to additive systematic error.

A more problematic form of systematic error is correlational systematic error, which consistently deflates or inflates the relationship between two or more variables. Correlational error occurs when individual responses vary consistently to different degrees over and above true differences in the construct being measured; that is, it is a result of different individuals responding in consistently different ways over and above true differences in the construct. Using the weighing machine example, if readings are off in a certain direction and [Page 1484]also in proportion to somebody's weight, then that is an example of correlational error (e.g., if the weighing machine shows an additional 5 pounds for a 100-pound person and an additional 10 pounds for a 200-pound person). An item might have correlational systematic error resulting from a common method, such as extreme response anchors (e.g., hate-love), leading to the use of middle response categories. Correlational systematic error occurs if different individuals interpret and use response categories in consistent but different ways. Examples include an item with correlational systematic error caused by a common method, such as extreme response anchors, leading to the use of middle response categories, or an item with moderate response anchors (e.g., like-dislike), leading to the use of extreme response categories. Correlational systematic errors can be caused by the use of response scales of a similar format across items, including what are referred to in the literature as method factors. For example, a certain set of response categories is employed (say, very good to very bad) and respondents interpret the categories in certain ways (very good means more or less positive for different respondents). In this scenario, the covariance across items will be, at least partially, a result of the method factor, or the use of identical response formats. Similarly, correlational systematic error might arise as a result of other aspects of the research method, such as variation in the interpretation of the instructions and the questions.

...