Order Effects

In educational research, an order effect occurs when the order in which research subjects participate in experimental conditions affects the outcome variable being measured. For example, a researcher may be interested in examining the relative effectiveness of two versions of an online reading program on student performance. Students in the study could first complete Program A and then complete Program B, each followed by a standardized assessment. If Program B yields greater improvement than Program A, the researcher might conclude that Program B is more effective. However, without additional evidence, this conclusion may not be valid. Improvement under Program B could have been greater for reasons other than the effects of this version of the reading program. Extraneous factors, such as increased familiarity with the online format of the program or the standardized assessment, could have resulted from completing Program A; these effects could influence the outcomes for Program B. That is, the order in which the participants received the experimental conditions may have affected the measurement outcome. The impact of order effects is important to consider in educational research because of their potential biasing effect on outcome measures and, subsequently, the conclusions drawn from a study. This entry discusses design factors that may lead to order effects in experimental research, methods for addressing and controlling for order effects on outcome measures, and the impact of order effects in questionnaire design, another method commonly used in educational research.

To make causal inferences about what factors may influence the phenomena under observation, researchers design experiments to test the relationship among the factors being studied. In a design that calls for individuals to participate in more than one experimental condition, a phenomenon called sequencing effects may arise. An order effect is one type of sequencing effect that is a consequence of the order in which participants are administered the experimental conditions. Order effects are distinct from another type of sequencing effect called a carryover effect. A carryover effect is a biasing effect that occurs when the effects of a prior experimental condition continue to influence a participant’s performance in the subsequent condition. For example, in the example previously provided in this entry, it is possible that Program A led to changes in students’ reading strategies. Performance in the second condition, Program B, could be influenced by both order effects (increased familiarity with test conditions or the assessment) and carryover effects (changes in reading strategies).

Broadly speaking, research experiments may incorporate a between-subjects or a within-subjects design. In a between-subjects design, research participants are each assigned to a different condition, and changes in outcome measures between groups are compared. Because research subjects participate in only one experimental condition, [Page 1183]order effects do not occur. In a within-subjects design, instead of assigning participants to different experimental groups, participants would be administered more than one and perhaps all of the conditions. Order effects are an issue for any experiment in which research subjects participate in more than one condition.

A common method used for addressing order effects in a research design is counterbalancing. A simple way to counterbalance is to give all research subjects all the experimental conditions, presenting each with a different order while representing all possible orders across subjects or groups. When the number of experimental conditions is small, representing all order combinations across all groups may be feasible. For example, with two experimental conditions, A and B, only two orders are possible, A–B and B–A. However, such an approach becomes more impractical as the number of experimental conditions increases and the total number of possible combinations increases as well. Counterbalanced designs can include a subset of the possible combinations. Such a design would ensure that each experimental condition is adequately represented both in the order in which it is presented and in its appearance both before and after the other experimental conditions included in the research.

...