Solomon Four-Group Design

Threats to Internal and External Validity in Pretest–Posttest Designs

Using a pretest–posttest study design can lead to issues with both external validity (the ability of a research design to generalize the results of a study to a population or to similar populations or situations) and internal validity (the ability of a research design to provide evidence of a cause-and-effect relationship between an independent variable and a dependent variable). Specifically, when researchers employ a pretest–posttest design there may be an interaction between the pretest and the treatment that may lead to different scores on the dependent variable, had no pretest been administered. For example, perhaps a researcher wishes to examine whether people score higher on a current events quiz if they are shown pictures of governmental officials in the United States. The researcher includes a pretest of several current events questions, then either shows participants photographs of officials or does not show participants photographs of officials, and finally administers a posttest. The researcher finds that participants who were shown the photographs scored higher on the current events quiz, but the researcher wonders whether higher scores were due to the combination of the pretest and the photographs or simply due to exposure to the photographs. The researcher may have trouble generalizing to all situations. Are both the pretest and exposure to the photographs necessary to improve scores on current events quizzes? If that’s the case, then results cannot be generalized to situations where a pretest is not given.

Pretest–posttest designs can also lead to concerns about internal validity, particularly the internal validity threat of testing. Let us alter the pretest–posttest design example above and say that a researcher used a single group. The researcher asked participants to take a pretest about current events, exposed participants to photographs of government officials in the United States, and then administered a posttest about current events. The researcher finds that participants scored higher on the posttest than on the pretest and concludes that showing participants photographs of officials led to higher posttest scores. However, the researcher has failed to consider a testing effect. Perhaps participants scored higher on the posttest simply because they had some practice with the questions in the pretest or had more time to think about the questions when they took the posttest.

...