Ex Post Facto Designs

Experimental Research

A simple example involves giving a test to two different groups of students who were randomly assigned to conditions. One group participated in an educational program in a junior high school involving interventions to increase bystander communication in bullying situations and the other group received no educational materials on this topic. At a later date, both groups complete a simple test of knowledge about bullying and bystanders. If knowledge gain is greater for the educational group, the reason for observing an increase reflects participation in the educational program. The intervention “caused” the increase in knowledge score as measured by the test. The experimental group is compared with the control group and any differences observed are attributed to the cause associated with the experimental manipulation.

Only one set of measurement is employed and any difference between groups assumes that the difference in outcome reflects the difference in experience (intervention) of each group. Random assignment theoretically should result in experimental groups that started with the same initial value. A pretest, theoretically, would demonstrate that each group started at the same level in terms of test results. However, no pretest was administered, only a posttest. As a result, after the fact, the difference between groups is assumed to reflect the intervention received by one group.

Field Research

The same logic of differences in experience applies to naturally occurring groups that have different experiences due to events. For example, persons in a city may experience a tragic event (like a mass shooting) while such an event does not take place in a similar city. The comparison of the two cities after the event may provide some type of insight into the impact of the event. The implementation of any program provides the potential to generate evidence for places not currently using the program. The argument is that the unique distinction between groups based on experience generates differences in the measure of interest.

Examples of this kind of reasoning often involve comparing the experience, in the United States, of various states with legal changes or program implementation. The comparison is generally between places with different histories (often states, cities, or counties are compared) and differences in outcomes. The argument is that a pilot or demonstration program producing desirable outcomes in one place could be implemented in another place and produce the same outcome. Essentially, the various locations—when implementing some action—become an experimental group compared with the control group where no such program became implemented. But does the passing of a law for gun registration 10 years ago in one state necessarily permit comparison with other states without such laws when comparing levels of violence related to gun use?

...