Internal Validity

Establishing a Causal Relationship

Although accounts of the nature of causation differ somewhat between sources, the logic of causal claims in educational and psychological research can generally be understood as invoking counterfactual reasoning of the form “all else being equal, if A had not occurred, B would not have occurred.” Although it is obviously not possible to know for certain what would have (not) occurred had events and circumstances been different than they actually are, researchers often work to approximate such counterfactual conditions via carefully controlled studies. Inferring a causal connection between A and B thus requires evidence that, all else being equal, differences in A (e.g., receiving or not receiving an educational intervention) are associated with differences in B (e.g., knowledge as measured by an academic test).

The “all else being equal” clause is important, as causal inference requires researchers to be confident that, other than A, there are not any additional influences on B that could account for observed variation in B. (Such influences, if present, are often referred to as confounding variables, and a perceived relationship between A and B that is actually due to a confounding variable is referred to as a spurious relationship.) Thus, researchers attempt to experimentally and/or statistically control such potential confounds. Different study designs may achieve this control in different ways; for example, a classic two-group experimental design (sometimes also referred to as a randomized controlled trial) involves random assignment of participants into either a treatment group or a control group; the random assignment to groups, in combination with a sample size sufficiently large to control random variation, helps ensure baseline equivalence (i.e., there is nothing systematically different about the populations of participants in each of the groups). If successful, such a design ensures that the only systematic difference between the two groups is the presence or absence of A, and thus, if differences in B are observed between the two groups, the best explanation for those differences is that they were caused by differences in A.

Tight control of all conditions other than variance in A is clearly a strength in terms of isolating potential causal relationships, but it can also be a weakness insofar as it may leave researchers unclear as to the generalizability of the observed relationship to other situations (i.e., the study’s external validity). For example, it may be that A causes B only under specific conditions or only for persons with specific characteristics (i.e., there are moderating influences on the causal relationship between A and B); and within a highly controlled experimental context, there may not be sufficient variation in other factors to allow researchers to explore the extent to which causal inferences can be generalized to other settings. For this reason, it is sometimes said that the very strategies that maximize the internal validity of a study may also limit its external validity.

...