Observational Studies

Design and Practicality Issues

Two key design issues distinguish observational and experimental studies: (a) the method used to form the comparison groups and (b) the nature of the process that determines whether a participant receives the treatment condition or the control condition. The

[Page 705]

two-group experiment involves two essential characteristics: (a) random assignment of available participants to form two comparison groups and (b) experimenter-dictated assignment of the treatment condition to all participants in one group and the control condition to all participants in the other group. Because random assignment yields two groups that are probabilistically equivalent on all variables (before treatments are applied), a simple comparison of the group means (or medians) usually provides a meaningful estimate of the causal effect. In many situations, however, random assignment and experimenter-controlled treatment manipulation are impractical or unethical. For example, if one suspects that cocaine use by mothers causes brain damage in neonates, it would be unethical to randomly assign a sample of pregnant mothers to two groups, with one group receiving cocaine and the other (cocaine-free) group serving as the control group. Observational studies are often useful in such situations.

The observational study has neither random assignment of participants to form comparison groups nor experimenter-controlled assignment of treatment and control conditions to the groups. Rather, data are obtained from participants in nonrandomly formed groups: one that received the treatment condition and one that did not receive the treatment condition. Participants' exposure to the treatment occurs for some reason(s) other than the action of an experimenter. Suppose, for example, that two samples of mothers are identified: One sample used cocaine during pregnancy and the other did not. Although a comparison of the two samples on a measure of neonatal brain damage may seem interesting, this comparison does not necessarily provide an estimate of the causal effect. Such a comparison is often called the naive effect estimate because it almost certainly provides an invalid (biased) estimate of the causal effect.

Key Interpretational Concern

Naive estimates almost always contain bias because differences between treatment and control groups on the outcome may exist for many systematic reasons other than causal effects of treatments. Consequently, one does not know how much of this difference should be attributed to initial group differences and how much (if any) should be attributed to treatment effects. It is naive to presume that the outcome difference reflects only treatment effects.

Analytic Issues

A thorough evaluation of an observational study will include statistical analyses that go beyond naive estimation to yield better estimates of the putative causal effect of the treatments. These analyses focus on two types of bias-producing variables: overt, which have been accurately measured on each participant before treatments were applied, and hidden, which have not been so measured.

...