Context

The concept of context figures centrally in all evaluation theories, and the challenges of context are inescapably present in all evaluation practice. Yet, the [Page 83]meanings of context, what dimensions of it are most important, and its role in practice vary substantially across different genres of evaluation. Moreover, the idea of just what context is—how it is conceptualized in various evaluation theories and how it is engaged in diverse evaluation practices—fundamentally differentiates one evaluation approach from another.

Broadly speaking, context refers to the setting within which the evaluand (the program, policy, or product being evaluated) and thus the evaluation are situated. Context is the site, location, environment, or milieu for a given evaluand. It is an enormously complex phenomenon. Many evaluands are implemented in multiple contexts; for example, in multiple sites, which always differ in some important ways. Most contexts have multiple layers or levels, as in classrooms within schools within districts within communities within states. Perhaps most challengingly, contexts have multiple strands or dimensions, all of which can be intertwined in important ways with the character and quality of an evaluand. These dimensions include (a) the descriptive and demographic character of a setting, in terms of the numbers, characteristics, and diversity of people who inhabit it; (b) the material and economic features of a setting, in terms of the quantity and quality of its physical features (buildings, gathering spaces, resources such as books and technology), along with other indicators of material wealth or scarcity; (c) the institutional and organizational climate in a setting, in terms of the character of the organization (agency, public institution, private business) that is administering or implementing the evaluand—its norms, decision-making structures, climate features, and so forth; (d) the interpersonal dimensions of a setting, in terms of the nature of interactions that take place and the norms that frame and guide relationships; and (e) the political dynamics of a setting, particularly in terms of contested issues and interests and in terms of power, influence, and privilege.

Context, in all of its complexity, is variously engaged in evaluation theory and practice. In experimentalist evaluation, which seeks primarily to address the causal effects of a treatment on desired outcomes, context is viewed as a source of influence to be controlled. Randomization and matched comparison groups are strategies expressly designed to hold constant the contextual influences on outcomes so that observed influences can be more confidently attributed to the treatment alone. To illustrate, in an experimental evaluation of a teen pregnancy program implemented in four northeastern cities, site variations such as the demographic diversity of program participants, the organizational characteristics of the implementing agencies, and the political history of adolescent sexuality policies in each city would be viewed as potentially important influences on desired program outcomes and, therefore, influences to be controlled. If such contextual characteristics are similar in the experimental and control or comparison groups, then observed differences in outcomes can be more confidently attributed to the treatment program.

In realist and some theory-oriented approaches to evaluation, context is viewed as an inevitable and thus a rich source of explanatory influences on desired program or performance outcomes. These evaluation approaches seek causal explanations of how observed changes in outcomes are accomplished or understandings of the contextualized mechanisms that underlie such changes. In these approaches, context is thus an inextricable part of causal explanation. An evaluation of a problem-focused middle school science education program, for example, would intentionally assess teacher, school, and community characteristics in all program sites, expecting programsite interactions, some of which could hold important explanatory value. Exceptionally strong outcomes in one district might be explained, in part, by the district's recent adoption of problem-based learning throughout its elementary schools, causing its children to bring some proficiency in real-world problem solving to the problem-focused science they encountered in middle school.

...