Mixed Methods

Mixed-method evaluation involves the planned use of two or more different kinds of empirical designs or data gathering and analysis tools in the same study or project. A substantial amount of contemporary evaluation practice routinely involves a variety of different kinds of methods—structured and unstructured, quantitative and qualitative, standardized and contextually responsive. Evaluators routinely use a variety of methods because the field now accepts the legitimacy of various methodological traditions and because diverse methods enable better understanding of the complex, multifaceted, real-world social phenomena evaluators aim to understand. What distinguishes mixed-method evaluation is thus the intentional or planned use of diverse methods for particular mixed-method purposes using particular mixed-method designs.

Methods are intentionally mixed in evaluation for purposes of (a) triangulation, or enhancing the validity or credibility of evaluation findings through results from the different methods that converge and agree, one with the other; (b) development, or using the results of one method to help develop the sample or instrumentation for another method; (c) complementarity, or extending the comprehensiveness of evaluation findings through results from different methods that broaden and deepen the understandings reached; (d) initiation, or generating new insights in evaluation findings through results from the different methods that diverge and thus call for reconciliation via further analysis, reframing, or some other shift in perspective; and (e) value diversity, or incorporating a wider diversity of values and thus greater consciousness about the value dimensions of evaluation through the use of different methods that themselves advance different values. Because practice is characteristically more complex than theory, many mixed-method evaluation studies incorporate two or more of these mixedmethod purposes.

Beyond the intentional identification of purpose, mixed-method evaluation involves considered attention to the philosophical assumptions that underlie or accompany the use of social scientific methods. These assumptions—as captured in philosophical paradigms, frameworks, or mental models—pertain most importantly to views of the social world (for example, realism or constructionism), perspectives regarding the nature of social knowledge (for example, objective or value laden), and positions regarding what is most important to know (for example, generalizable causal relationships or contextual meaningfulness). The controversial issues here are, (a) When evaluators mix methods, are they also mixing philosophical assumptions? and (b) Should they? There are currently three primary stances on this issue. First, proponents of aparadigmatic stances argue that philosophical assumptions are useful conceptual tools, but they should not drive practice decisions. Rather, practical decisions about evaluation design and method should be steered by the demands of the context or by the constructs and parameters of the relevant substantive or program theory. So, within this stance, paradigms are not really relevant to mixed-method (or any other kind of evaluation) practice. Second, proponents of a dialectical stance argue in favor of intentionally mixing philosophical assumptions when mixing methods. From this perspective, philosophical assumptions do and should meaningfully influence practice decisions. Because all sets of philosophical assumptions (all paradigms or mental models) are partial and limited, more comprehensive and insightful mixes are attained via the intentional inclusion of more than one methodological tradition or framework. In this dialectical perspective, possible tensions and dissonance from different paradigmatic assumptions are welcomed as generative. Third, proponents of pragmatic stances advance an alternative, inclusive paradigm or philosophical framework within which multiple assumptions and diverse methods can comfortably reside. In this third stance, as in the first, differences in paradigmatic traditions are de-emphasized and thereby not considered either particularly beneficial or problematic in mixed-method work. Various forms of contemporary realism and pragmatism are the most popular alternative frameworks advanced within this pragmatic perspective.

...