Entry
Reader's guide
Entries A-Z
Subject index
Reliability
Reliability refers to the consistency and stability of research results and is one of two foundational elements (the other being validity) in conducting rigorous research. Reliability assesses the extent to which the results and conclusions drawn from a case study would be reproduced if the research were conducted again. Reliability in case study research is normally addressed through three techniques: (1) triangulation, (2) interrater reliability, and (3) an audit trail.
Conceptual Overview and Discussion
The concept of reliability is associated with positivist research and addresses the reproducibility of results. By contrast, validity assesses the accuracy of results. The goal of reliability is to minimize bias and error in the collection and analysis of data to the point that the same results and conclusions would be reached if the research were conducted again.
A common example of reliability is the task of weighing oneself on a bathroom scale. If repeated attempts indicate the same weight, the scale can be said to be reliable. Note that a reliable scale is not necessarily an accurate one: Even though the scale gives a consistent measure, it may indicate a weight that is consistently higher or lower than your actual weight. Thus, reliability can exist without validity, but not vice versa. Put another way, reliability is a necessary but not sufficient condition for validity.
Consistency and stability are two dimensions of reliability. Consistency refers to the degree to which the results can be independently re-created within an acceptable margin of error and is a form of measurement error. Consistency can be thought of as the level of variability in the method or instrument of measurement. Stability refers to the degree to which the results can be replicated independently at a later point in time and is similar to the replication of an experiment; if the same case were to be re-examined at a later point in time, would the results be the same?
As the use of case studies has gained acceptance within the positivist community, concepts of rigor such as reliability have been increasingly applied to the methodology. However, the importance of reliability in case studies depends to some extent on the researcher's epistemological perspective. Researchers who adhere to a social constructive or interpretive research philosophy may see case studies as a way to examine a phenomenon embedded within a unique situation at a certain point in time. They may therefore conclude that evaluating reliability is inappropriate, because the research cannot be reproduced.
Application
Reliability in case study research can be assessed by applying three commonly used techniques to address the dimensions of consistency and stability: (1) interrater reliability, (2) triangulation, and (3) an audit trail. These techniques are discussed next in the larger context of consistency and stability.
Consistency
There are two components to consistency: equivalency and internal consistency.
Equivalency
Equivalency is concerned with consistency of observation at a point in time. Case study research is susceptible to error in observation, in particular when a single researcher performs the observation and analyzes the data. In case study research the researcher can be viewed as part of the measurement process. Just as a physical instrument may have error in measurement, so too can an individual in observing or in applying coding or categorization to the qualitative data, introducing bias that impacts reliability. Addressing equivalency requires that steps be taken to minimize the measurement bias of the researcher.
Equivalency can be addressed through the use of multiple researchers who collect and/or analyze the data. Multiple researchers reduce the overall error in measurement by allowing triangulated observations and data analysis that minimizes the error of any one observer. A technique for measuring the equivalence of the researchers' analyses—interrater reliability—measures the degree to which two or more researchers agree on the application of a judgment scale or coding process. Several approaches to interrater reliability exist, such as kappa statistics (e.g., Cohen's kappa, Fleiss's kappa), correlation coefficients (e.g., Pearson's rho, Spearman's rho), and intraclass correlations. The appropriateness of the individual approach depends on the type of measurement desired.
A common example of an attempt to achieve equivalency is the use of multiple judges during Olympic ice skating competitions. The score assigned to a figure skater's performance is determined by a human judge, whose observations and ratings are potentially influenced by a wide variety of factors—different interpretations of the rules, the judge's country of origin, the style of music being played during the skater's performance, political considerations, and so on. These factors introduce bias and error into the judgment. The use of multiple judges is designed to counterbalance the bias and error introduced by these factors acting on each individual judge.
Internal consistency
Internal consistency refers to the uniformity among similar data points thought to be measuring the same construct. Unlike equivalency (in which the measurement method or instrument introduces potential bias), in internal consistency data are the potential source for bias and error. For example, suppose a manager is interviewed for his perspective on why the CEO has just been fired. It is possible that the individual simply does not know the true reason, or he or she may have a perspective bias that resulted in inaccuracies in the data collected from the interview.
In case study research internal consistency can be increased by collecting data from multiple sources and by using different types of data, an approach referred to as triangulation. Triangulation allows for more confidence in the value of data because the data are derived from multiple perspectives. Triangulation can include the use of multiple sources, such as interviewing individuals in multiple departments or at varying levels of management (line workers, supervisors, middle management, etc.) or the use of multiple data types (e.g., public documents, such as newspapers, and internal documents, such as memos and e-mails). The measurement principle behind triangulation is that the less reliant the data set is on a single type of data or a single source of data, the more likely that independent researchers would be able to recreate or re-establish the order of occurrence, the degree of influence, or the attitudes and opinions concerning organizational events or characteristics from the past.
Consider the different perspectives of documents generated external to an organization compared with documents generated internally. Documents that are external to the organization—newspaper or magazine articles, government reports, or industry-based promotional material—provide an external representation of facts, figures, and interpretations of events that are gene rally understood and widely available. On the other hand, documents that are internal to the organization—memos, committee or board meeting minutes, company e-mails, or other correspondence—provide internal representations of facts, figures, and interpretations of events that an individual or organization may not necessarily want a general audience to know. Examination of both internal and external documents allows researchers to view data points from multiple perspectives, and this can minimize the bias from any one individual data source.
Stability
Stability represents the consistency of results obtained over repeated measurements and is often measured through test–retest procedures in which a variable is measured at two points in time and then compared to determine whether similar results are generated. In case study, stability depends on whether the case study time line, sequence of events, and changes in the variables under study and their interrelationships across time are repeatable. Because so much of the analysis in qualitative research methods such as case studies relies on researchers gathering, documenting, and inferring variable measurements across multiple data sources, it is vital that the specific process of getting from the raw data to the final evaluations or measurements is made explicit. The absence of an explicit description of the process makes replication by an independent researcher impossible.
An important technique for addressing replication in case study research (and therefore demonstrating the potential for stability) is the audit trail—the documentation of the research process, including how and why the data were collected; how the data were analyzed; and any other decisions or considerations related to the data, the results, or the conclusions that were drawn. Such documentation provides enough detail that another researcher can examine the data collection and analysis process and not only understand what the researcher did and why but also be able to reach conclusions similar to the original researcher's. Even if the nature of the study does not allow a literal replication, the documentation provides a trail that allows for the research—from data collection to conclusion—to be logically replicated.
Critical Summary
Reliability assesses the reproducibility of results and conclusions. Reliability in case study research requires paying attention to both consistency (equivalency and internal consistency) and stability.
There are several techniques researchers can apply to increase the reliability of their research. Using multiple researchers and interrater reliability techniques counterbalances the biases that may be evident when an individual researcher makes observations or analyzes data. Triangulation within and across data sources addresses the potential threat to reliability caused by a lack of internal consistency among data points. Finally, stability can be addressed by documenting the research process so that an independent third party can reproduce the research process from data collection to conclusions.
Further Readings
- Case Study Research in Anthropology
- Before-and-After Case Study Design
- Agency
- Abduction
- Action-Based Data Collection
- Activity Theory
- Case Study and Theoretical Science
- Analytic Generalization
- ANTi-History
- Case Study Research in Business and Management
- Blended Research Design
- Alienation
- Bayesian Inference and Boolean Logic
- Analysis of Visual Data
- Actor-Network Theory
- Chicago School
- Audience
- Case Study as a Teaching Tool
- Case Study Research in Business Ethics
- Bounding the Case
- Authenticity and Bad Faith
- Bricoleur
- Anonymity and Confidentiality
- ANTi-History
- Colonialism
- Authenticity
- Case Study in Creativity Research
- Case Study Research in Education
- Case Selection
- Author Intentionality
- Case-to-Case Synthesis
- Anonymizing Data for Secondary Use
- Autoethnography
- Constructivism
- Concatenated Theory
- Case Study Research in Tourism
- Case Study Research in Feminism
- Case-to-Case Synthesis
- Case Study and Theoretical Science
- Causal Case Study: Explanatory Theories
- Archival Records as Evidence
- Base and Superstructure
- Critical Realism
- Conceptual Argument
- Case Study With the Elderly
- Case Study Research in Medicine
- Case Within a Case
- Contentious Issues in Case Study Research
- Chronological Order
- Audiovisual Recording
- Case Study as a Methodological Approach
- Critical Theory
- Conceptual Model: Causal Model
- Collective Case Study
- Case Study Research in Political Science
- Comparative Case Study
- Cultural Sensitivity and Case Study
- Coding: Axial Coding
- Autobiography
- Character
- Dialectical Materialism
- Conceptual Model: Operationalization
- Configurative-Ideographic Case Study
- Case Study Research in Psychology
- Critical Incident Case Study
- Dissertation Proposal
- Coding: Open Coding
- Case Study Database
- Class Analysis
- Epistemology
- Conceptual Model in a Qualitative Research Project
- Critical Pedagogy and Digital Technology
- Case Study Research in Public Policy
- Cross-Sectional Design
- Ecological Perspectives
- Coding: Selective Coding
- Case Study Protocol
- Closure
- Existentialism
- Conceptual Model in a Quantitative Research Project
- Diagnostic Case Study Research
- Decision Making Under Uncertainty
- Ideology
- Cognitive Biases
- Case Study Surveys
- Codifying Social Practices
- Families
- Contribution, Theoretical
- Explanatory Case Study
- Case Study Research in Tourism
- Deductive-Nomological Model of Explanation
- Masculinity and Femininity
- Cognitive Mapping
- Consent, Obtaining Participant
- Communicative Action
- Formative Context
- Credibility
- Exploratory Case Study
- Case Study With the Elderly
- Deviant Case Analysis
- Objectivism
- Communicative Framing Analysis
- Contextualization
- Community of Practice
- Frame Analysis
- Docile Bodies
- Inductivism
- Ecological Perspectives
- Discursive Frame
- Othering
- Complexity
- Critical Pedagogy and Digital Technology
- Comparing the Case Study With Other Methodologies
- Historical Materialism
- Equifinality
- Institutional Ethnography
- Healthcare Practice Guidelines
- Dissertation Proposal
- Patriarchy
- Computer-Based Analysis of Qualitative Data: ATLAS.ti
- Cultural Sensitivity and Case Study
- Consciousness Raising
- Interpretivism
- Experience
- Instrumental Case Study
- Pedagogy and Case Study
- Ethics
- Pluralism and Case Study
- Computer-Based Analysis of Qualitative Data: CAITA (Computer-Assisted Interpretive Textual Analysis)
- Data Resources
- Contradiction
- Liberal Feminism
- Explanation Building
- Intercultural Performance
- Event-Driven Research
- Power
- Computer-Based Analysis of Qualitative Data: Kwalitan
- Depth of Data
- Critical Discourse Analysis
- Managerialism
- Extension of Theory
- Intrinsic Case Study
- Exemplary Case Design
- Power/Knowledge
- Computer-Based Analysis of Qualitative Data: MAXQDA 2007
- Diaries and Journals
- Critical Sensemaking
- Modernity
- Falsification
- Limited-Depth Case Study
- Extended Case Method
- Pragmatism
- Computer-Based Analysis of Qualitative Data: NVIVO
- Direct Observation as Evidence
- Dasein
- North American Case Research Association
- Functionalism
- Multimedia Case Studies
- Extreme Cases
- Researcher as Research Tool
- Concept Mapping
- Discourse Analysis
- Decentering Texts
- Ontology
- Generalizability
- Participatory Action Research
- Healthcare Practice Guidelines
- Terroir
- Congruence Analysis
- Documentation as Evidence
- Deconstruction
- Paradigm Plurality in Case Study Research
- Genericization
- Participatory Case Study
- Holistic Designs
- Utilitarianism
- Constant Causal Effects Assumption
- Ethnostatistics
- Dialogic Inquiry
- Philosophy of Science
- Indeterminacy
- Pluralism and Case Study
- Hypothesis
- Verstehen
- Content Analysis
- Fiction Analysis
- Discourse Ethics
- Pluralism and Case Study
- Indexicality
- Pracademics
- Integrating Independent Case Studies
- Conversation Analysis
- Field Notes
- Double Hermeneutic
- Postcolonialism
- Instrumental Case Study
- Processual Case Research
- Juncture
- Cross-Case Synthesis and Analysis
- Field Work
- Dramaturgy
- Postmodernism
- Macrolevel Social Mechanisms
- Program Evaluation and Case Study
- Longitudinal Research
- Decision Making Under Uncertainty
- Going Native
- Ethnographic Memoir
- Postpositivism
- Middle-Range Theory
- Program-Logic Model
- Mental Framework
- Document Analysis
- Informant Bias
- Ethnography
- Poststructuralism
- Naturalistic Generalization
- Prospective Case Study
- Mixed Methods in Case Study Research
- Factor Analysis
- Institutional Ethnography
- Ethnomethodology
- Poststructuralist Feminism
- Overdetermination
- Real-Time Cases
- Most Different Systems Design
- Fiction Analysis
- Interviews
- Eurocentrism
- Radical Empiricism
- Plausibility
- Retrospective Case Study
- Multimedia Case Studies
- High-Quality Analysis
- Iterative Nodes
- Families
- Radical Feminism
- Probabilistic Explanation
- Re-Use of Qualitative Data
- Multiple-Case Designs
- Inductivism
- Language and Cultural Barriers
- Formative Context
- Reality
- Process Tracing
- Single-Case Designs
- Multi-Site Case Study
- Interactive Methodology, Feminist
- Multiple Sources of Evidence
- Frame Analysis
- Scientific Method
- Program Evaluation and Case Study
- Spiral Case Study
- Naturalistic Inquiry
- Interpreting Results
- Narrative Analysis
- Front Stage and Back Stage
- Scientific Realism
- Reporting Case Study Research
- Storyselling
- Natural Science Model
- Iterative
- Narratives
- Gendering
- Socialist Feminism
- Rhetoric in Research Reporting
- Number of Cases
- Iterative Nodes
- Naturalistic Context
- Genealogy
- Symbolic Interactionism
- Statistical Generalization
- Outcome-Driven Research
- Knowledge Production
- Nonparticipant Observation
- Governmentality
- Substantive Theory
- Paradigmatic Cases
- Method of Agreement
- Objectivity
- Grounded Theory
- Theory-Building With Cases
- Paradigm Plurality in Case Study Research
- Method of Difference
- Over-Rapport
- Hermeneutics
- Theory-Testing With Cases
- Participatory Action Research
- Multicollinearity
- Participant Observation
- Hybridity
- Underdetermination
- Participatory Case Study
- Multidimensional Scaling
- Participatory Action Research
- Imperialism
- Polar Types
- Over-Rapport
- Participatory Case Study
- Institutional Theory, Old and New
- Problem Formulation
- Pattern Matching
- Personality Tests
- Intertextuality
- Quantitative Single-Case Research Design
- Re-Analysis of Previous Data
- Problem Formulation
- Isomorphism
- Quasi-Experimental Design
- Regulating Group Mind
- Questionnaires
- Langue and Parôle
- Quick Start to Case Study Research
- Relational Analysis
- Reflexivity
- Layered Nature of Texts
- Random Assignment
- Replication
- Regulating Group Mind
- Life History
- Research Framework
- Re-Use of Qualitative Data
- Reliability
- Logocentrism
- Research Objectives
- Rival Explanations
- Repeated Observations
- Management of Impressions
- Research Proposals
- Secondary Data as Primary
- Researcher-Participant Relationship
- Means of Production
- Research Questions, Types of Retrospective Case Study
- Serendipity Pattern
- Re-Use of Qualitative Data
- Metaphor
- Rhetoric in Research Reporting
- Situational Analysis
- Sensitizing Concepts
- Modes of Production
- Sampling
- Standpoint Analysis
- Subjectivism
- Multimethod Research Program
- Socially Distributed Knowledge
- Statistical Analysis
- Subject Rights
- Multiple Selfing
- Spiral Case Study
- Storyselling
- Theoretical Saturation
- Native Points of View
- Statistics, Use of in Case Study
- Temporal Bracketing
- Triangulation
- Negotiated Order
- Storyselling
- Textual Analysis
- Use of Digital Data
- Network Analysis
- Temporal Bracketing
- Thematic Analysis
- Utilization
- One-Dimensional Culture
- Thematic Analysis
- Use of Digital Data
- Visual Research Methods
- Ordinary Troubles
- Theory, Role of
- Utilization
- Organizational Culture
- Theory-Testing With Cases
- Webs of Significance
- Paradigm Plurality in Case Study Research
- Utilization
- Within-Case Analysis
- Performativity
- Validity
- Phenomenology
- Practice-Oriented Research
- Praxis
- Primitivism
- Qualitative Analysis in Case Study
- Qualitative Comparative Analysis
- Quantitative Single-Case Research Design
- Quick Start to Case Study Research
- Self-Confrontation Method
- Self-Presentation
- Sensemaking
- Sexuality
- Signifier and Signified
- Sign System
- Simulacrum
- Social-Interaction Theory
- Storytelling
- Structuration
- Symbolic Value
- Symbolic Violence
- Thick Description
- Writing and Difference
- Loading...