Case
Abstract
In 2014, I enrolled as a PhD candidate within the discipline of Public Health with the aspiration of conducting research on young children’s health and wellbeing as they transition to school. Initial review of the literature on young children’s health and wellbeing highlighted two key aspects of the field: (a) there is no shared or agreed upon definition of child health and wellbeing or how to measure or assess it, and (b) the field of child health and wellbeing crosses distinct disciplinary lines, specifically with the fields of health and education. During the initial review of the literature, it was clear that the interdisciplinary nature of the field would pose significant challenges for a systematic review of the literature. Vast theoretical and methodological differences spanning the education and health divide in early childhood research would require a review strategy that was rigorous, yet flexible enough to meet the complexities of this field.
This case study provides an account of how the Critical Interpretive Synthesis method was used to complete a systematic review of literature spanning across the disciplines of health and education. Specific attention is given to clearly outline the processes and challenges of the systematic review of diverse literature and systematic review protocols. This case outlines the step-by-step process used to undertake a rigorous systematic review of literature using the Critical Interpretive Synthesis method, and reflects on important considerations and challenges that were encountered throughout the process.
Learning Outcomes
By the end of this case, students should be able to
- Understand the challenges of completing a systematic review across interdisciplinary research and diverse research methods
- Define and describe the process for completing a systematic review of literature using the critical interpretive synthesis (CIS) method
- Employ various methods described to complete a systematic review, or add rigor to a literature review
Research Context and Overview: Young Children’s Health and Wellbeing During the Transition to School
In 2014, I enrolled as a PhD candidate in the discipline of Public Health to explore the ways in which young children’s wellbeing during the transition to school is conceptualized, operationalized, and assessed. This interest stemmed from my previous work as a kindergarten teacher who supported the transition of up to 44 children each year, and my current work in early childhood and health teacher education. The transition to school is widely accepted as an ideal point in time to assess children’s “school readiness,” and is a contested, yet widely used, concept which seeks to measure areas of child development such as academic skills, social and emotional development, and health and wellbeing. Despite the interest and prevalent use of school readiness or developmental instruments internationally for assessing a wide range of development areas including elements of health and wellbeing, these concepts are rarely defined and have even less consensus. The lack of clarity around these terms and their use is also exacerbated by the range of disciplines with interests in child health and wellbeing, such as education, health, early childhood development, psychology, economics, and social and behavioral sciences. An initial review of the literature illuminated the complexities and diversities of this field in regards to both theoretical perspectives and research designs, making a review across interdisciplinary literature challenging.
To demonstrate my understanding of the field and rigor in the review of the current state of child health and wellbeing across the transition to school literature, I chose to complete a systematic review for my PhD thesis rather than a more traditional literature review. Coming from an early childhood education background where systematic reviews are rarely used, attempting to systematically review diverse and interdisciplinary literature was a daunting task. However, through preliminary search of systematic review methods, a systematic review framework that could accommodate diverse and interdisciplinary literature, the critical interpretive synthesis (CIS) method, was located. In 2014, I began what became a yearlong process of systematic searching, analysis, and critique of the current conceptualizations of children’s health and wellbeing using the CIS method, culminating in the systematic review of the state of the literature is this area.
Throughout this case study, I will outline the step-by-step process taken in the systematic review of interdisciplinary literature, and considerations and processes that postgraduate students, early career researchers, or researchers new to systematic reviewing can employ in completing their own systematic reviews, or adding rigor to other literature review designs.
Systematic Review—A Brief Overview
The Cochrane Collaboration, a global network of independent researchers and professionals in health care, have produced rigid protocols for systematic reviews and maintain the Cochrane Database of Systematic Reviews (Bero & Rennie, 1995). As defined in the Cochrane Handbook for Systematic Reviews of Interventions, systematic reviews attempt to do two things: (a) seek to collate all evidence that fits pre-specified eligibility criteria in order to address a specific research question and (b) aim to minimize bias by using explicit, systematic methods which include transparent, procedural explanation of the search strategy, inclusion/exclusion criteria, and analysis of findings (Higgins & Green, 2011).
While systematic reviews of randomized controlled trials (RCTs) are considered “gold standard” within the systematic review literature of disciplines such as medicine and health (Robinson & Dickersin, 2002), RCTs are inappropriate or improbable in many areas of health research. This has led to a wider variety of systematic review methods being used across health disciplines. Regardless of the area of research, a systematic review of the literature can be completed; however, the process will differ considering the field of literature and research area. To begin a systematic review, the first step is to ascertain the purpose of the review and the type of literature/research to be included. There are broadly two types of systematic reviews: aggregative reviews and interpretive reviews.
- Aggregative reviews are concerned with assembling and pooling data. They are summative in nature and use a defined research question and hypothesis to synthesize empirical studies considered to be “combinable.” Aggregative reviews are used almost exclusively for the synthesis of quantitative research such as RCTs (Egger, 1997; Noblit & Hare, 1988).
- Interpretive reviews are concerned with a synthesis of research/literature that involves both induction and interpretation, allowing theories and concepts to emerge during the review. Interpretive reviews use a preliminary research question to guide the review; however, the research question may be further developed, refined, or refocused during the review process. Interpretive reviews are generally used for qualitative research. Rather than summarizing what is known, an interpretive review interprets and questions “what is known,” treating the literature itself as an object of scrutiny (Dixon-Woods, Cavers, et al., 2006; Eisenhart, 1998).
Research Practicalities
When beginning this systematic review, preliminary scoping of literature surrounding the health and wellbeing of children as they transition to school was completed to uncover key ideas, theorists, key words, and databases for systematic searching. Such preliminary searches uncovered empirical research using qualitative, quantitative, and mixed methods; numerous editorials and commentaries; case studies; program and early intervention evaluation; reviews of the literature; and government documents—in other words, extremely diverse literature. Due to the interdisciplinary nature of the field, and the diverse methodologies and theoretical underpinnings in the area of young children’s health and wellbeing during the transition to school, an interpretive systematic review framework was chosen. An interpretive design was necessary to define an explicit and transparent process for combining and analyzing literature that, according to the criteria of an aggregative review, would not be “considered combinable.”
In response to the challenges presented by the scoping searches, an interpretive review framework was sought which would allow for the synthesis of substantively disparate literature. However, methods for completing interpretive reviews have been less developed and remain a more complex and sometimes contested territory than aggregative reviews (Thomas & Harden, 2008). Despite these challenges, a variety of methods and frameworks have been published which demonstrate how interpretive reviews can use systematic and transparent protocols to produce rigorous reviews (e.g., see Dyba, Dingsoyr, & Hanssen, 2007; Popay, Rogers, & Williams, 1998; Popay et al., 2006; Thomas & Harden, 2008). Due to the complexity of creating a systematic search protocol and comparing and analyzing findings in an interpretive review, finding an existing framework to support the development and analysis of your review is a helpful and arguably necessary strategy for those new to systematic review. Given the interdisciplinary nature of the literature to be reviewed for this review, the CIS method created by Mary Dixon-Woods, Debbie Cavers, et al. (2006) was selected. Regardless of the systematic review framework chosen, the key tenet of a systematic review is that its processes are clearly defined, and continuously documented in order to ensure that the review is transparent and replicable.
Research Design—A Critical Interpretive Synthesis
The use of the CIS method allows for the production of a mid-range theoretical account of the evidence and existing theory that is neither too abstract (so as to lack applicability) nor too specific (that explanatory scope is limited). The CIS method positions a research question(s) as a compass rather than an anchor, where formative and guiding questions and outcomes are developed throughout the review process into an iteratively (continuously) defined research question. Uniquely, the CIS method offers an interpretive approach which can be applied to a “whole corpus of evidence (regardless of study type) included in a review” (Dixon-Woods, Bonas, et al., 2006, p. 2), a necessary consideration when attempting to complete a review that crosses the diverse nature of early child research and policy perspectives.
In addition to a preliminary research question(s) acting as a compass rather than an anchor, and the ability to synthesize disparate forms of literature, the most distinctive characteristic of the CIS method is its goal of being critical. As the CIS method treats the literature itself as an object of scrutiny, a systematic review using this method questions the ways in which the literature “constructs its problematics” or the nature of the assumptions that underpin the literature, and the conclusions to which it has come (Dixon-Woods, Bonas, et al., 2006). In this way, the CIS method moves beyond a simple summary of the data reported to a more fundamental critique, allowing for the questioning of taken-for-granted assumptions, and disparity within multi-disciplinary literature.
The CIS method was used as a guiding framework to produce a mid-range account of the current conceptualizations and operationalizations of child health and wellbeing during the transition to school. The next section will outline step by step how this systematic review was completed using the CIS method.
Method in Action
Step 1—Formulating the Guiding Research Question and Identifying Outcomes
To begin the systematic review using the CIS method, a preliminary research question needs to be identified to act as a compass and guide the review. The preliminary research question for this review was
- How and in what ways do traditional (non-integrated) and integrated approaches to school transition impact and support early childhood health and wellbeing?
Primary and secondary outcomes are also identified as a way to add rigor to a review (Cochrane Collaboration, 2011). Outcomes were used to determine the scope of information that would be considered meaningful to the intended audience, yet not necessarily reported in individual studies, used to facilitate the screening of papers during the review process. The primary outcome was identified as
- How is health and wellbeing assessed during the transition to school?
Secondary outcomes were identified as:
- How do different transition models impact or support children and families?
- What are the different models of school transition?
- How can we measure/assess health and wellbeing across transition?
- How can child health and wellbeing be supported across transition?
The outcomes were chosen to move from descriptive questions, to ascertain the current state of the literature, through to normative questions to guide the critique of the literature. With the guiding research question and outcomes identified, the next step in the systematic review process, systematic searching, begins.
Step 2—Searching the Literature
As with all systematic reviews, the CIS method requires a transparent account of the search protocol and findings so that the process is reproducible. Structured, protocol-driven search strategies across electronic databases are a highly utilized and effective strategy for finding “comparable” papers during systematic review. However, relying only on electronic database searches can be limiting for a systematic review which attempts to cross multi-disciplinary and/or methodological divides (Dixon-Woods, Cavers, et al., 2006).
When completing a systematic review, thought needs to be given about what types of literature or documents are to be included. The CIS method illustrates how a multi-strategy search protocol can be used to locate relevant literature from a wide variety of sources. Following the CIS method, this review used diverse search strategies such as searching websites, combing reference lists, and contact with experts in addition to systematic database searching.
To begin systematic database searching, the first step is to identify relevant electronic databases, key words, and inclusion/exclusion criteria for the searches. This is done through preliminary scoping or searching of the literature. Through preliminary scoping, the following criteria were established to begin this review’s systematic search protocol:
- Electronic databases to search: ProQuest and Informit
- Key words: (“early childhood”) AND education AND (health OR well-being) AND (“transition to school”) AND (childcare OR “integrated centres”)
- Additional criteria: peer-reviewed material from 1998 onwards
To document the process and findings of the systematic database searches, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagrams (a systematic searching tool) were used. Below (Figure 1) is the PRISMA diagram for the first systematic database search.
Figure 1. Initial search of ProQuest database.
A PRISMA diagram is used to demonstrate the systematic search procedure in the most transparent way possible. As evidenced in Figure 1, the PRISMA diagram states the
- Search strategy (key words and conclusion/exclusion criteria)
- Number of records found (7623)
- Number of records screened by title/abstract (4000)
- Number of records excluded by title/abstract and why (3858)
- Number of records screened by full text (142)
- Number of records excluded after full text screening and why (115)
- Number of records included and why (27)
As the first database search identified 7623 records, it was apparent that the search terms were too broad. In response, a secondary search of the same database (ProQuest) was completed with narrower search criteria retrieving an additional 22 records (see the web resource section at the end of this case for the complete record of PRISMA diagrams used in this systematic review). Next, a search of the database INFORMIT (see link in web resource section) was completed retrieving nine records. After the initial three database searches, the included records were comprehensively catalogued and the reference lists of each of the included records were combed to identify further relevant papers and authors. From this process, another 43 records were identified and included.
While only two database searches were identified in the initial search protocol, it was clear from the 43 relevant papers found in reference list searching that further database searching was needed. To find where the gaps in the current search strategy may be, the 43 papers found through reference list searching were analyzed. Analysis of the 43 papers revealed a gap in the literature surrounding integrated services. To address this gap, a further systematic search for integrated services literature was conducted in INFORMIT (see web resource link). From this search, four relevant papers were also included.
Despite continued analysis of the 43 papers not found through initial database searches, there remained 10 papers for which a gap in the search strategy remained unidentifiable. To address this gap, two further searches were completed in the databases ERIC and Web of Science, selected due to their relevancy to the research question and outcomes. These searches yielded only three records, and after combing the remaining reference lists, only one additional record was found. As only one new record was found, it is reasonable to conclude that saturation, a subjective determination that new data will not provide any new information or insight (Creswell, 2002), can be assumed. As saturation was determined, systematic searching was concluded. In total, the systematic searches uncovered 109 papers relevant to this review (see Figure 2).
Figure 2. Search protocol and included records.
Step 3—Sampling
Once saturation of the literature has been determined, sampling of the included records takes place to further refine the inclusion/exclusion criteria for the papers to be included in the systematic review. The role of sampling for interpretive reviews differs from aggregative reviews in the way in which the “field to be known” is constructed. While both types of reviews limit the number of papers to be included, interpretive reviews require both purposive sampling and theoretical sampling to develop the concepts and theories emerging from the analysis of the included literature (Dixon-Woods, Cavers, et al., 2006).
For this review, purposive sampling was used initially to select papers that were highly relevant to the research question and primary/secondary outcomes (109 records). Theoretical sampling was then used to remove papers identified as not of suitable quality or relevance to this review. Theoretical sampling may also include the addition of further papers through the other search mechanisms identified in the protocol, such as web searches, theses database searches, leading author/expert in the field searches, and so on.
Step 4—Determination of Quality
Determining the quality of the included records is an essential component of refining the inclusion/exclusion criteria of the review, regardless of review type. However, a measurement of what defines quality varies greatly between aggregative and interpretive reviews. Aggregative reviews generally define quality through a hierarchy of design methods (such as RCTs vs case-control studies) and structured quality checklists. Conversely, interpretive reviews synthesizing a variety of research methods (such as quantitative, qualitative, reviews, commentary/theoretical papers) are faced with a variety of challenges in the determination of quality (Dixon-Woods, Cavers, et al., 2006).
As outlined in the CIS method, a two-pronged approach is used to determine quality. First, criteria were chosen for both quantitative and qualitative research studies to determine whether the primary papers (empirical studies) found through systematic searching were of appropriate quality to be included in the review. In this review, the following criteria were used:
- For quantitative studies, the PICO(T) framework was used to determine quality (Higgins & Green, 2011).
- For qualitative studies, the eight “big-tent” criteria for excellent qualitative research (Tracy, 2010) were used.
- For mixed-methods studies, a mixture of the two criteria was used according to the data being evaluated.
Second, records needed to be evaluated by their relevance to the research question and outcomes. This is an important element of the CIS method creating
- A critique of the assumption that any and all papers determined to fulfill criteria can contribute equally to a synthesis,
- A focus on an “ongoing critical orientation” and critique of the literature rather than an attempt at a critical appraisal of un-combinable studies,
- A way to assess the relevancy of both primary and secondary papers to determine their weighting in the review.
Weighting the papers in terms of rigor and relevancy is additionally an important element in interpretive reviews to ensure that the review accurately reflects the state of the literature. In this review, a grading system that allowed for the design of a weighting for a diverse range of literature (empirical research, re-analysis of empirical research, commentary and editorial work, and reports and policy documents) was sought. In this review, a four-point grading scale, proposed by Attree (2004), was used to achieve this result (see Table 1).
| Table 1. Grading scale used for the review based on Attree’s (2004) grading scale. | |
|---|---|
Grade | Criteria |
A | Primary papers of rigorous quality with the highest level of relevancy to the research question/outcomes |
B | Primary papers of rigorous quality relevant to the research question/outcomes |
C | Secondary papers which provided useful information and demonstrated relevancy to the research question/outcomes |
D | Papers that were excluded during the determination to quality process outlined above |
The grading of papers facilitated the emergence of prominent themes during the iterative nature of the analysis. All papers underwent the above two-pronged approach to determining quality and relevancy. Of the 109 papers included at the end of the systematic searching, 54 were excluded leaving 57 papers included in the systematic review and undergo the next step in the systematic review process, data extraction.
Step 5—Data Extraction and Coding
A rigorous and transparent data extraction process was employed in this review. The purpose of data extraction is twofold. The first is to identify themes, theories, instruments, disagreements, and so on within the literature, and as a vehicle for scrutiny of the included papers. The second is to create the most transparent process possible for the analysis of the literature. A data extraction proforma was created to ensure that each paper underwent the same analysis process (see Table 2). The proforma was created in relation to the research questions and outcomes, such as the specific inclusion of instruments/measurement tools which was highly pertinent to this review.
| Table 2. Data extraction proforma. | ||||||
|---|---|---|---|---|---|---|
Details | Purpose/aims | Sample/method | Frameworks | Instruments | Key findings | Critical analysis |
*Author/Year *Title *Country *Paper type *Grade *Key words | *Purpose of the research *What is the research aiming to learn, inform, or challenge | *Who is being researched? *Sample size *Methods used | *Theoretical or conceptual frameworks informing research | *Instruments of measurement tools employed | *Key findings according to author(s) *Key findings for the systematic review *Calls for further research | *Strengths and weaknesses of the paper *Underlying assumptions/positions employed by the author *Discrepancies in relation to the wider literature |
Through the process of searching and creating inclusion criteria for the literature included in the review, the preliminary research questions were refined throughout the continued investigation of the literature. From this iterative process, the preliminary research question was developed into its final iteration which was used to guide the extraction of data. The finalized research question that guided the analysis and critique of the literature was
- How can social indicators and socially critical ways of viewing health and education be used to inform practices and support the health and wellbeing of children transitioning to school?
During data extraction, each paper underwent a rigorous process which included thorough initial reading of papers, and the re-reading and re-evaluation of papers as themes emerged during iterative analysis (the complete data extraction table for this review can be viewed via a link to online supplementary materials). During this process, themes, categories, and discrepancies in the literature were analyzed and coded systematically and continually.
Coding is a process of using words or short phrases that “symbolically assign a summative, salient, essence capturing, and/or evocative attribute” to a portion of data (Saldaña, 2012, p. 3). Through initial coding, re-reading, and re-inspection of the raw data (the included papers), and secondary coding of the information contained in the data extraction table, categories describing explicit segments of the data were created. Figure 3 outlines the initial and secondary codes identified, and how the analysis took “shape” through categorization of the findings. For further information on the creation of categories during the coding process, see Rossman and Rallis (2003).
Figure 3. Data categories and codes.
Step 6—Conducting the Analysis
Through the recursive and reflexive processes employed in the revisiting, questioning, and problematization of the papers, early categories were further refined into seven prominent and distinct themes which formed the basis of the findings and discussion sections of the review. Rather than the explicit descriptions of data segments used to create the above categories, themes are phrases which describe more subtle and tacit processes uncovered in the data (Rossman & Rallis, 2003). The seven finalized themes below were presented in order of prominence in the literature:
- Conceptualizations of health and wellbeing for young children: Australia and internationally
- The measurement of health and wellbeing of children transitioning to school
- Parents and families as actors and agents in supporting the transition to school
- Service Integration in early childhood education and care
- School readiness and “ready schools” within the context of health and wellbeing
- A focus on those most at risk during transition
- The voice of the child in their own wellbeing
The distinctive characteristic of the CIS method is its movement beyond a summary of the data reported to a “more fundamental critique which may involve questioning taken for granted assumptions” (Dixon-Woods, Cavers, et al., 2006, p. 4). Thus, during analysis, application of the CIS method required going beyond a summary of thematic findings to a critique of the literature that is dynamic, recursive, and reflexive. Within each of the seven themes, a critique of the assumptions, perspectives, and contradictions were discussed. Contradictions within individual studies and the literature as a whole played a key role in the discussion section of the review. The discussion section was used to highlight three key findings of the systematic review:
- The continued tension between pervasive developmental perspectives of wellbeing and the less frequent holistic (encompassing aspects such as personal and social needs) definitions found in the literature. This was a key finding as the literature as a whole made substantive claims that developmental views were no longer prominent within the current research and policy landscape. This finding demonstrated the need for further theoretical development of holistic indicators for child wellbeing, and questions whether it is appropriate for wellbeing to be positioned within current understandings of school readiness/transition.
- Service integration continued to be a focus within the literature with the vast majority of included papers making claims of the positive impact of service integration on young children’s health and wellbeing. Despite this focus, there is in fact scant empirical evidence to support these claims. Further empirical evidence is needed to substantiate, refute, or moderate claims of the efficacy and importance of service integration in the early years.
- There continues to be a dearth of children’s perspectives and understandings of their own wellbeing within current child wellbeing conceptualizations. This was evidenced in both the implicit and explicit positioning of children either incapable or unnecessary contributors to this field of research in the vast majority of included papers. This finding is important in light of substantive recognition for the need of those being researched to play an active role in the research process (Baum, MacDougall, & Smith, 2006), and three decades of research evidencing the capability of children to participate in the research process and the value this brings to childhood research (Christensen & Prout, 2002; Clark, 2005; Pascal & Bertram, 2009). This finding indicates that further investigation of children’s perspectives on wellbeing can offer new insight into or necessary corroboration of current conceptualizations of child wellbeing.
The critical lens employed within the CIS approach formed the basis of the analysis of the systematic review, evidenced by the key findings discussed. It also served the secondary purpose of identifying gaps in current knowledge within this research area. As a chapter in my thesis, a clearly evidenced gap in current knowledge and according calls for further research were an essential element of the systematic review as they presented the rationale for my research project.
Step 7—Writing the Review
This systematic review formed the literature review chapter of my thesis. Whether completing a systematic review for a thesis or a journal article, the systematic review process is similar; however, the style and brevity with which it will be written will vary. While there is no one formula for writing an interpretive systematic review, they are generally written with the following (or similar) components:
- Background/Introduction
- Objective/Aims
- Search Methods
- Selection Criteria
- Data Extraction/Data Collection
- Analysis
- Findings/Results
- Discussion/Author’s Conclusions
While the headings will vary depending on journal style or thesis formatting, headings that indicate the systematic search protocol and structure of the review are essential in clearly communicating the transparency, rigor, and value of your review to the reader.
Practical Lessons Learned
As my previous education and degrees were in the field of education, completing a systematic review (an uncommon style of literature review in education) was a challenging task. While I received guidance from my supervisors, and completed a large amount of independent research to identify systematic review processes and methods, the greatest lesson I learned was that while there are many different ways to complete a systematic review, the most important aspect is that the protocol that is chosen is reasonable, explicit, and transparent. This was helpful in the fact that it provided me with freedom to tailor a systematic review protocol to the challenges of my research field, but also gave me clear parameters of what was required.
Another important lesson learned from this process was the need to keep detailed documentation at every step of the process, and for every decision made. While the researcher has the license and agency to make decisions from a wide variety of methods and processes about a systematic review protocol, it is essential that this is clearly documented so that an independent researcher could complete the same process and come to similar findings.
Conclusions
A systematic review as the basis for my PhD research project was instrumental in providing me with detailed knowledge of the “state of the literature” in my research area. It also helped in imbuing me with confidence in regards to my understanding of the nature of field and previous research in this area. This experience greatly shaped and continues to guide my research process, playing an integral part in the design of my research question, aims, and methodology. While systematic review is not the only, or necessarily, best option for a review of the literature in a thesis, aspects of a systematic review such as a clear documentation and processes for evaluating and weighting literature can add rigor and demonstrate in-depth understanding within traditional literature reviews. As evidenced in this review, the CIS method is also a useful framework for researchers working in, or investigating, literature and research that span across disciplinary lines.
Exercises and Discussion Questions
- What is the purpose of a systematic review and how does it differ from a literature review?
- List two key points of difference between an aggregative and interpretive systematic review.
- An important aspect of the critical interpretive synthesis (CIS) method is that it is iterative in nature. In your own words, define the concept of an iterative research design and give an example of how the iterative nature of this systematic review informed and/or guided the case study review protocol.
- Think about a literature review you’ve done in the past, are currently working on, or about to begin. What aspects of systematic review could be incorporated into your work to demonstrate rigor in your review process?
Further Reading
Web Resources
Link to PRISMA diagrams: https://drive.google.com/file/d/0B57KznIMSKUdMzEtc25jV0UzR00/view?usp=sharing
Link to Data Extraction Table: https://docs.google.com/document/d/1zBaC6nK_vi1kVOVMxy0NdOBRR7xLIiO_zO9pWZbBxfs/edit?usp=sharing
