A systematic review is a useful method to answer a research question where prior studies have been conducted. A well-designed and executed systematic review can inform policy and/or practice change. It can also identify gaps and generate new research questions. Although the requirements considered essential for conducting a rigorous systematic review are well defined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement, the approaches taken to synthesize the data vary. This case study describes the narrative synthesis of heterogeneous quantitative studies and the meta-synthesis of qualitative studies used to answer a complex research question from the consumer perspective. The study design focused on the analysis of consumer data only. As a result, the synthesis of both quantitative data and qualitative data has provided a detailed insight into consumers' unique perspectives and needs. The synthesis approach for both datasets is described, and linkages to key tools and resources to help facilitate this approach are provided. Processes used by the research team to enable effective research governance and collaboration throughout are also detailed.
By the end of this case, students should be able to
- Define a systematic review and outline the key elements required to ensure a review is rigorous
- Outline practical suggestions on how to systematically source and manage data to answer a defined research question
- Consider how to design a study to represent the appropriate population through careful data extraction and synthesis techniques
- Discuss methods for synthesizing quantitative and qualitative data across multiple studies when a meta-analysis is not possible
- Outline key strategies to support a research team to collaborate effectively and gain consensus where required
- Consider key factors for communicating synthesized data within a publication format
This case study describes the approach taken as part of preliminary work completed to develop consumer-centered quality indicators, focused on end-of-life care, for use within the Australian hospital setting. For the purposes of this study, end-of-life care is defined as the care received within the last 12 months of life, inclusive of the care received in the final days of life. Understanding what consumers (patients and their families, informal carers, and/or next of kin) identify as important in relation to end-of-life care within the hospital setting was considered to be a crucial first step in the development of relevant quality indicators. Therefore, our team discussed how we might answer our research question with a specific focus on whether we would conduct another primary study or look to research already completed.
On discussion, the team agreed that given significant work had already been completed on this question, a rigorous review of such work would be the most effective method to inform our next steps. Therefore, we chose to conduct a systematic review of published peer-reviewed literature to extract the relevant data, synthesize these data, and present our findings in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement. The PRISMA statement outlines an evidence-based minimum set of items for reporting within a systematic review to enable a quality review (Moher, Liberati, Tetzlaff, & Altman, 2009). Importantly, our review was designed to rely on consumer data only with this design evident within eligibility criteria, data extraction, and synthesis:
A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. (Moher et al., 2009, p. 334)
In the era of evidence-based medicine, systematic reviews are required to adhere to the same rigorous methodological standards as primary studies. The subjective nature of a non-systematic literature review, whereby the evidence or included studies are chosen selectively (not systematically) based on what appear to be most suited to the research question, increases the likelihood of generating biased results and reducing confidence in the review findings. The emergence of checklists for reporting such as the PRISMA statement has greatly enhanced transparency and reproducibility of methods, and most journals now require manuscripts reporting reviews to adhere to this guidance. Furthermore, explicit guidance about review design and conduct is available from the Cochrane Collaboration and ought to inform planning (Higgins, 2008).
Ensuring that systematic reviews are conducted and reported using the highest standards is essential because, given the plethora of reviews, the methodologic rigor is rated using various rating systems, and the extent to which reviews contribute to the evidence synthesis process is ultimately determined by the level of quality their conduct and reporting adhere to (Shea et al., 2007). Although quality of conduct and quality of reporting are two different aspects to a review, the reader can only make an informed judgment about quality of conduct if a review has reported all the necessary details.
Each stage of a systematic review requires careful thought and consensus discussion within the research team. That is, agreement on the research aim, eligibility criteria, search terms, and databases to be searched along with other information sources is required prior to commencing the search itself. Following this, detailed data extraction occurs to inform both a quality assessment and the synthesis itself. In relation to quality, agreement is required on the purpose and means of assessing the quality of included work and how this affects inclusion/exclusion. Finally, clarity about data and the approach for data synthesis are fundamental to ensuring a quality review occurs and contributes to development of new knowledge.
Completing this planning work within a research team requires meetings for planning, discussion, and debate. Importantly, the governance structures for our work were discussed and agreed upon at the commencement of the study and included the following: a schedule for regular meetings (inclusive of face-to-face and teleconference options to enable international collaboration); planning for recording of meetings, outcomes, and key actions (this was the role of the PhD student to maintain); and a shared repository for working documents and the valuing of debate to ensure mutual agreement. Our meetings often highlighted additional questions for further consideration, and these were often managed by group email until a final protocol for the research was available. Careful attention to each team member’s thoughts and ideas was managed through record keeping and updating documents accordingly. As newer versions of work became available, earlier iterations were archived for future review if needed.
This approach valued the iterative nature of unfolding research through ongoing reflection and refinement as required. Although published reviews seem quite straightforward, our experience of getting to this stage was one of needing ongoing reflection, discussion, and debate with an explicit focus on our research aim to guide thinking and avoid project creep. In addition, it is vital to establish clear governance procedures for the research team to value all contributions and ensure collegiality, consensus, and confidence in publishing results collectively. Our experience reflects one of rigorous debate throughout different stages of the study, which led to improved critical analysis of our work and an improved outcome. We did not experience any discord given the team remained respectful of all views and considered how these did or did not align to our research aim and, therefore, in what ways they affected study planning, conduct, or reporting.
Research aims underpin work completed and in relation to reviews inform either comprehensively synthesizing all results from included primary studies or focusing on specific outcomes or other measures. Our review of qualitative studies chose to prioritize patient and family views over the views of the authors of the primary studies, and therefore, our search design, data extraction, and synthesis methods continued to focus on the voice of the consumer through extraction and use of their raw data rather than “Results” sections in their entirety as is more commonly the case. Indeed, this approach also occurred within our quantitative review (Virdun, Luckett, Davidson, & Phillips, 2015) where any data from health professionals were excluded from analysis.
This case study describes the first stage of our program of research. The point of difference in relation to our work was the design of our review to focus purely on consumer data. It is important to think about and design to enable review results to represent a particular population, and this case study explains one approach to achieve this.
This review had two key areas of consideration in relation to research practicalities, namely,
- How to manage the large number of articles retrieved by the original search and determine which programs would be the best ones to use to support data extraction, analysis, and synthesis (article and data management);
- How to synthesize data from heterogeneous study designs, including both quantitative and qualitative research (methods for synthesis).
We retrieved close to 2,000 articles from a search across nine electronic literature databases (Virdun et al., 2015). In addition to this, further articles were found via desktop searching of the Internet and through handsearching the reference lists of included articles. Consideration about how to best manage articles retrieved from each search was important to ensure accurate conduct and reporting in accordance with the PRISMA method. A decision was made to import all search results into EndNote (version X5), consolidate this list via the “remove duplicates” function, and then manage the title and abstract review through the use of smart groups.
EndNote is a software package that supports referencing, managing a library, and organizing such material. This was freely available to the team, appeared to enable the key functions we required, and our institution provided librarian support to assist with learning and troubleshooting. All potential articles for inclusion were managed within one smart group for ease of review by the whole team. Smart groups within EndNote are created by the author and correspond with key topic areas. For example, you can create a smart group for “included” and within this group; all articles to be included in the synthesis are available. This practical approach had both positive and negative aspects when implemented, as described later.
In addition to the use of EndNote for managing imported searches, a detailed document that outlines search strategies used and results obtained is also fundamentally important. The use of both EndNote with this additional search overview document (Microsoft Word file) ensures the search used could be reproduced, building confidence that the pool of studies was likely to be comprehensive, given the research aims stated, and enabling readers with different purposes to identify the extent to which the pool might answer their own questions.
This case study reports one method used to manage this (EndNote and Word file). However, there are multiple online sources available to assist in managing searches and the resultant body of review documents. It would be useful to discuss such options with a librarian and investigate these further prior to study commencement. We spent time initially with a librarian and also spoke with two PhD students who had recently completed published systematic reviews. The information gained from such discussions was invaluable and led us to choose the approach taken.
Decisions about which programs were the best for use in this review was based on what data were to be extracted, how these data could best inform synthesis, accessibility across multiple team members, availability of programs, and competence to use chosen programs well. Programs used included EndNote (version X5), Microsoft Word, Microsoft Excel, and EPPI-Reviewer 4 (Thomas, Brunton, & Graziosi, 2010). EPPI-Reviewer 4 is a web-based program designed specifically for managing and analyzing data for systematic reviews. Developed in the United Kingdom, it is accessed through a fee payment that is based on individual or team access and is time limited. Therefore, you can choose to purchase only a few months if this suits your research.
Training for the lead researcher was required in effective use of Endnote and EPPI-Reviewer 4. Endnote training occurred locally through the student’s University library and training in EPPI-Reviewer occurred through reading their manual (available online), queries sent through to their online support team (responses received within 24 hrs), and discussion with two other researchers who had recently used this tool. Due to the type of data we obtained and heterogeneous study types, meta-analysis was not possible. However, a software option for those planning to complete a meta-analysis is Review Manager, available from the Cochrane Collaboration. An alternative option for the management of qualitative data (we used EPPI-Reviewer 4) is NVivo.
After identifying articles for inclusion in this study, it became clear that data were available from very different study designs. Given that methods for integrating results from quantitative and qualitative research are in their infancy and require separate synthesis of each before integration is attempted (Higgins, 2008; Popay et al., 2006), we made a practical decision to separate studies using these two different approaches. That is, the review used one search strategy but used two different approaches to synthesis based on data type.
The process that led to this decision was iterative and was based on regular scheduled discussions. That is, our team was led by the research, and as this unfolded, it became clear that the data were large in number and heterogeneous in both type and research design. Our team wanted to enable a clear synthesis that could inform clinicians and policy-makers in their practice; thus, clarity and brevity were important. We therefore agreed that conducting two syntheses, connected within the second work, would suit our audience’s needs most effectively. This led to communicating such findings through two publications:
- Dying in the hospital setting: A systematic review of quantitative studies identifying the elements of end-of-life care that patients and their families rank as being most important (Virdun et al., 2015);
- Dying in the hospital setting: A meta-synthesis identifying the elements of end-of-life care that patients and their families describe as being important (Virdun, Luckett, Lorenz, Davidson, & Phillips, 2016).
Most approaches to integrating qualitative and quantitative studies in systematic reviews use the qualitative to inform interpretation of the quantitative rather than the other way around. In our case, the qualitative synthesis provided a deeper understanding in a way quantitative data could not. An example of this can be seen in relation to the concept of “environment.” Our study was looking at what is important to patients and families in relation to end-of-life care within the hospital setting. Quantitative data were mostly obtained from surveys in which predetermined questions were asked of participants. Synthesized results of the quantitative data showed that an adequate environment for care was of importance to patients (ranked highly). However, what aspects of the environment were of importance remained unknown. The qualitative data were able to provide focus on this and showed that patients described concepts of space in unique ways (for some, a private room was important, but for others, this was not the case) and also highlighted the importance of smaller hospitals for end-of-life care, focusing on these having a nicer atmosphere, feeling closer to home, having more accessible parking, and an increased feeling of safety. In addition, qualitative data from families outlined the importance of both environmental and organizational characteristics for optimal end-of-life care, centered around three key aspects:
- Hospital rules and processes should not detract from optimal patient care;
- The need for privacy, cleanliness and quiet; and
- Space is required to support cultural practices (Virdun et al., 2016)
This example shows the importance of working closely with two datasets to fully answer a research question, but with important distinctions in the approach taken in each case. Whereas a synthesis of quantitative data is usually summative, a synthesis of qualitative data assumes that “the whole is greater than the sum of the parts” and seeks to expand interpretation beyond the primary studies themselves. Commencing with a synthesis of quantitative work is a useful approach for some studies as it enables a summative reflection of work to act as an analytical framework to inform the subsequent meta-synthesis. This was certainly the case for our work.
A systematic approach was used to source articles for inclusion to ensure rigor and reliability and to enhance the application of study results to practice and policy. This occurred in line with guidance provided by the Cochrane Collaboration (Higgins, 2008) and PRISMA (Moher et al., 2009) as summarized in Table 1.
|Table 1. An overview of work completed and resources used to complete the PRISMA stage of these reviews.|
What we did
Resources to support this stage of work
Development of search terms
PICO/PICOS (quantitative), and SPIDER (Qualitative and mixed methods) frameworks
Key publications in the area
Review repositories—for example, CareSearch (palliative care search filter https://www.caresearch.com.au/caresearch/tabid/353/Default.aspx), Agency for Healthcare Research and Quality, Cochrane Collaboration
Consideration of which databases were most relevant to inform our research question
Research team—consensus discussion about relevant databases to inform research aim
Running the searches
Librarian to assist with altered truncations per database as required
Importing of all citations into EndNote (X5) for review
Manually adding search results from desktop searching and handsearching
Maintaining a separate Word file that detailed search strategies used and outcomes (date of search, number of results, etc.)
Librarian—discuss which system may best support the identification and screening of articles as there are many possibilities
Word file to note key details of searches run to ensure reproducibility of the search if needed (perhaps to update the search over time, to audit the review, or for another research team to utilize the same approach)
Development of explicit inclusion and exclusion criteria
Discussion with research team
Review of key publications
Screening and Eligibility
Review of article titles. Where not relevant at all—moved to an “exclusion” smart group in EndNote
Review of article titles and abstracts—again moved to “exclusion” smart group as appropriate
Accessed full text for review where an abstract indicates this study may be eligible. Consensus discussion as required to inform final set of included articles
Quality review of full texts
Completed by one member of the research team with auditing of such work completed by a second member. Note that the Cochrane guidance is for two people to independently screen all articles (Higgins, 2008)
EndNote used to record this process with a separate Word file to document discussions with the team where consensus was required. Again, there are templates and/or systems to support this aspect of work and are worth looking into.
There are a number of resources to support a quality review of papers. A repository of such tools is available at http://www.equator-network.org/
This study used one tool for the quantitative studies (Australian Government, 2006) and another for the qualitative studies (Kitto, Chesters, & Grbich, 2008)
Final set of included articles retrieved. Citations available within one EndNote smart group and full texts collated for ongoing study and data extraction
EndNote used for this review
Data extraction commences—initially into a detailed summary table that is refined over time
Word and Excel used for this review
EPPI-Reviewer 4 used for management of qualitative raw data
Originally, we had hoped to be able to use meta-analysis as the synthesis method for quantitative studies. However, due to included study designs, this was not possible. Therefore, the team met and discussed the included articles and what synthesis approach would appear to best represent the data. After discussion and review of possible approaches, a narrative approach to synthesis was used (Popay et al., 2006). A meta-synthesis was used for the included qualitative studies informed by the three-stage thematic analysis approach described by James Thomas and Angela Harden (2008).
Importantly, this study focused specifically on extracting and synthesizing data from consumers only. Therefore, extraction of data from quantitative studies included results specifically noted as from patients and/or their families. Where health professional data were available, these were excluded. Likewise, extraction of data from the qualitative studies relied solely on raw quotes from patient or family participants. Author narrative and health professional quotes were excluded. This approach to data extraction enabled synthesis of consumer-centered data and provided confidence that our review provided the consumer view of what was important for end-of-life care within the hospital setting.
Development of search terms was informed by both team members and a librarian. This was important so as to better understand appropriate truncations for use across multiple databases and accuracy of search terms for use. For example, adding an * to the end of words enables searching for multiple versions of that word (patient* = patient and patients). The lead author (Claudia Virdun) is a current PhD student and had a lot to learn about optimizing a search and found the time spent both with team members and librarians vital to success. As a result, Claudia worked with a librarian to run versions of search terms until we felt we were getting results that more specifically met our research question. This actually meant that we used our sixth search strategy, with the first five attempts providing information for us to use in improving the search terms further.
Ensuring the search had sufficient breadth (retrieving a broad range of articles refined according to search terms) was important as was the need to focus the search onto our research aim. This balance was complex to achieve and we succeeded through discussion, review of other key publications, and a trial run of some terms prior to finalizing those used for our study.
- Spend time with a librarian to inform this aspect of your work. Ensure you have finalized your research aim prior to this meeting.
- Find some key publications in your area and review their search terms for relevance to your work—both for the terms to support your subject area (e.g., end-of-life care) and research type (e.g., empirical, qualitative, and quantitative).
- Review your search terms closely with your full research team to refine these further and ensure they explicitly relate to your research question and will draw the data type (e.g., qualitative, quantitative, and empirical) you are seeking.
- Run some “trial searches” so as to review the accuracy of your chosen search terms and refine as necessary. It is very helpful to do this with a librarian if possible.
Taking the time to discuss screening methods and inclusion criteria in detail at the commencement of our study was very important and indeed fundamental to the success of both systematic reviews. That is, you need to be clear from the outset (prior to the search itself) about what you are seeking to find and understand why such factors are important to the quality of your work. We held focused meetings to enable this development with email feedback for further refinements as needed. Being new to this work, Claudia underestimated the importance of this step (she was keen to jump into the search and get going!) and in hindsight can see how important this stage of planning and preparation is.
Once the eligibility criteria were clear, reviewing articles for inclusion and exclusion was mostly straightforward. Where there was any ambiguity or disagreement, several members of the team reviewed these for discussion and consensus agreement. Disagreements occurred around two aspects of article screening: quality review and inclusion for some articles that perhaps had some aspects of importance but predominantly did not answer the research question. Claudia kept a record of such articles or quality screening disparities (screened by two team members independently) and discussed these at scheduled meetings for team consensus. Careful listening to all team members and collegial working enabled this to be an easy process to engage in.
It is important to keep a record of agreement consistency and predefine approaches to rating of articles. If possible, dual review at all steps is the ideal approach to decrease the effects of bias from one author’s perspectives and also to draw focus to areas where differences have occurred and discussion is therefore warranted.
- Define the inclusion and exclusion criteria carefully—be specific and take time to get this right. Ensure you are led by your research question for this.
- Develop a document to record agreement consistency and consensus discussions.
- Create smart groups within EndNote to match your exclusion categories to inform your PRISMA reporting.
To ensure a rigorous quality review, it was important for our team to use different appraisal tools for the two different datasets we worked with. That is, we used tools specifically designed for either quantitative studies or qualitative studies. For our quantitative review, we used a tool that had been designed for use within some recently published palliative care guidelines (Australian Government, 2006), and for our qualitative review, we used a tool designed specifically for qualitative studies (Kitto et al., 2008). This work was completed independently by two members of the team using an Excel template.
This was a great approach as it allowed the lead author, Claudia, to review the completed screens, highlight any differences in ratings, and bring this to a meeting for full team discussion prior to finalization. This supported full team engagement in this process, supported learning for Claudia as a PhD student, and provided confidence in the rigor of this screening process.
- Be clear about why you are measuring quality and therefore how this should be conceptualized. There is particular controversy about whether reviews can and should attempt to evaluate quality of qualitative research due to the large numbers of diverse methods used and the fact that some research may require subjective judgment, thereby being hard to review (Dixon-Woods, Shaw, Agarwal, & Smith, 2004). We approached this by being clear about whether key aspects of quality would affect inclusion of an article—predefining this is essential.
- Spend time initially finding the right tool to assist your quality review being mindful of your data type (quantitative, qualitative, or mixed) and the capacity to compare quality ratings across studies of different kinds (Viswanathan et al., 2012).
- Add sufficient details within your quality review to assist discussions between team members following independent review.
- Outline what (if any) components from a quality review would affect inclusion or exclusion of a study within your review.
Synthesis of the quantitative data in our review used a narrative approach, following recommendations made by experts from the Cochrane Collaboration (Popay et al., 2006), including tabulation and content analysis. The content analysis needed to evolve with several versions required before we found the optimal approach to enable accurate representation of the data across all included studies (n = 8). This approach was driven by the desire to privilege the patient and family perspectives and convey this information as concisely and accurately as possible to readers so as to provide a solid foundation of evidence with the potential of affecting policy and practice change.
Arriving at the best way of privileging and honoring the patients’ and families’ voices took time and required us to consider many different approaches to the synthesis before arriving at what we considered to be the best approach (Virdun et al., 2015). We managed this through regular scheduled meetings to review different approaches, discuss these, and decide on possible improvements. As the lead author, Claudia worked on suggestions between meetings and sent this updated work to team members ahead of scheduled meetings so they could review, comment, and inform discussions accordingly. Considering alternative approaches to the synthesis was important as it enabled us to gain a deeper understanding of the data and informed the final synthesis approach.
The final approach we used focused on the use of categorical data from patients and families (available within each study) and reporting the top five elements found within each study. These elements were subsequently themed into key domains. Because there were eight studies included in this synthesis and each study reported on multiple elements of importance for quality end-of-life care in the hospital setting, attempting to synthesize all of these was too cumbersome and did not allow for a clear message to evolve from the data. Choosing to report on those listed as the “top five” elements in each study allowed a clear focus to emerge. Theming such data led to the identification of six domains of importance for patient participants and five for family participants. This approach brought a sharp focus on areas of highest importance. This framework was used to inform the subsequent meta-synthesis.
The approach for synthesis of the qualitative data was straightforward and in line with a three-stage approach (Thomas & Harden, 2008):
- Line-by-line coding (reading each sentence and attributing key concept areas to each line);
- Descriptive theme development (working with the key concepts from Step 1 and collating these into broader descriptions);
- Analytical theme generation (using a framework to critique the work from Step 2 and develop new knowledge in the form of themes).
In practice, this meant extracting the data for coding through copying patient and family data from original articles into a Word file, per article. Each Word file was then uploaded into EPPI-Reviewer 4 and line-by-line coding followed.
Of note, this study adopted a novel approach to meta-synthesis in that we coded only the raw quotes available within each study. Traditionally, coding of raw data as well as the full Results section (i.e., the author’s discourse in relation to raw data presented) occurs. We chose not to do this to focus attention specifically on the consumer view through their presented quotes—again ensuring we privileged the patient and family data. This approach could have resulted in some loss of meaning and could also represent a biased view given the quotes provided are chosen by the authors only. However, we felt the focus on consumer quotes was important to ensure the centrality of the consumer voice in analytical theme generation.
Once line-by-line coding was completed within EPPI-Reviewer 4, reports were generated and exported. These reports collated all coded items so that it was possible to view all quotes coded to one particular aspect of care. These data were then exported into a Word template to assist with development of descriptive themes. This template had the following headings:
We moved from EPPI-Reviewer 4 to Word at this stage purely for ease of viewing the data across one page and for accessibility across multiple team members. Given the volume of data being analyzed (across the 16 included studies), the development of descriptive themes occurred in two steps:
- Development of all descriptive themes led by data;
- Collation of any areas of similarity across themes to enable a second set of descriptive themes to develop.
Again this work was completed in Word, using the following template:
Descriptive themes—family data (28)—V1
Grouping of similar descriptive themes to inform V2
Descriptive themes—family data (16)—V2
This was important to enable further refinement of the data before reviewing with an analytical framework. As you can see, this step took the family data from an initial 28 descriptive themes to 16. This also allowed the patient data to move from an initial 34 themes to 10. An example of part of this work is available in Table 2, where you can see eight original descriptive themes were reviewed, collated, and developed into two descriptive themes to inform analytical theme generation:
|Table 2. An example of the refinement of descriptive themes|
Grouping of similar descriptive themes from the family data to inform V2 descriptive themes
Descriptive themes—family data—V2
Determining the best approach in relation to treatments is complex and ambiguous being influenced by medical possibilities, emotion, finances, considering futility, doing all that is possible, and enabling a peaceful death
Importance of family members feeling sufficiently informed in a timely manner: “Nothing, nothing was important to me as much as just being able to talk to the doctor and to get the information there.” (Nelson et al., 2010)
Importance of staff proactively talking with family members—providing information, updates, and support
The importance of medical staff nominating a time for families to meet/speak with them on a regular basis
Navigating communication across staff and families—the need for family meetings
Effective communication enabling shared understanding about the current situation and expectations for the future—”please, tell me in English.” (Nelson et al., 2010)
The importance of timely information and support to access appropriate financial assistance to enhance overall comfort for patients and families
The complexity involved within end-of-life decision making
Regular effective communication is fundamental to optimal end-of-life care in relation to shared decision making and overall support
The need for scheduled updates from the medical team and regular family meetings is essential
The framework used to further analyze the final versions of descriptive themes was the domains found in the earlier quantitative synthesis (Virdun et al., 2015). Again, this was done within Word using the following template:
Descriptive theme V1
Descriptive theme V2
Additional areas of focus from qualitative literature
This approach to meta-synthesis was systematic and it remains easy to track how each included article and indeed each included quote contributed to final themes.
- Allow your initial data extraction to contain sufficient detail (e.g., demographics of participants, specifics about research methodology, and details about findings). This can be refined into more of a high-level summary for publication. However, the detailed extraction is useful to keep going back to as the study unfolds.
- In addition to your data summary table (Word or Excel), keep a demographics overview (in Excel) to inform aspects such as mean age, overall cultural representation, and male/female inclusion. Using some of the formula options within Excel is useful for this aspect of reporting.
- Allow flexibility in data synthesis approaches. Keep trying different approaches until a method is found that allows the data from multiple studies to accurately speak for itself and succinctly inform a readership. Ensure auditing and consensus discussions by the research team continue to occur throughout this process.
- Carefully record all stages of data synthesis to enable auditing of work completed.
- Consider the audience for this work from the outset as this will inform the most appropriate methods for data presentation.
There were five key practical lessons learnt throughout this study:
- The use of EndNote for managing the sourcing and inclusion/exclusion of articles through their smart group function is an excellent method. However, completing this approach within an existing EndNote library is not advised. Due to the fact that multiple databases are used, duplicate references are expected. This leads to a huge reference library with many of these references not relevant or, indeed, duplicates. On this occasion, this led to a corrupted library that was not usable. We were able to recover the library but did not use this again for any other purpose other than tracking inclusion/exclusion. A key tip in relation to this would be to create an EndNote library specifically for a systematic review and use it for this function only. Then, once you have established your “included” articles, these citations can be imported into your working library for use when publishing this work. Furthermore, in addition to the use of EndNote, it is necessary to track other information for ongoing reference. We did this within Word and recorded details such as the database searched, date of the search, search strategy used, and results found.
- Work with software that is simple and specifically meets your needs. Most of the work completed was in fact done within Word and Excel. EPPI-Reviewer 4 was an easy-to-use tool to enable line-by-line coding for the meta-synthesis. NVivo is another tool that would enable such data analysis.
- Keep the initial data extraction as detailed as possible to prevent having to continue reverting to the original articles. Once there is a detailed overview, it can be refined for other purposes such as publication. Think carefully about the categories of data extraction as these need to inform study overview (aim, design, method, participants, setting, and results), data for synthesis, demographic information, and data to inform a quality analysis. This can be done within Word or Excel, depending on your preference.
- Presenting findings in such a way that busy clinicians and policy-makers can grasp an overview of available literature with confidence is vital. Considering the main audience for the work is the first step. Second, writing up the work succinctly is important and will be informed by publisher guidelines. Finally, drawing explicit links to policy and practice within a discussion section of a publication paves the way forward for readers.
- Establishing clear working procedures for the research team from the commencement of the study is important. You need to consider aspects such as methods of communication, processes for consensus, documentation of discussions held/email communications, delegation of work roles, frequency of contacts, and a method for managing any urgent concerns to enable work to progress as required. We also established a shared work folder online (we used DropBox) for the team to have access to as needed. Having a discussion about these components of team work at the outset should enable effective collegiality and collaboration.
Completing a systematic review of published literature to answer a well-researched question is an excellent method of informing policy, practice, and future research. There is significant published guidance about how to design, conduct, and report a quality systematic review. This case study reports on the specific approaches this team used to privilege the consumer voice in relation to our study aim. We actively excluded data from health professionals and author narratives to enable a sharp focus on what patients and families reported. As a result, we now have a base of consumer-centered information to inform practice and policy change alongside an evolving research agenda. We have provided information on the processes used for effective team work; methods for accurate data collection, extraction, and analysis; and key tools to assist in such work. It is hoped the detail provided will assist readers in the design of their own studies accordingly.
- When is it most appropriate to consider conducting a systematic review? Is a systematic review better than a primary study? Why or why not?
- What are the key considerations for the development of search terms and eligibility criteria?
- What should you consider when choosing how to complete your data synthesis?
- How would you highlight the voice you are representing within a systematic review?
- What are key considerations when planning to publish a systematic review within a peer-reviewed journal?
- What are some key differences between a meta-analysis of quantitative data and meta-synthesis of qualitative data? Why might you choose one over the other or need to do both?
Amstar—Assessing the methodological quality of systematic reviews: http://amstar.ca/
CareSearch—Palliative care knowledge network: https://www.caresearch.com.au/
The Cochrane community: http://tech.cochrane.org/revman
EPPI Centre: http://eppi.ioe.ac.uk
The Equator Network—Enhancing the QUAlity and Transparency of health Research: http://www.equator-network.org/
PRISMA—transparent reporting of systematic reviews and meta-analyses: http://prisma-statement.org/
QSR International—NVivo: http://www.qsrinternational.com/nvivo-product