A Case of Methodological Premises Underlying Literature Reviews

Abstract

Literature review is an essential predecessor to research, a way of detecting what previous research has yielded, identifying disparities and where to look next. It represents a summary of existing research on a specified topic that compares studies based on design and methods, encapsulates the findings of each, and points out flaws or likely confounding variables that may have been overlooked. Perhaps the most significant criticism of literature reviews is the probability for bias and for unreliable conclusions to be formulated. Taking up a critical perspective, this case study seeks to contribute an account of methodological and theoretical aspects underlying the process of literature examination. The main purpose is to reflect on constituents of the procedures surrounding the topic, by providing a brief case exemplification.

Learning Outcomes

At the end of this case, you should

  • Understand how to set clear aims and objectives of the review
  • Know how to critically analyze relevant sources for your review according to the study protocol
  • Be able to attentively apply inclusion and exclusion criteria
  • Understand how to report the main findings of your review
  • Understand the importance of reporting any source of bias in the data

Background and Overview

This case study seeks to outline a couple of methodological issues and steps surrounding the planning, conducting, and reporting of critical literature reviews. More specifically, it describes how critical reviews should be performed in order to support and justify a research project with clear and reliable rationale. The details of this case were acquired while the first author (G.-I.P.) implemented a research project on health-related quality of life (HrQoL) of women with breast cancer from a multidisciplinary perspective, which was funded by Grant POSDRU/CPP 107/DMI 1.5/S/78342. The main argument for employing such an examination was that there had been a considerable amount of studies delving into medical, psychological, and social issues associated with breast cancer, which encouraged calls for multidisciplinary approaches of the concept of HrQoL, but very few studies had explicitly looked at how stable the dynamics of the association between HrQoL and its dimensions were. Usually, studies encompassed a descriptive approach of data, without operating with a critical and systematic analysis. Broadly, these studies exemplified links between HrQoL and diverse domains at different points of survival, without any contestation of how congruous the evidence was. The reviewer felt that a critical analysis, based on clear examination criteria and methodology, would lead toward the clarification of this theoretical shortcoming. To address this gap in knowledge, the project encompassed two key strategies: (a) a multidisciplinary approach of the liaison between sociodemographic, illness-related, and psychological predictors and HrQoL of breast cancer survivors in both the short term and the long term and (b) a systematic methodology underlying the review. For the purpose of this case study, focus is placed on the methodology guiding the systematic review.

According to Michael Borenstein, Hedges, Higgins, and Rothstein (2009), literature reviews contain essential shortcomings, such as subjectivity and lack of transparency in the analysis, and errors in reporting the final conclusions. In addition, if a great amount of data is included randomly in the review, without any guiding methodology, the review is likely to lack scientific relevance and consistency. Typically, data would include published journal papers, with the focus on the topic of interest. The selection of sources may become a subjective option, with certain reviewers focusing on large samples, and others on ‘quality’ studies. To avoid all these shortcomings, this case study proposes a critical approach of literature reviews, by advancing a methodological framework which will guide reviewers on how to ensure the transparency and accuracy of the entire process.

The aim of this case is to encourage you, the reader, to look at your data identified in literature critically and to question your theoretical premises before establishing the directions of your research project. We argue in the sections to follow that literature examination is itself a prerequisite research with its own methodological and ethical framework. For you to get a better understanding of how to conduct your reviews, a small sample of a literature examination is provided later as an example. The sections that follow argue that it is important to engage in a critical appraisal of literature in order to identify research gaps and direct your research project. Furthermore, this case study guides you toward the steps of a review process to look at the evidence found in selected research articles.

This case study presents, as an example, a case involving a multidisciplinary examination of literature regarding women with breast cancer. It outlines the steps needed for the literature examination and discusses a series of methodological issues: aims, objectives, research protocol, inclusion and exclusion criteria, chronological framework, search strategy, variables, data analysis, and synthesis. Before delving into this exemplary case, a couple of key concepts relating to research practicalities are outlined in the next section.

Key Concepts

Before starting your review, it is essential to decide on the questions, aims, and objectives. Aims of the review are defined as general statements describing the purpose of the literature examination and emphasize what the researcher wishes to acquire. Generally, a project should not include more than two or three aims. Objectives are the steps that guide the literature exploration and support you to answer the research questions; they underline how aims need to be attained. Research objectives should be necessarily focused and precisely outlined, and should support accurately the use of concepts. The review needs a structured, written plan, termed protocol, in order to ensure its conclusions. The protocol consists of several stages: (a) an overview; (b) aims and objectives; (c) a detailed account of the methodology, which itself consists of several elements; and (d) a supporting list of references.

To conduct the literature search, the researcher needs to establish several keywords. Then, the researcher sets the criteria for inclusion and exclusion of studies encompassed in the review. These criteria generally refer to the chronological framework of the analysis, type of original studies, ways of analyzing the studies, and reporting the results. The search strategy describes the means and databases used to collect evidence for the literature examination. After evidence has been collected, the researcher then describes how the data were analyzed and summarized. To check the validity of the literature examination, criteria relating to the coherence between aims and research directions need to be verified. Variables are defined, within literature reviews, as dimensions underlying a key concept, which are aimed to be examined in relation to other concepts.

Methodological Premises

During the research project on HrQoL, G.-I.P. found it essential to make the review process as well-defined, systematic, and unbiased as possible, while maintaining a practical perspective. Measures to minimize external and internal effects that could lead to bias were considered. This section describes how the processes of data identification, collection, analysis, and summarization were attained. A concise account of these stages is provided in Figure 1 as a synoptic view of the entire process.

Figure 1. Organizational pattern of the literature review.

None
Aims of the Review

The main aim was to understand the various predictors of HrQoL of breast cancer survivors in both the short term and the long term, and the relationships between them. A secondary aim was to address some shortcomings in the existing theoretical frameworks relating to HrQoL of women with breast cancer. More specifically, issues of and reasons for conflicting evidence were looked at.

Objectives

Four objectives supported the main aim of the review:

  • (O1) to identify a range of peer-reviewed articles in international databases
  • (O2) to analyze critically, thematically and chronologically these articles by using the selection criteria
  • (O3) to write up a critical review report
  • (O4) to examine the validity of the literature review
Search Strategy

A literature search was conducted using the following key data sources: Loughborough University online library, PubMed, BMJ, and PsycINFO; and focused on the following keywords: breast cancer, health-related quality of life, treatment side effects, survivorship, predictors, and quantitative data. These keywords were used independently and in combination to generate as much data as possible for the purpose of the analysis. The search process embraced relevant online resources and prevented redundant knowledge.

Selection Criteria

The current analysis included primary data; quantitative studies; original cited articles, where possible; data collected from breast survivors; and reviews conducted by other scholars. Studies involving proxies (i.e. a person authorized to act for another), book or article reviews, research published in languages other than English, qualitative studies, incomplete evidence, and editorials were excluded. This literature review focused on relevant information published in English in peer-reviewed international journals. To attain proximal similarity, as Catherine Pope, Mays, and Popay (2007) recommend, a reasonable number of studies were selected, until saturation (no new relevant sources emerging) was achieved. From the list of identified journals, any irrelevant material based on the title or abstract was removed before starting the analysis. Then, inclusion and exclusion criteria were individually applied to determine which studies would be relevant for inclusion and subsequent appraisal.

Variables

HrQoL served as the dependent variable, and demographics (age, time since diagnosis, illness phase, education, area of residence, and marital status), social support, and personality were independent variables. As you may know, a dependent variable represents something that is influenced by other factors, called independent variables.

Chronological Framework

The search strategy was concerned with identifying electronic published research articles between early 1980 and 2013. The decade of the 1980s was selected as the outset of the review, as it was defined as the early period of quality of life and cancer survivorship research, according to Karen Meneses and colleagues (2010), although sporadic significant evidence was previously published on the topic.

Data Analysis

All the remaining articles were then stored in a folder organized around the search keywords. Critical appraisal checklists were compiled for each article, and appraisal and data extraction were undertaken by the lead researcher. For all relevant appraised studies, data on the type of population, medical treatment (if any), sample size, and keywords were recorded in evidence tables and considered carefully for accuracy and completeness. All procedures were compliant with the aforementioned methodology. No formal contact was made with authors of the original articles due to the short time allowed for the completion of the project. To avoid methodological pitfalls, the review derived from a prudent selection and examination of data. All relevant studies were collated and reviewed. Three criteria adapted from Floortje Mols, Vingerhoets, Coebergh, and van de Poll-Franse (2005) were used to critically evaluate the evidence identified for each independent variable, as described in Table 1. Slight changes in the definition of these criteria were used in the present case, and only ‘strong’, ‘weak’, and ‘inconclusive evidence’ guided the analysis.

Table 1. Criteria for the analysis of peer-reviewed articles.
None

A sample of 118 sources met the review criteria and were chronologically, thematically, and methodologically analyzed in order to draw up conclusions on the results reported in the literature to date. The review focused on themes, ideas, and outcomes rather than on authors. The studies were compared and contrasted with controversies highlighted. Before writing up the review report, a summarization of data was stored in a table, indicating the source, authors and year of publication, and keywords. Then, a final narrative report, organized around main independent variables, was written up and checked by the second author (D.C.) of this case.

How to Write Up a Review Report: A Brief Case

In writing up a review report, a few prerequisites should be followed. First, recapitulate the main aim of the analysis. Second, briefly present the major results, before approaching each variable independently or in relation to others. Third, place your research project within the scope of published literature as a means of informing (a) the route of your project (by determining identified research directions according to the initially established chronological framework) and (b) your research design (by providing explanations for conflicting results). These steps should support you in addressing the research questions of your project. In the following subsections, a short example of a review relating to the association between HrQoL of women with breast cancer and education is given.

  • Main aim

    This critical literature analysis was organized mainly around predictors of HrQoL and associations between them in order to inform the research process. More specifically, the predictive significance of sociodemographic variables as well as psychological, illness, and treatment attributes for HrQoL were ascertained.

  • General results

    A total of 118 peer-reviewed articles, including both empirical studies and reviews conducted by other scholars, informed the analysis process. Therefore, 29 reviews and 89 primary quantitative studies provided accounts of weak, strong, and inconclusive evidence for the predictors of HrQoL of women with breast cancer. The reliability of the literature examination was provided by the in-depth investigation of each article, the elaborate utilization of search criteria, and the substantial sources of information included. Inconclusive evidence was identified for the relationship between HrQoL and the following predictors: age, marital status, education, illness stage, time since diagnosis, subdomains of social support, religious coping, and personality. Strong associations between type of treatment and HrQoL were described in several articles. More research needs to be conducted for the relationship between area of residence and religious denomination and HrQoL, as the evidence for these predictors was weak.

  • Identified research directions

    The research directions of the early decade of quality of life and cancer survivorship research (1980–1990) explored acute and psychosocial effects of treatment and quality of life outcomes. Eleven sources marking this time period were included in this analysis. Research conducted during the 1990s focused on the conceptualization of quality of life and cancer survivorship. Thirteen papers made reference to this period of research. The study directions of the period aimed

    • to conceptualize quality of life and cancer survivorship (breast cancer physical and psychosocial needs during active treatment),
    • to study cancer survivorship with a homogenous population of breast cancer survivors during active treatment (the first half of the decade),
    • to study quality of life during active treatment and post treatment (in the mid- to late 1990s),
    • to identify psychosocial treatment side effects,
    • to use longitudinal designs with long-term survivors.

The next period (2000–2013) was dedicated to early intervention studies to maintain or to improve HrQoL in cancer survivors. A total of 94 articles were encompassed in this section of the analysis.

  • Education as an independent variable

    Inconclusive evidence about education as a predictor of HrQoL was reported in the literature. According to some authors, breast cancer survivors with a higher educational achievement had a better HrQoL. Education was not reported as a predictor of quality of life in the surveys done by Ganz et al. (2002), Mols and colleagues (2005), and Safaee and colleagues (2008).

  • Potential explanations for the findings

    The explanation for this result might be that more educated women retained more skills to look after themselves. Another explanation might be that higher levels of education imposed high professional expectations that could not be accomplished due to the illness consequences. It is important to mention that in assessing the relationship between the two variables, it is critical to possess a vindicated understanding of the definition and measurement of ‘education’ and ‘quality of life’. Educational achievement was measured in terms of the highest level of education achieved. The decision to include the variable of formal education in the present study may be seen by other scholars as somewhat ‘narrow’. Nevertheless, it seems that studies that have found significant associations between educational achievement and subjective quality of life included narrow definitions of the concepts. According to Dockery (2010), the discrepancy between studies might be the consequence of the disparity in the conceptual frame: whereas some studies defined education by the highest level of achievement, others looked at the total years of education.

Review Assessment

Up to this point, we have explained some essential methodological steps for conducting a literature review. In the section to follow, we provide several strategies to ensure your review reliability and to support you in checking the literature examination process. To address these conditions, in this section, we recommend assessment guidelines, comprising the following assessment criteria: (a) yes, (b) unsure, (c) partly, and (d) not at all.

The first step is to check whether the research evidence reviewed was wide-ranging. In doing so, you need to verify whether (a) the aim for the literature analysis was clearly focused, (b) appropriate databases were used for literature search, (c) the period of time included in the search was adequate, (d) appropriate keywords were used, (e) an appropriate combination of keywords and terms was used, (f) the search strategies were provided, and (g) whether a hand search of the reference lists was completed, by looking at relevant sources that have not been identified in the initial search.

The second step refers to checking whether bias in the selection of articles was avoided. Therefore, you should evaluate whether (a) inclusion and exclusion criteria were reported, (b) the reasons for excluding references were provided, (c) the criteria for inclusion and exclusion were clinically and methodologically applicable, (d) the reasons for exclusion conformed to the selection and exclusion criteria, and (e) whether the process for selection of studies was effectively explained.

The validity of the evidence can be looked at by examining whether (a) given the search strategy, the risk that relevant evidence has been missed was low; (b) the criteria for selecting the evidence were precise; and (c) whether the selected studies matched with the aim.

For a review to support strong arguments for the research project, you should ascertain the coherence between the evidence and future research directions. Therefore, at least three prerequisites are recommended: (a) conclusions were supported by data and the review, (b) issues included in the studies were comparable to those targeted by the research directions, and (c) the conclusions were methodologically pertinent.

To ascertain whether the scientific quality of future research directions does not present risk of bias, the following circumstances should be met: (a) the strength of evidence ascribed to the research proposal is adequately described and justified and (b) the research directions are compatible with the culture and values within the context in which they are to be used.

Strategies to Conduct Critical Literature Reviews

We have aimed to discuss the implications and prerequisites of conducting literature reviews through the lenses of methodological practicalities. A brief example of how to report literature analysis rooted in clinical psychology has been provided earlier in this case study. Nevertheless, we aimed to define the steps underlying the literature examination in such a manner that they can be applicable to a multidisciplinary framework. This section provides several suggestions to guide any novice or less experienced reviewer:

  • identify significant online databases to conduct your literature examination. If possible, contact the authors if full content of their article has not been made public.
  • make a list of potential sources relevant to your research project. Begin this process by searching for each writer's contribution in the field. Assess each publication's reputation and intended audience. Then look at how each contribution fits your main aims.
  • look at keywords and topics of the research articles. Identify main ideas and research directions in relation to your research questions. Organize your ideas in a table thematically and chronologically. By doing this, you can easily identify sets of variables relating to your research topic and identify themes that support the formulation of your research arguments. This process can help you draw up clearer conclusions in the end.
  • In case of conflicting evidence, try to provide explanations for underlying discrepancies by looking at how studies were conducted and the methodological premises that were used by the authors. In other words, try to understand what compelled different scholars to answer the same research question in different manners. Then, establish how the evidence informs your assumptions or research questions.
  • Keep in mind that literature reviews are time and energy consuming. Beware that sources do not provide direct answers to your research question, but they can suggest approaches toward the formulation of your arguments or counterarguments.
  • When you write up your review report, try to identify any source of bias that might lead to confusion or misleading conclusions. Assume that how the literature review is conducted will have a great impact on how your research rationale is formulated. Put simply, literature reviews need to be conducted in critical and ethical manners.

Exercises and Discussion Questions

  • Make a list of potential databases and sources of information that might provide you with evidence relating to your research project.
  • What are the shortcomings of literature reviews?
  • List ways to avoid bias in data analysis and report. Give examples of potential sources of bias that might hinder your analysis.
  • Establish a list of aims and objectives that might guide your review. Provide a justification for your choice.
  • Design a brief review protocol to ensure the ethical framework of your literature examination.
  • Analyze your evidence thematically and chronologically, by using columns in a table for year of publication, name of authors, source, keywords, main findings, and explanations for conflicting evidence.

Further Reading

Weyns, D., Iftikhar, M. U., & Ahmad, T. (2012). Protocol for systematic literature review on formal methods in self-adaptive systems (Version 1.0-2012-03-19).

References

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta analysis. Chichester, UK: John Wiley. doi:http://dx.doi.org/10.1002/9780470743386
Dockery, A. M. (2010). Predictors of quality of life in breast cancer patients under chemotherapy (NCVER Research Report). Adelaide, Australia: NCVER.
Ganz, P. A., Desmond, K. A., Leedham, B., Rowland, J. H., Meyerowitz, B. E., & Belin, T. R. (2002). Quality of life in long-term, disease-free survivors of breast cancer: A follow-up study. Journal of the National Cancer Institute, 94, 39–49. doi: http://dx.doi.org/10.1093/jnci/94.1.39
Meneses, K., & Benz, R. (2010). Quality of life in cancer survivorship: 20 years later. Seminars in Oncology Nursing, 26, 36–46. doi: http://dx.doi.org/10.1016/j.soncn.2009.11.006
Mols, F., Vingerhoets, A. J. J. M., Coebergh, J. W., & van de Poll-Franse, L. V. (2005). Quality of life among long-term breast cancer survivors: A systematic review. European Journal of Cancer, 41, 2613–2619. doi: http://dx.doi.org/10.1016/j.ejca.2005.05.017
Pope, C., Mays, N., & Popay, J. (2007). Synthesizing qualitative and quantitative health evidence. Berkshire, UK: Open University Press.
Safaee, A., Moghimi-Dehkordi, B., Zeighami, B., Tabatabaee, H., & Pourhoseingholi, M. (2008). Predictors of quality of life in breast cancer patients under chemotherapy. Indian Journal of Cancer, 45, 107–111. http://dx.doi.org/10.4103/0019-509X.44066
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles