This case study describes our experience in conducting a systematic review of the relevant literature to address the question of how religious and occupational identities relate to each other in the workplace. In so doing, we offer practical advice for the novice researcher in the steps involved in conducting a systematic literature review, highlighting the questions that need to be taken into consideration, particularly in deciding when to include or exclude a publication. We describe the practical lessons learned throughout the course of our research process.
By the end of this case, students should be able to
- Describe and explain the rationale for conducting a systematic literature review
- Gain an understanding of the pros and cons of using a systematic literature review
- Gain basic understanding on how to structure a systematic review
- Discuss the key factors that researchers need to take into account when evaluating a systematic review
We conducted a systematic review of relevant literature to address the question of how religious and occupational identities relate to each other in the workplace. We were interested in this topic because religious beliefs can affect employees in their performance of occupational duties. For example, in 2014, the case of two Catholic midwives seeking to avoid the supervision of nurses involved in abortion procedures reached the United Kingdom’s Supreme Court, which ruled against them (“Catholic Midwives,” 2014). In 2017, the European Court of Justice, in response to the sacking of a Muslim receptionist, ruled that workplace bans on wearing hijabs need not constitute discrimination (“EU Workplace,” 2017).
Thus, religion can significantly affect work-related practices such as management of workplace discrimination (Brotheridge & Lee, 2007) or diversity (Gebert et al., 2014) but when we looked into this topic, we found surprisingly little research that investigated the intersection of religion and the workplace (Lynn, Naughton, & Van der Veen, 2010). That is, before delving into our own empirical investigation of the topic, we decided that we needed to have a better idea of what was already out there and what might be missing. To this end, we aimed to conduct a systematic review of the relevant but fragmentary literature in management, psychology, and sociology. But before we move into a discussion of the practicalities, let us first consider what a systematic review is.
What Is a Systematic Review?
When faced with any question, being able to conduct a literature review is an important skill for any student and researcher. That is, research projects (should) start with a literature review to identify the relevant current literature, its limitations, quality, and potential. A literature review enables the researcher to see what studies exist and to assess what is missing in the published research on a specific topic. Hence, literature reviews are a key tool at the beginning of any research project. A literature review should serve as a guide for determining the next step in doing research as well as a method for gathering evidence for informing policy and practice.
Yet, literature reviews can be criticized on the grounds that what is presented consists of a subjective and unsystematic account of the current state of knowledge. How is the reader to know which literature was not taken into account in the review, what is missing, or how a researcher went about conducting the literature review?
This is why the medical sciences, and gradually also the social sciences (see Tranfield, Denyer, & Smart, 2003; Xiao & Watson, 2017), have attempted to provide clearer guidelines to improve the quality of the review process. Thus, there is a drive to conduct what we call systematic literature reviews, which entail conducting a comprehensive review of the current research around a specific topic that is valid, reliable, and repeatable (Xiao & Watson, 2017). In other words, the method of a systematic literature review makes it possible that someone else will find the same relevant work as did the researcher who conducted the initial search. That is, Chandler, Higgins, Deeks, Davenport, and Clarke (2017) define a systematic literature review as “an attempt to identify, appraise and synthesize the empirical evidence that meets pre-specified eligibility criteria to answer a given research question” (p. 5).
Literature reviews can be useful (1) as a background for an empirical study (this is most often the case and often these reviews are not “systematic” as such) or (2) as a stand-alone piece (Templier & Paré, 2015). Our work is a stand-alone review that aims to make sense of the research on religious and occupational identity through aggregation, interpretation, and integration of the literature (Rousseau, Manning, & Denyer, 2008).
For an excellent overview of the kinds of systematic reviews one can conduct, we refer the reader to Xiao and Watson (2017). Our work falls into the category of a textual narrative synthesis (see also Popay et al., 2006), which is characterized by having a standard data extraction format by which to record study characteristics (context, quality, findings, etc.). Such a synthesis will then compare similarities and differences across studies and includes commentary about the strengths of evidence available for the research question.
In what follows, we will describe the design and procedure of our research in greater detail.
Research Design and Practicalities
To plan (first step in a review process) our systematic review, we first conducted interviews with experts in the field. This step is recommended by Tranfield et al. (2003), who state that an advisory panel can help to direct the search and review process.
We interviewed 11 academics (professors in management, health care, research methods, and organizational psychology) and practitioners (religious officials, a university librarian), who served as an advisory panel throughout the review process. These interviews and our own reading of some initial literature helped us to sharpen our review questions and refine our protocols. Review questions guide the review process, and all steps in the process should be geared toward answering these questions. We decided on two review questions that framed our search:
- Research Question 1: How do religious and occupational identities interact? (a) In what situations are these identities compatible? (b) In what situations do they create tension and conflict for employees and their occupational practice?
- Research Question 2: What are the implications of conflict and compatibility between religious and occupational identities for the well-being of employees and organizations?
The second step is to generate search terms. Search terms are the terms used in the search engines and databases and should be directly derived from the research questions. Search terms can be broader than the concept of interest. For example, the core concept we were interested in is identity, but we also included role in our search terms. Many search engines allow for Boolean operators, which are simple words (AND, OR, NOT, or AND NOT) used to combine or exclude keywords in a search, resulting in more focused and productive results. The creation of search terms is not trivial. They need to be precise enough to yield results for the concepts in question but also exhaustive in the way that most if not all of the relevant literature is found. Xiao and Watson (2017) recommend that at an early stage of a search, being exhaustive is more important than being precise. Moreover, it is important that our search terms can be used by others to repeat our search for cross-check and verification.
Next, we identified sources for our literature search. Wishing to be comprehensive, we chose to focus on peer-reviewed articles and non-peer-reviewed articles (i.e., reports, summaries, opinion pieces, and essays, although no non-peer-reviewed material met our selection criteria for inclusion in the review) and books/book chapters identified via 11 electronic academic databases (i.e., Business Source Complete, Web of Science, Scopus, Emerald, Center for Evidence-Based Management, Psychology Cross Search, Applied Social Sciences Index and Abstracts [ASSIA], Sociological Abstracts, Medline, Cochrane Library, PubMed). We also considered conference proceedings (i.e., Academy of Management Proceedings), papers in press (identified via SAGE Premier), dissertations (identified via ProQuest Dissertations & Theses Full Text), as well as publications recommended by our expert panel. We also considered Open Gray and Google Scholar. After initial pilot searches, we developed our search strategy specifying search strings regarding identity, religion, occupation, complementarity, and conflict (Table 1).
|Table 1. Search strings used in literature searches.|
Identi* OR self OR role
Religion, occupation, faith
AND religi* OR occupant* OR profession* OR faith
AND conflict OR tension OR ethic* OR compatib*
AND well-being OR predictor OR emotion* OR mental health OR stress OR mental depression OR burnout
We then developed our criteria for selection. Adopting an inclusive approach, we did not specify a start date for the literature to be searched, nor did we delimit publication type or format. Publications included needed to be in English with an abstract or summary available on or before March 2017. Most importantly, inclusion required relevance to one or more of our review questions.
Our searches resulted in 70,752 non-duplicated, potentially relevant outputs. Most were peer-reviewed articles, with some non-peer-reviewed materials (e.g., reports and summaries) and books or chapters. Other outputs were admissible if they were informed by empirical work (e.g., reviews and theoretical papers based on research). We also considered conference proceedings, papers in press, and dissertations. All results were reviewed by one researcher in light of our criteria and screened for relevance first by title and then by abstract. For further examination, we included papers using equivalent concepts to those specified in our review questions, such as “religiosity” as a proxy for “religious identity.” At this step, many sources were excluded on the grounds that they did not meet our criteria and were not relevant for at least one of our review questions. At the next review step, we looked at full texts of papers.
A subsequent full-text evaluation of the resultant 373 papers focused on relevance and quality of evidence. In addition, 20% of papers (hence, 75 papers) were also reviewed by two co-authors to check for evaluation consistency. At this stage, we decided to include 125 papers for further consideration. These were cross-checked for relevance by three researchers. This last step in the selection process let us to 53 publications addressing at least one review question and formed the main body for our systematic literature review. As with other forms of data, collected data from systematic searches should be documented in an appropriate format. We recorded our findings in an EndNotes library and a Microsoft Excel sheet. We also stored full texts of papers in a shared folder for all researchers to access.
At this stage, the synthesis stage began. We used a standardized data extraction form to extract information in relation to our two review questions, thereby helping us to group the data. Our data extraction form included information on the date, purpose, theoretical framework, methods, data analysis, results, relevance to research questions, and a conclusion. This information was extracted from all 53 papers we included in our review. It helped us to capture the key findings or theoretical postulates from the reviewed papers. The extracted data formed a base for a further ideas webbing analysis, a method for conceptualization and exploring the connections among findings and theories in a visual way (Popay et al., 2006). This approach helped to identify possible relationships across empirical results and theoretical points. In addition to the data extraction, we also reviewed the quality of evidence. It seems that there is little consensus on how quality is evaluated in a review (Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005). We classified whether research meets specific levels of evidence based on criteria from the Center for Evidence-Based Management (2018). Level 1 refers to high-quality, replicable, comprehensive literature reviews. Level 2 evidence constitutes multi-site case studies or large-sample quantitative studies, while Level 3 evidence reflects small-sample single site quantitative or qualitative studies. Level 4 evidence refers to descriptive studies and/or self-report stories. We recorded the level of evidence but did not exclude work based on these levels.
In the writing-up of our review, we reported findings from literature search, screening, and quality assessment. We aimed to be as transparent as possible in how we conducted our review. In our review, we grouped our findings according to the research questions and reported on the included literature. In the latter part of the review, we pointed out gaps in the literature as well as opportunities and directions for future research.
All these steps encompass what we call a review plan. Our review plan included the purpose of the study, the review questions (see above), inclusion criteria, search strategies, quality assessment criteria, data extraction, and finally the synthesis and reporting (Xiao & Watson, 2017). All in all, a review protocol is similar to a research design in other forms of data collection in social science. It is a concrete plan that specifies the method used to conduct the review. The development of a plan is key for the validity and reliability of the systematic review. The rigor of the review plan sets a systematic review apart from a “regular” literature review. When conducting a systematic review, students (master’s degree and doctoral students) should work closely with their supervisors to lay out their review protocol for consideration before conducting their review; researchers might want to validate their protocol with collaborators and potentially their advisory panel. Moreover, it is the key element of the “Method” section in the paper/report that should give readers the necessary information to replicate the search and synthesis of research.
Method in Action
Having described the review process in detail, we will now focus on the method in action. The first problem one encounters is access to most papers in full text. We were largely successful in gaining such access, which was available via online resources; missing papers were available through interlibrary loan or were requested directly from authors. However, recent advances in online resources make the conducting of a systematic literature review much more feasible.
It’s important to keep in mind, however, that a systematic literature review is a time-consuming exercise, and conducting a systematic review with an interdisciplinary approach like ours is even more time-consuming. Researchers will need considerable amounts of time to conduct the initial screening, which can easily involve thousands of papers. Even in the later stages of review, the process of screening and data extraction is costly and elaborate. Yet it also is highly satisfying, especially when you are new to a particular topic, as you will get an extensive overview of that topic.
One pitfall of the method is that for publication, your data search will need to be updated. For example, because the writing-up and publication of results took time, at several points, we needed to update our search to include the most recent available data. At this stage of the work, a rigorous search strategy is important to help update the search quickly. It is worth noting that a good record of each review stage is useful to keep the search consistent.
All in all, the method we used served the purpose of surveying the field and providing an extended overview. The literature that we reviewed was very diverse, however, in terms of quality, methods, and so on. Accordingly, it was not easy to synthesize. The range of work we reviewed ranged from case studies to theoretical papers to qualitative methods to quantitative and even experimental work. For example, the diversity of work did not allow us to conduct a meta-analysis of results. A meta-analysis is a method established by Gene V. Glass (1976) that requires the extraction of quantitative data to conduct statistical tests of combined studies.
Practical Lessons Learned
The process described above allowed us to write a paper that was then presented at conferences and was submitted for publication. It also led us to design follow-up empirical work. Among the important lessons we learned from conducting a systematic literature review are the following:
- Performing a rigorous review requires a significant amount of time. To ensure a reliable and valid process will take considerable effort; this is not a method that can easily be done in a short time. A recommendation for future researchers, especially postgraduate students, is to plan to devote at least one semester for the careful construction of a review plan and the initial search and screening as well as synthesis.
- For a truly systematic literature review, it is useful to have a team of researchers. This is helpful because the method is complex and time-consuming, and entails making many decisions concerning search terms and inclusion criteria when extracting the data. Thus, inter-rater reliability is an important element. For postgraduate researchers, who often do not have a large team, it may be useful to include peers, a librarian, and one’s supervisor to ensure a rigorous review process. It is also useful to have access to different perspectives when the data are analyzed, synthesized, and written up. In addition, team members can also help each other to stay motivated throughout the course of lengthy process that requires tremendous persistence.
- The use of cross-check in the early stage of our systematic review was particularly helpful. This helped us to ensure not only consistency in the inclusion and exclusion processes but also a clear understanding between the researchers.
- The use of pilot review was another useful element in our process, particularly at the early stage of systematic review. We conducted pilot review (i.e., using the full search terms but limited to recent years of publication) to test out the consistency of our search terms within different databases. This step gave us the confidence to continue and carry out our main search.
- Careful consideration should be given when using inclusion criteria to calculate scores for the quality of each publication. We have learned to view such criteria as a useful resource but not as a rigid rule to follow. For example, one of the criteria for scoring a publication is to look at the rating and impact of the journal. For the purpose of systematic review, this is not necessarily the most important element to consider. Indeed, by the overly rigorous use of this criterion, we may have excluded some publications that might have offered insight into our review question.
- Avoid the tendency to include everything. It is attempting to be very inclusive when conducting a systematic review. However, one should bear in mind one’s review questions at any stage of a systematic review. Here, we offer a helpful guideline: If a publication is not answering at least one of your review questions, then exclude that publication. However, if it has some background relevance or may be potentially helpful to your research in other ways, it is a good practice not to completely abandon it but to note it for possible future use.
A systematic literature review is an essential step in defining a research area and to explore a specific research space. It is a valuable exercise for novice (postgraduate) and experienced researchers. There are a variety of different methods and types of reviews but they all share the use of a rigorous method to conduct a review, which makes the work reliable, transparent, and valid. To summarize, a good review provides an overview of a specific research topic, identifies key concepts, explains the relevant prior literature, proposes a guide for future research, and notes the implications for researchers and practitioners alike.
Exercises and Discussion Questions
- Why is a systematic literature review worth conducting?
- What is the value of having an advisory group for your review? Who should be asked to be part of the advisory group?
- What are the first three steps you would take to conduct a systematic review, and why?
- Explain what must be done to ensure that your search protocols are reliable.
- Explain what criteria should be used in your decisions whether to include or exclude a source from your search results?
- What makes cross-checking a useful step in performing a systematic review?
- At what point is it appropriate to conclude your systematic review?