This case study illustrates distinctive features of case study methodology that were responsive to context in ways not often seen in case study research. Located in Alberta, Canada, the study explored the factors that supported or hindered licensed practical nurses' ability to work to the full scope of practice assigned to them by legislation but that they had not been able to fully implement in the field. While the case study method as described by Robert K. Yin provides a number of useful design components, there are several limitations, most particularly with regard to lack of rigor and the potential for bias. Due to the politically charged environment of this study, particular attention was given to devising an objective method for the selection of case study sites as well as to a number of other strategies to strengthen study rigor. An extensive literature review, theory development, a research framework, the success case method, a province-wide survey, and statistical modeling were used to produce an objective and defensible platform for site selection as well as to enhance study rigor. Study findings were well received by the diverse stakeholders who represented key sectors in the health-care system.
- Gain an understanding of the research components required in case study design
- Understand the limitations of case study research and learn some strategies to mitigate these weaknesses
- Learn how evidence can be strengthened by the use of multiple data sources
- Understand the value of conducting a literature review to highlight methodological challenges observed in other research projects so that they can be avoided in future
- Understand how theory building can provide a strong foundation for case study research
- Learn how a research framework can provide a useful structure for tool design, data analysis and synthesis, and the preparation of individual and cross-case reports
This case study describes the recent study conducted to explore the impact of workplace factors on a particular group of health-care professionals. It tells the story of how, as researchers, we changed our methods in response to a complex environment and stakeholders' need for an evidence-based approach.
Coming into the study as Research Director, I was convinced that my experience in case study methodology would make the whole thing pretty straightforward. Between 1992 and 2008, I had conducted five program evaluation studies using case study methodology. I had explored a wide range of topics from customer service training in tourist facilities to new ways of teaching science in schools. I had written 34 individual case study reports, and because I like to write, I enjoyed doing this very much.
I based my work on a close reading of the work of case study methodologist, Robert K. Yin. I used a revised version of his 1984 classic reference book, Case Study Research: Design and Methods (SAGE, 1989). While a number of revised editions have been published since, I preferred my dog-eared copy, complete with highlighting, underlining, and marginal notes.
These early case study projects were certainly challenging in terms of the time, effort, and resources required, but I do not recall questioning the methodology itself. The case study method seemed self-contained and robust, and I replicated it in each study. Now, however, those experiences seem simplistic and lacking in rigor. The complex context and political sensitivities evident in this study caused me to rethink my assumptions about case study research. A much different approach would be the result.
In 2006, in the western Canadian province of Alberta, legislation was changed to allow licensed practical nurses (LPNs) to use a broader set of skills in health-care settings. LPN training was expanded from a 1-year certificate to a 2-year diploma, and programs were offered at a number of community colleges. Working to full scope became the expectation for LPNs, but change in practice was slow. By 2009, their annual professional survey results suggested that only 50% of LPNs were working to full scope. So what was the problem?
This study was commissioned in 2011 to find out more information about LPNs' scope utilization by the Workforce Division of the provincial government's department of health. Resource constraint and workforce efficiency were key drivers, but staff also needed sound evidence on which to base decisions related to policy and staffing. As a result, the study became a priority.
The grant flowed through the College of Licensed Practical Nurses of Alberta (CLPNA) to Bow Valley College in Calgary, Alberta. The college had a large LPN training program and a department that specialized in applied research and evaluation. I was hired as Research Director and acted, with my colleague, Dr Rena Shimoni, as Co-Principal Investigators. A longtime LPN practitioner and former instructor became our Project Manager. A steering committee was established to provide guidance. It included a number of carefully selected stakeholders as well as some nurse researchers to ensure that all groups with a vested interest in study outcomes were represented.
We wanted to find out what workplace factors were influencing the LPNs' ability to use their new skills. Typically, they worked in care teams with registered nurses (RNs) and health-care aides (HCAs). Each of the three nursing groups had their own training and certification processes and their own skill sets. Each group was paid on a different wage scale. While there were some clearly delineated tasks assigned to each group, in other areas, there was role overlap. The LPNs generally found themselves to be the ones ‘in the middle’.
The relationship between the RN and LPN professional organizations was strained. There was much anecdotal evidence of turf battles. Some RNs feared that LPNs would take over part of their jobs. At the same time, many LPNs felt undervalued by RNs, many of whom are unaware of the new legislation and upgraded training.
In reality, the range of education for both groups was quite broad. While RNs continued to upgrade their own skills, some of their most senior members had graduated when a 2- or 3-year diploma program had been the norm. Similarly, some senior LPNs had graduated many years before from a 6-month certificate program. The confusion and misinformation generated by these issues were evident in both the literature and public discourse. As researchers, we could see that this topic was clearly complex, political, and very, very tricky.
We proposed using a comparative case study design to conduct six case studies in different health-care facilities around the province. We looked to Yin for the guidance we needed. As he pointed out, case study research was useful to understand complex social phenomena within a real-life context. It provided a good way to explore interventions with no clear single set of boundaries or outcomes. These rationales certainly described our research context, but Yin also pointed out some methodological limitations:
- the time-consuming nature of the research
- the massive documentation that is produced
- the limited control an investigator has over actual events
- the lack of rigor and the potential for bias
- the inability to generalize study findings
He confessed that case studies were typically viewed as a ‘less desirable form of inquiry’ than either experiments or surveys. As a result, he suggested using strategies to mitigate these limitations such as referring to the literature, using multiple sources of evidence, and adhering closely to the research design.
There were five components that Yin believed were especially important:
- The study's question
- The study propositions
- Its units of analysis
- The logic linking the data to the propositions
- The criteria for interpreting the findings
Some of these components, particularly the units of analysis, left a lot of discretion to the researchers. Keeping in mind the critical mind-set of our stakeholders, some of whom would be looking for any hint of bias in the study, we needed to develop strategies that would enhance objectivity, increase rigor, and produce defensible evidence. And so we embarked on a journey to modify the case study method to fit the complex demands of our research environment.
We took the advice of Rossi, Lipsey, and Freeman (2004), and within the loose structure of the case study method, chose to be as rigorous as possible. We wanted to establish a confident basis for action, to withstand any criticism that might try to discredit our study, and to ensure that our information would be judged sufficient under scientific standards.
To get us started and to inform our approach, we conducted an extensive literature review. It had three objectives:
- To understand available evidence in order to provide a strong foundation for the study
- To highlight the methodological challenges associated with examining one professional group working within a complex, interactive team
- To identify gaps in knowledge associated with the impact of LPNs' scope utilization on quality of care
Very quickly we discovered that little research had focused specifically on LPNs or on other equivalent occupations (e.g. ‘registered practical nurse’.). Out of over 150,000 publications with the word ‘nurse’ cited in the PubMed database, only 374 included the term ‘licensed practical nurse’, and only 29 referred to scope of practice. We extended our search to unpublished policy documents and reports, and eventually, we identified nearly 100 documents for review. We produced a report that summarized our findings (Shimoni et al., 2011) and circulated this early product to stakeholders.
Many of the studies on scope that we reviewed had methodological problems, data limitations, or attribution issues. Flaws included unreported variables, confounding factors, small sample sizes, inappropriate use of summarized scores and aggregated data, and attribution issues associating outcomes to specific team members. Still, it was clear that some of these studies continued to influence nursing thought.
Our next step was to build a theory to test our assumptions. Our study purpose as stated by our funder read as follows:
To provide objective, research-based evidence focused on LPNs in typical health-care settings and to explore the factors that promote and/or inhibit successful LPN scope utilization.
Initially, we had proposed a set of research questions, but these were refined after we had reviewed the literature. Several nurse researchers on our steering committee also offered us some sound advice. The final questions were as follows:
- What can we learn about LPNs' individual practice that promotes or inhibits their ability to practice to full scope? How can these supports be enhanced? How can these barriers be reduced?
- What can we learn about LPNs' work teams and systems that promote or inhibit their ability to practice to full scope? How can these supports be enhanced? How can these barriers be reduced?
- What can we learn about LPNs' organizations that promote or inhibit their ability to practice to full scope? How can these supports be enhanced? How can these barriers be reduced?
- Is there any evidence of differences in the patient experience when LPNs are working to their full scope? What are these differences?
We theorized that four key factors influenced LPNs' scope of practice in the workplace. These included (1) the individual LPN and related characteristics, (2) the care team in which the LPN worked, (3) the organization or site in which the LPN worked, and (4) the patient or client and their required nursing care. We designed the Scope of Practice Factors Model, and it provided the theory that guided our study (see Figure 1).
We developed a research framework or Data Collection Matrix (DCM) to link our model to the research questions. Many of the topics identified in the literature were linked to the four key factors and provided the basis for study sub-questions and related indicators. We used the DCM to guide our tool development, and all tools were coded to its numbering system. Later that same numbering system was used to code and track the data we collected. This created a structured evidence trail that lead directly from the model through the DCM to tool development, data collection, data analysis, data synthesis, and final report preparation. The excerpt from the DCM in Figure 2 shows the links between research questions, indicators, tools, and item numbers. By checking back and forth between the model, the DCM, and the tools, we continually sharpened the study focus.
In the past, when identifying case study sites, I had developed a simple sampling framework (e.g. rural vs urban and large vs small) and filled in the cells with reasonable or accessible choices. Now faced with heightened demands for rigor, that approach felt like throwing darts to see where they landed. Everyone we talked to had a recommendation for a ‘good’ case study site. We questioned the wisdom of this approach because it was based on personal opinion.
Needing a stronger rationale, we turned to the success case method developed by Robert O. Brinkerhoff (2003) to understand the impact of training. He claimed that it was a fast, credible, and effective way to evaluate organizational change. He believed that we learn the most about a phenomenon by interviewing both those individuals who are the most successful at implementing change and those who are the least. The separation of high and low scope sites seemed a promising way to understand scope utilization issues.
However, there was a small problem. Brinkerhoff's method was predicated on having survey data. He suggested setting high and low cutoff scores on several survey items and then randomly selecting individuals from each group (i.e. most successful and least successful). Thus, in order to identify high and low scope sites in an unbiased way, we first needed to survey all LPNs in the province.
Luckily, the steering committee could see the value of our suggested selection method and approved the addition of a province-wide survey. There were a number of reasons why this was a good idea, particularly because the survey allowed us to go beyond perceptions of scope to explore actual recorded practice. We used the competencies identified by the CLPNA as the basis for assessing actual scope. We also asked questions about site location and work setting and, based on the literature, added questions asked about the work environment, including communications, team work, safety culture, job satisfaction, and stress.
We sent out 8549 surveys to all practicing LPNs providing both an online and mail-in option; 2313 LPNs responded for a response rate of 27%. While we would have liked a higher return rate, we found that the respondents tracked proportionately to staff deployment across the province. The absolute number of returns also added to our confidence.
In the end, the decision to add a survey to our design strengthened our study immeasurably. The mix of quantitative and qualitative methods added depth and credibility to our findings, but it also allowed us explore a number of issues more fully. By staging the research over two phases, we had time to refine our focus as we went, so that by the time we actually visited the sites, we knew a lot about more about LPN characteristics and salient workplace issues than we would have if we had gone there directly as initially planned. We were able to refine our case study tools based on survey findings, focusing quickly on key topics. For us, administering a survey first followed by in-depth case studies was a winning strategy.
We hired our colleagues at Science-Metrix, an evaluation firm located in Montréal, Québec, to conduct our statistical analysis and to help us with site selection. We asked them to provide four categories of sites with three possible selections in each one. The categories included
- acute care sites in which LPNs work to low scope,
- acute care sites in which LPNs work to high scope,
- long-term care sites in which LPNs work to low scope,
- long-term care sites in which LPNs work to high scope.
For their analysis, the research analysts, David Campbell and Olivier Beauchesne, selected Question 28 (Q28) of the survey (see Figure 3). They determined that a score of 1 would be associated with a low scope of practice and 5 with a high scope. They produced aggregated statistics at the site level and removed respondents with less than 75% valid answers (9 out of 12 items). Invalid answers were considered to be blanks or ‘not applicable’ answers. They also removed sites where less than five respondents had replied. In the end, 52 sites remained in the analysis.
Three statistical procedures were performed on these data:
- a factorial analysis distinguished between Q28 variables that occurred more often in either acute care or long-term care settings. Dimension 1 variables related mainly to the administration of intravenous medications or blood products, more common in acute care. Dimension 2 variables related to developing and revising care plans and to teaching clients and families, more common in long-term care. The two dimensions were plotted on a graph.
- based on their average score for each of the 12 items in Q28, sites were clustered into four groups, discriminated by their propensity to allow LPNs to practice to full scope. This procedure was called k-clustering (see Figure 4).
- multi-criteria analysis was used to find the sites that performed highest or lowest in either acute care or long-term care settings. Scope performance was displayed on the graph by making the size of the bubble proportional to the site score.
Based on this analysis, the research analysts drew up a list of recommended sites and forwarded it to the research team.
The research team reviewed the list of suggested sites. Only at this final stage did qualitative considerations enter our deliberations so our choices lay within the parameters of the list produced through statistical analysis. As a final screen, we added some inclusion and exclusion criteria. For example, we wanted sites that were not too technically dependent on a specific treatment or too specialized in their target population (see Table 1).
Finally, we considered geographic location and size. We developed a table of high and low scope sites and identified our first and second choices in each cell. Invitations were sent to all the first choice sites. We were really excited when all six agreed to participate.
The final sample included three acute care sites, one mixed site (providing both acute and long-term care), and two long-term care sites. Three sites were high scope and three were low; three were urban and three were rural. Now we could proceed with our research knowing that our sites had been selected based on the best possible evidence. No dartboards for us!
Of course, the site selection activity occurred early in the research process. To provide multiple lines of evidence, we used a number of tools, adapting standardized instruments where possible. We used standardized tools to measure patient and family experience. Where no tool was available, we created our own tools based on the DCM. These included four interview guides (for senior administrators, team leaders, RNs, and LPNs) and a focus group protocol for the HCAs. These tools were validated extensively but even so, once we were in the field, we still modified some of the wording after our first site visit.
The data were collected by a team of two researchers, including me and a junior researcher. Logistical support was provided by our experienced LPN Project Manager. We recorded each interview digitally, taking notes as a backup. Although we had many minor adventures in the field, because we had planned our study so carefully, the research rolled out smoothly, and ultimately, we collected data from 193 individuals across the six sites.
The recorded data were transcribed into individual Word documents, validated by a second researcher, and imported into MAXQDA, a qualitative software program. Data summaries were compiled and organized by DCM topic and emergent theme. Finally, the information was summarized in narrative form in six case study reports, using the Scope of Practice Factors Model and the DCM to organize our material.
The reports were sent to the site administrators who each reviewed their own report for accuracy. They sent us corrections as needed and also completed a validation survey rating the report's validity, relevance, utility, and value. All were satisfied that the reports reflected LPN scope issues and suggested that the information would be useful. The following comments (from both high and low scope sites) were typical:
I found the report to be an impartial and balanced perspective of the LPN scope of practice. I appreciated that it included possible areas for further study and some direction on care/assignment aspects that may require some education and discussion with all staff. Thank you for involving us in this study. (Senior Administrator, Site 2)
It was a privilege to be involved in this study. I found the report fascinating. Although I thought I understood the LPN role at this site well, it was very advantageous to see the promoters and barriers that relate to work setting and scope of practice summarized in a table. This summary has prompted me to think about other ways we could utilize our work force at this site … This report is excellent and is an excellent tool. Thank you for the data. (Senior Administrator, Site 4)
Each site report was revised based on administrator feedback and was returned to them for their use. A high level of confidentiality was employed throughout the process. The administrators were never told if their site was characterized as high or low scope, and no one else ever saw these reports.
As part of the final report, we prepared a cross-case summary, using our model as an organizing guide. Factors that promoted or inhibited scope utilization were described. No site names were mentioned, and only very general setting descriptors were used, such as urban, rural, acute, and long-term care.
To ensure rigor, we created a data triangulation table to summarize the key findings across all the sources of evidence including the literature review, survey, case studies, and key informant interviews. Thus, we were able to demonstrate which case study findings concurred with or strengthened previous studies, which findings were new, and which, if any, contradicted the reported literature.
The final report provided a great deal of well-documented evidence about LPN scope of practice. The results and recommendations were well received by the steering committee and other diverse players in the health-care system. The report was approved for circulation, and the CLPNA posted the complete report on their website. The research team was invited to present key findings to the annual LPN conference, and a panel of key stakeholders responded with comments about their own next steps.
This case study research explored the factors that supported or hindered LPNs' ability to work to the full scope of practice assigned to them by legislation but not yet fully implemented in the field. While the case study method as described by Yin provides a number of useful design components, there are several limitations, most particularly with regard to lack of rigor and the potential for bias. This study devised an objective method for the selection of case study sites and also used a number of strategies to strengthen study rigor throughout. An extensive literature review, theory building, a research framework, the success case method, a province-wide survey, and sophisticated statistical modeling were used to produce an objective and defensible platform for site selection.
Six case study sites were identified representing high or low scope work places for LPNs, environments in which they were either supported or hindered in their ability to use the competencies for which they were trained. The use of a survey greatly enhanced the information obtained by the study, and the two-phased approach allowed the researchers to incorporate early findings into later research activities.
The resulting case studies provided a rich and detailed description of six particular health-care sites in Alberta and as such have been able to inform the broader discussion about LPNs' scope of practice. While decision makers have suggested that they will use the evidence produced in this study for policy change, it is too soon to tell what impact it will eventually have.
- If you were to choose the sites but you did not have the resources to conduct such an extensive survey, what other methods can you think of in order to identify the high and low scope utilization sites? What are the reliability and validity strengths and issues of these options?
- Could additional research be conducted to see if the key findings from this study could be generalized beyond the findings of these six sites? What would you do next?
- In this research, a quantitative survey was used to identify sites with high and with low utilization of scope for the case studies. Can you think of a situation when a quantitative survey may follow completion of the case studies?
- Do you think that the view still exists that case studies are considered a less desirable form of inquiry? On what do you base this opinion? What changes have occurred (e.g. in society, in people's mind-sets, and in research) that may have influenced present thinking about case study methodology?
- This study took place in a complex, politically charged health system. Can you give an example of another system where the traditional case study method would be insufficient? Are there further steps beyond the ones described here that would be necessary to ensure rigor in this type of context?
- In your own area of research, what features of case study methodology would be the most important? What rationale would you develop to support using this methodology in this context?