The research referenced in this methodological case study chronicles the journey of the researcher in identifying a research design that would utilize both quantitative and qualitative research strategies to explore education reform initiatives across two schools implementing the same whole school reform model. Challenges of selecting the components of the mixed-methods design, data triangulation, and priority and timing of the quantitative and qualitative strands are discussed in this case.
After reading this case, students should be able to
- Identify and describe key components of mixed-methods research design
- Understand triangulation and cross-validation of data
- Understand the importance of timing and sequencing of data collection and analysis
This case study discusses the process of designing research around urban school capacity for reform as experienced by both school leaders and teachers. I discuss several aspects of research design, including my decision to utilize a mixed-methods approach. My interest in studying the capacity of urban schools to implement reform came directly from my own professional experiences as a school administrator.
At the time of this study, I worked in one of the largest urban school districts in the northeastern region of the United States, which consequently was also located in one of the poorest cities in the United States. Poor school districts in this state were required by the State Department of Education to implement whole school reform models in every school to improve the academic performance of its students. This district, like other urban school districts across the United States, was not very diverse in terms of ethnicity and socioeconomic status but was extremely diverse in student academic achievement. The biggest challenge plaguing this school district was the consistent improvement of academic achievement for all of its students.
After 3 years of school reform implementation, improved academic achievement was not achieved by all students as expected. After having spent millions of dollars to implement whole school reform with mixed results, the school board and community demanded answers for its failure. Central office staff blamed the school-level staff for the failure of the school reform models to improve student academic achievement, and school-level staff blamed the central office staff for the failure of the school reform models to improve student academic achievement. These two divergent opinions as to who was at fault for the failure to successfully educate all students in this poor school district ignited my interest to engage in the research described herein. It was important to explore why all students in this district did not perform well as expected after the implementation of whole school reform models that were designed to improve the academic achievement of poor public school children. Where had this district gone wrong?
The research on student academic achievement and student socioeconomic status was clear: schools with high concentrations of students who are poor tend to experience higher rates of student failure on standardized exams than do schools with student populations that are not poor. Ironically, the district where I worked primarily comprised poor children of color. How was it possible that student academic achievement differed so greatly in schools located in the same school district when implementing the same whole school reform initiatives, with the same resources and with the same student populations in terms of ethnicity and socioeconomic status? I needed to explore this phenomenon from the field—the schools—to get a better idea as to where the school district may have gone wrong, to identify what had been missed.
Thinking through the Research
To gain an authentic perspective of the inconsistent academic achievement of students who attended schools that implemented whole school reform models, I needed to go directly to the source. I believed that teachers charged with implementing the whole school reform models in their classrooms would be the best group to provide me with information about the challenges and successes of implementing school reform. To gain insight from a large number of teachers, I believed the use of a survey could adequately capture their experiences. As I thought more about the research, I realized that the survey would provide only one side of the narrative of school reform implementation in this school district. What about the budgets and staff development? What about teacher quality? What about principal support? What if I needed clarification or verification of information provided by the teachers, how and where would I find that validation? It became glaringly apparent that I could not rely on one source of data or one stakeholder's (teacher or principal) perspective of their school's capacity to implement reform. In fact, I needed multiple stakeholder perspectives to provide an in-depth picture of the implementation of school reform at the school level and varied data sources to triangulate my findings.
An important aspect of this research focused on school capacity to implement reform, and the availability and use of resources to implement reform. There were several data sources that could provide information about reform implementation at each school; however, no single data source could provide the complete picture. For example, when exploring the availability of school resources to implement reform, the school budget provided me with the amount of monies that were allocated for instructional resources for reform. However, the budget alone could not provide me with the actual allocation of those resources within the school building to the teachers. It also could not tell me the school-level processes for resource distribution or teacher perception of the adequacy of resources available to implement reform. Interviews with the principals could explain the resource allocation process and budget justifications for the amounts allocated in the budget for instructor resources within each respective school but could not provide actual proof of teacher receipt of instructional materials or the teacher perceptions of the adequacy of resources. Teacher surveys provided teacher perceptions of the adequacy of resources provided for reform implementation but not the actual use of the resources, amount of monies in the budget allocated for resources, or the process for resource allocation.
Designing the Research
It was important to identify a research method that would allow me to explore diversity of views, confirm and discover themes, and illustrate findings in order to explain the phenomenon of school capacity to implement reform in this school district. I was not an expert in either quantitative or qualitative research. I had been briefly introduced to quantitative research during my doctoral studies and was gently directed away from qualitative courses because of the assumed length and volume of work required when engaging in qualitative inquiry. I was even more removed from the philosophical debates on the merits and validity of mixed-methods research. I knew that I would need to look at quantitative data such as school district spending per pupil, monies allocated by the school principals, and their respective school budgets for the operation of their buildings. These elements could be discovered by using quantitative research methods. I also knew that I wanted to have a better understanding of choices that school principals make when implementing reform. This would be better obtained using qualitative research methods.
It was my belief that a combined methodological approach would allow me to better understand the implementation of reform in schools. Present in many debates on mixed-methods research is the belief that one method is better than the other, thus diminishing the value of one. In my perspective, both the qualitative and quantitative methods of data collections and analysis would be weighted equally. Mixed methods would be used to achieve triangulation of the data collected for this study. My findings would be strengthened by collecting and analyzing both qualitative and quantitative data. According to Joseph Maxwell and Diane Loomis (2003), a researcher should weigh five interconnected components when designing a mixed-methods study: the study's purposes, conceptual framework, research questions, methods, and validity considerations. I followed Maxwell and Loomis's plan.
I hoped to identify themes or patterns on how schools used resources to implement reform as this may have had an impact on the school's capacity to achieve the goals of school reform to raise student academic achievement for all students. I knew that I needed more than just numbers; I would need to provide thick descriptions and capture the essence of each school's process in implementing school reform.
As is common with doctoral students and novice researchers, I struggled with the literature review. I searched zealously in library databases, EBSCO, and ProQuest to identify relevant literature. I used search terms such as whole school reform, school reform, urban school reform, and comprehensive school reform. I utilized search strategies described by Jane Webster and Robert Watson in 2002: backward and forward author and citation searches. Backward author searches helped me to identify older published works by the same author; perhaps the authors' position about my topic had evolved or even changed over time. Forward reference searches involved searching citations to determine whether the author's research was cited in articles by other researchers. I reviewed bibliographies to see whether I had missed key studies, articles, reports, or books on this topic. My search efforts returned extensive literature in various formats: books, book chapters, journal articles, white papers, dissertations, conference proceedings, foundation, and government reports.
As a result of these literature search strategies, the number of references I had grew exponentially. To my amazement, I discovered that there were decades of research on school reform initiatives, documenting successes and failures of hundreds of schools nationwide. Even though I had hundreds of articles on whole school reform, I still had not narrowed the focus of my research. Persistence and diligence were necessary when reviewing materials during this discovery process. I could not be deterred by my feelings of doubt and being overwhelmed. I expanded my key word searches in the databases to include examples of reform model implementation requirements, type of school, and timelines. The inclusion of these new search items resulted in the identification of two very important themes:
- the importance of school capacity prior to reform implementation
- the importance of school capacity developed as a result of reform
The first of these themes was that school capacity to implement reform could affect a school's ability to implement reform, which appeared to be very insightful. The oxymoronic practice of the federal government, state education officials, and local school districts mandating schools to implement reform initiatives without regard to school ability or capacity perplexed me and piqued my interest to delve deeper. The research reports in the literature provided evidence to support this claim that schools that were effective in implementing reform appeared to have capacity to implement the reforms. On the other hand, in reports of failed school reform—where student academic achievement did not improve—research reports often cited numerous capacity deficiencies, which were barriers to success. Thus, my assumption was ‘Schools with the most resource capacity prior to reform were more successful than those with less resource capacity’. Given the barriers and successes reported by schools when implementing school reform, I chose a conceptual framework that viewed the implementation of school reform via a ‘capacity’ lens. Researchers Betty Malen and Jennifer Rice (2004) provided a much broader perspective on school capacity than other frameworks found in the literature on school capacity to implement reform. They believed that the first step in assessing school capacity is to look at the resource requirement of the reform initiative to determine whether the school's available resources and the resources required for the implementation of reform are aligned. The conceptualization of this theory provides a framework that supports the inventory approach and the analysis of resource utilization, which, in turn, provided a framework to examine how policies mediate the capacity of schools to achieve the reform outcomes, given their available resources.
The interaction of the two dimensions of the framework developed by Malen and Rice in 2004—the resource base and the productivity base—was key to determining the impact of school capacity to implement reform in this research. Why do some schools that appear to have the same resources and capacity to engage in school reform improve their student achievement after reform whereas others do not? The framework developed by Malen and Rice (2004) provided me measurable concepts and constructs to examine each school's capacity to implement reform by looking at available resources, as well as the use of those resources within the reform context. Given the reported equity of resources for reform implementation, I chose to investigate those differences in school capacity and the utilization of school resources while implementing reform. Specifically, I asked the following questions:
- What was the capacity of the schools at the time of this study with respect to their fiscal, social, cultural, information, and human capital?
- Did the schools display evidence that resources were productively aligned with the reform effort, or is there evidence of organizational freneticism and fragmentation?
- Did schools appear to differ in their capacity, including their use of resources, in ways that may have had an impact on the implementation of the district reform?
- Do the data help us understand whether the same reform initiative was equally well suited to two different schools?
The selection criteria for schools to be included in this research were as follows: school must have implemented the same reform model, had the same per-pupil expenditures, similar student demographics, and differing student academic outcomes. The identification of schools that met the aforementioned criteria was daunting. The demographics of the district where the schools used in this study were located are as follows:
- 70 elementary schools implementing the same school reform
- 85% African American
- 11% Hispanic
- 3% White
- 1% Asian
- poverty levels: 50%-100% of students are classified as living in poverty
- passing rate on state-mandated standardized exams: 20%-100%
- Schools are racially segregated
Historically in this district, schools with higher percentages of poor African American students performed significantly lower on state-mandated standardized exams than did students from higher socioeconomically advantaged homes. Due to the lack of economic diversity of the student population in this district, schools that would be included in this research would have to be purposefully selected. Thus, a nonrandom sampling method was employed to identify schools that met specific criteria, which would allow for me to answer the research questions posed for this research. Given these parameters that I set for school selection, researchers Kathleen Collins, Anthony Onwuegbuzie, and Qun Jiao (2007) advocate for the use of the critical case sampling schematic for mixed-methods research, which allows the researcher to select settings, groups, and/or individuals for research based on specific characteristics. Thus, the critical case sampling schematic was instrumental in assisting me with identifying schools that had similar students with different academic outcomes that had implemented the same reform model within the same school district. Additionally, the use of critical case sampling assisted me in minimizing the impact of extraneous confounding variables of student race and poverty not included as part of this study on the relationship between the implementation of school reform and school capacity.
Using the critical case sampling schematic recommended by Collins, Onwuegbuzie, and Jiao (2007), I was able to select schools for this research that could provide the best information about school capacity to implement reform in academically contrasting schools, which would allow me to achieve the objectives of the research. The use of independent and dependent variables as identifiers for this research was essential in the selection of schools that were identified for this research.
In the first stage of selection process, I identified all elementary schools in this district that had implemented the same school reform model at the same time. Next, I used the dependent variable ‘academic achievement’ to categorize the elementary schools in this district as high or low performing. The determination of high- and low-performing designation was based on criteria set by the State Department of Education. According to the State Department of Education criteria for categorizing schools, a school that failed to meet pre-established state academic benchmarks for student achievement for two or more years, as solely evidenced by student passing rates on state standardized test scores, would be labeled as low achieving and the schools that met the state benchmark for a year or more were labeled high achieving. The third step in this identification process consisted of reviewing two independent variables—‘ethnic composition’ and ‘student poverty’—to determine student demographics of each elementary school in this district. This step allowed me to identify and group schools with similar student populations that also had contrasting academic achievement. At the conclusion of this selection process, two schools were identified as meeting the criteria for this study.
Each of the two schools in this study was located in a low-income high-crime neighborhood serving approximately 400 students who were predominately African American; both schools had 50% or more of its students receiving free or reduced lunch (students who qualify for free or reduced school lunch have family incomes that are at or below poverty guidelines and thus are identified by schools as poor); both schools were required to implement the same whole school reform model, the same per-pupil expenditure, with the same required instructional materials, professional development, and staffing mandates. The two schools differed with respect to student academic achievement, an essential characteristic for this study. One school had never been identified as a low-achieving school and the other school had been identified as a low-achieving school for three consecutive years. At the time of this study, each school had had the same principal for the previous three school years.
The question now was ‘How could I capture the resource base and use of resources for each school in the research study? What data were needed to be collected to achieve the goals of this research?’ I would need information from both the teachers and school administrators as well as information from public and school records on student academic achievement and budgets. An important tenet of mixed-methods research is the cross-validation of data. Triangulation of the data was important for validating the research findings. What data sources could be used to validate and corroborate my findings? Four sources of data would allow me to capture the essence of each school's capacity to implement reform: teacher survey, school documents, public records, and principal interviews.
The next step in this process was to determine in what order the data would be collected and analyzed. Both qualitative and quantitative strands of this research were of equal importance; thus, I had to decide at which point in the research process the data for each would be collected and analyzed. There are three options for data collection in mixed-methods research: data can be collected concurrently, that is, both qualitative and quantitative data are collected at the same time; sequentially, the qualitative and quantitative data are collected in phases and multiphase; the data are collected both sequentially and concurrently in multiple phases. Would data from one source help to inform information needed from another data source? When would I be able to validate my findings with data triangulation? In deciding the timing and analysis of my three data sources, I considered the following:
- The teacher survey (quantitative) had already been developed and was aligned to the conceptual framework. Survey questions would not be changed as a result of information provided from the principal interviews or information collected from the review of public and school records. However, after the analysis of responses from the teacher survey, findings from the survey may need to be triangulated by using the content contained in the public and school records as well as information obtained from the principal interviews.
- The content (qualitative) of the information contained in the public and school recordswould not be impacted by the administration of the teacher survey or principal interviews. However, after the coding of content contained in the public and school records have been grouped according to themes, the themes may need to be triangulated from the data obtained from the teacher survey or principal interviews.
- The questions (qualitative) asked of the principals will be impacted by the review of school and public documents as well as from the results from the teacher survey. The principals will be the only persons during this research whom I engaged with who will be able to provide explanations or further insight for themes identified from the teacher survey and document review.
Clearly, data obtained from the teacher survey and review of public and school documents would inform the questions that would be asked of the principals during the principal interviews. The principal interviews were semi-structured and open ended. It was important that the principals answer essential questions about capacity as outlined in the conceptual framework. However, ‘wiggle’ room was needed to be able to ask clarifying questions about information obtained from the review of public and school and records and teacher perceptions of the adequacy of resources available in the school. Therefore, the principal interviews should occur as a second phase of the research.
I decided to concurrently collect the public and school records data and survey data during the first phase of this research. Requests for school-level data were made to the school district via the Open Public Records Act. By law, the school district had 30 days to provide the requested documents for this research. Other data such as student achievement information and school reports cards were accessed from the state education website. As I waited for the school district to provide me with the requested documents, I administered the teacher survey. After the survey responses and public and school data had been collected and analyzed, I was able to draft questions for each principal, which helped me to compile a complete narrative of the capacity of each school in this study to implement school reform. In utilizing mixed-methods research, I successfully gleaned data from three sources: survey, school document and reports, and principal interviews. I obtained demographic data about each school's community, student data, itemized budgets, grade-level meeting agendas and minutes, school management team agendas and minutes, instructional priorities plans, teacher perceptions of their colleagues' pedagogical abilities, strengths and deficits, teacher perceptions of the availability of school resources and principal perceptions of their teachers, resource use, and reform implementation.
Research Findings: Did Capacity to Implement School Reform Matter?
Decades of research on successful schools implementing whole school reform highlight the correlation between school capacity and the implementation of reform. The capacity levels of a school will influence the ability of the school to implement reform to achieve its goals.
This study's findings suggest that the interaction of capacity and school reform looked different for each school. The most notable differences between the two schools while implementing the whole school reform model were in per-pupil allocations, budget allocations, staffing quality and quantity, cultural responsiveness, teacher skills and knowledge, collective teacher efficacy, teacher expectations of student academic ability, professional development activities, and principal expectations of students and staff. The data also reflect interesting differences in decisions made by the principals on the utilization of school resources to implement the reform, which may be attributed to differing leadership styles. The theoretical framework used to guide this study did not specifically address principal leadership, and principal leadership was not a focus of this study. However, two themes—decision making and principal leadership while implementing school reform—emerged during the interviews with each principal.
Lessons Learned from This Research Design: Missed Opportunities
When engaged in mixed-methods research, the researcher must take care to identify all possible data sources, collection methods, and the role of that data in validating or triangulating findings. A firm understanding that there is no single prescribed model of mixed-methods research would help a researcher, especially a novice researcher, in planning creative and comprehensive strategies in collecting and analyzing data.
Mixed-methods research design provides extraordinary opportunities for researchers to examine topics of interests from multiple perspectives utilizing both quantitative and qualitative data. For the research presented in this case study, I placed a stronger emphasis on collecting quantitative data than qualitative data. Thus, opportunities to further explore teacher experiences in implementing reform were limited to only the data obtained from the teacher survey. In mapping out the data to be collected and the methods to collect the data for this research, I missed opportunities to follow up with the teachers to further explore their thoughts and to observe authentic practices of the implementation of the reform.
The findings from this research suggest that teachers in both schools differed in terms of student expectations, teacher education and skills, teacher perception of the availability of instructional supplies, teacher perception of program coherence, collective teacher efficacy, engagement in deprivatized practices, and culturally responsive pedagogy. By failing to plan for additional opportunities to explore teacher engagement in and understanding of the reform being implemented in their schools, the teachers' ‘voice’ obtained from one data source—the survey—was incomplete and superficial. Additional data that would have provided a more complete picture of the teachers' experiences in implementing reform could have been obtained from observing teacher classroom practices, teacher participation in grade-level meetings, teacher participation and belief in staff development sessions, and conversations with teachers. The collection of data from the aforementioned sources would have helped me to better understand whether differences identified in the research could have been attributed to policies from the reform implementation, teacher beliefs, teacher capacity and disposition, or principal leadership.
Other missed opportunities resulted from my failure to consider the perspective of other key stakeholders who were also impacted by the implementation of school reform: parents and students. Given the design that I selected to use to research the implementation of school reform in contrasting schools, numerous opportunities via surveys, focus groups, and interviews with parents and students were not considered. Parents and student voices would have strengthened my research, providing a more thorough perspective of the impact of the reform implementation and stakeholder beliefs of school capacity to implement the reform. More inclusive data collection planning to explore student experiences could have included observations of students during instructional time and small focus groups. In addition, data could have also been collected during parent–teacher conferences and open houses to provide information on parental perceptions and concerns about their children, the reform initiative, and school capacity to implement reform to improve student achievement.
Finally, the importance of the literature in the identification of a conceptual framework should not be underestimated, disregarded, or discounted; a thorough review of the existing literature is immeasurable in providing the road map between the research questions and the research findings. The literature review provides a foundation for the proposed research. An exhaustive literature review will provide a researcher with a thorough road map for conducting research, inclusive of numerous constructs and varied methodologies to explore as well as the identification of the roads most frequently and least explored for the research topic. The data from the research presented here suggest that principal leadership and decisions made by each school principal may have attributed to differences in each school's resource base and the uses of the school's resources. In hindsight, principal leadership should have been analyzed as comprehensively as teacher quality when looking at human capital and capacity to implement reform. A more thorough analysis of the research base via the literature review would have produced theories that connected school capacity to implement reform to principal leadership. Researching the leadership style of the principals in this study and the schools' capacity to implement reform would have complemented the findings from the study, providing a more complete picture of school capacity to implement reform. Oftentimes, school principals are required to make decisions about staffing and budgets that have a tremendous impact on reform implementation and school capacity. Having the opportunity to explore the implementation of the reform initiative over a school year by observing principals facilitating staff meetings; the planning and execution of school improvement plans; engagement with teachers, students, parents and school community; and strategies used to address challenges would provide more data about the leadership of each principal while implementing reform.
Novice researchers in their zeal and their excitement to begin research prematurely, moving very quickly in the research proposal development, may fail to identify key data sources that would strengthen research findings and conclusions. Researchers, particularly novice researchers, should move cautiously and slowly when planning research methods to ensure as much as possible that all data sources have been explored, which can provide a more authentic comprehensive understanding of the phenomena being explored.
Exercises and Discussion Questions
- Identify a social problem that you might explore via mixed-methods research. What data should be collected and why? How should the data be collected and why? Explain why mixed methods would be the better option for conducting your research than using either qualitative or quantitative methods alone.
- Review a peer-reviewed mixed-methods research article. Identify the components of the mixed-methods design as provided in the article. Debate the merits of this design to address the research questions included for this research.
- Review a peer-reviewed mixed-methods research article. Identify the data collection method (e.g. concurrent, multiphase). Create a graph delineating each step of the data collection phase. Provide a succinct description as to the merits or flaws of the researcher's choice in the data collection process.
- Create a pros-and-cons chart that succinctly describes the positive and negative benefits of mixed-methods research.