Mixed Methods: Is This Appropriate for My Research?

Abstract

My DPhil research was a community-based mixed-methods study using cross-sectional study and focus group discussions to analyze the effect and the pathways of effect of households' and mothers' participation in an Indian wage-for-employment scheme on infant malnutrition. It was complex primarily because it required different methods to answer the research questions. Using mixed methods allowed triangulation of quantitative and qualitative methods for greater validity and completeness, thereby providing a comprehensive answer to the research questions. In this case study, I justify the use of mixed methods to answer my research questions and explain the selection of the design, prioritizing and weighting, and mixing of the quantitative and qualitative components. I also discuss the triangulation of the results of the cross-sectional study and focus group discussions. In addition, the case study includes a reflection of the practical lessons learned from the implementation challenges of a mixed-methods study and demonstrates that the overall research output of the mixed-methods approach was much more than the sum of its individual components. Using mixed methods, rather than a classic quantitative or qualitative method, can help a PhD student learn more, but the approach is resource-intensive, and therefore, it is important to justify the appropriateness for using mixed methods.

Learning Outcomes

By the end of this case, students should be able to

  • Understand the decision-making process involved in using a mixed-methods approach
  • Have an understanding of how the quantitative and qualitative components should be prioritized, weighted, and mixed in case of a parallel convergent design
  • Appreciate the practical challenges involved in conducting mixed-methods research
  • Assess whether their own research project requires the use of mixed methods

Project Overview and Context: Effect of the Mahatma Gandhi National Rural Employment Guarantee Act on Infant Malnutrition: A Mixed-Methods Study in Rajasthan, India

Recently, a few first-year DPhil (PhD) students came up to me and mentioned that they were interested in using mixed methods to answer their research questions. My response to them was “Is mixed methods appropriate for your research?” “Why do you need two different methods?” “Will your research outputs be enhanced by adding another method? If so, to what extent?” My DPhil project, which I completed in October 2013, used mixed methods to answer the research questions. I did not draft my research questions to somehow use mixed methods; instead, my research questions guided the selection of this methodology. The part of my research question that examined the effect of an intervention required a quantitative approach, and the part that explored the mechanisms that led to the observed effect required a qualitative approach. Mixed methods is not just using quantitative and qualitative methods in a program of work, but it involves the integration of two very different types of methods to produce a piece of work which is much more than the sum of its individual parts.

My DPhil research (October 2010 to October 2013) was a community-based mixed-methods study using cross-sectional study and focus group discussions to analyze the effect and the pathways of effect of households’ and mothers’ participation in an Indian wage-for-employment scheme on infant malnutrition. Wage-for-employment schemes are based on the policy of participation income, which is paying people for socially useful work (Atkinson, 1996; Besley & Coate, 1992). Malnutrition is a major risk factor for infant mortality in India. The Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA), a wage-for-employment policy of the Indian government, targets deprivation and food insecurity in rural households. My research hypotheses were as follows: MGNREGA could prevent infant malnutrition by improving household food security or could increase the risk of malnutrition by reducing the time devoted to infant care if mothers are employed.

Being a wage-for-employment policy, MGNREGA cannot directly affect infant malnutrition, but it could act by influencing other determinants. Therefore, not only was it important to analyze the effect of MGNREGA on infant malnutrition, but it was also important to understand the mechanisms and pathways through which it may assert any effect. My research questions were as follows:

  • What is the effect of MGNREGA on infant malnutrition?
  • What are the pathways through which MGNREGA affects infant malnutrition?

I conducted my study in the Dungarpur district of Rajasthan, a state in the Western part of India, in which MGNREGA had been implemented since 2005-2006. Briefly, Dungarpur is the smallest district of Rajasthan located in the southern plains of the state. The district is mainly inhabited by people belonging to the Schedule Tribe and Schedule Caste social classes who are socially and geographically isolated. Dungarpur is a sub-montane region and has recurrent droughts. Agriculture, which is seasonal, is the major source of livelihood. More than two-thirds of the households in Dungarpur are employed in casual labor. Thus, one of the major challenges in this district as identified by the Government of Rajasthan is seasonal unemployment and more than half of the total households live below the poverty line.

The United Nations World Food Programme ranked Dungarpur as an extremely food-insecure district which corresponded with the high proportion (57%) of underweight children below 5 years of age. Health and nutrition of infants were poor despite the district being considered one of the better-off districts of Rajasthan with regard to access to health care facilities. About 34% of the villages in Dungarpur have access to the Primary Health Centers.

The MGNREGA scheme has been implemented in Dungarpur district since its inception using recommended standards. Therefore, this district was selected as the study site to avoid differences in the anthropometric outcomes in the MGNREGA-participating and non-participating households being attributed to shorter duration and less-than-standard implementation rather than to the nature of the policy (Nair, 2013). Furthermore, as discussed earlier, the employment, and child health and nutritional challenges in the district are such that the district is likely to benefit from this scheme.

Why Did I Choose Mixed Methods?

My research was not merely about testing the effect of MGNREGA on infant malnutrition, but it involved the understanding of human behavior, household conditions, culture, and context which could modify the effect of MGNREGA on infant malnutrition, and the hypothesized positive and negative mechanisms of effect (Box 1) made this research even more complex. A classic quantitative or a qualitative study would not be able to emphasize the interplay of subjective and objective knowledge that was required to answer my research questions.

The following process guided my decision-making process for using mixed methods:

  • My first task was to develop a conceptual framework of the pathways and mechanisms through which MGNREGA could have an effect on infant malnutrition (see in Figure 1).
  • My next task was to identify whether all the factors included in the framework could be measured using quantitative methods and to identify instruments and tools that could be used to collect data to measure those factors. During this process, it became clear that some factors—such as discrimination, cultural practices, social status of women, and mothers’ knowledge and attitude toward infant feeding (highlighted in “red boxes” in Figure 1)—could not be measured using a quantitative tool, but required in-depth discussion with mothers of infants to understand them.
  • To understand the complex pathways, quantitative and qualitative methods are needed to measure and explore different components of the conceptual framework, and the results need to be triangulated to develop a holistic picture. Such an approach has been recommended by the framework of the Medical Research Council of the United Kingdom to evaluate complex interventions (Craig et al., 2008).
Figure 1. Conceptual framework of the pathways of effect of MGNREGA on infant malnutrition (adapted from Nair, 2013).
Figure

How Was the Study Designed?

The conceptual framework and the research questions guided the mixing of qualitative and quantitative components, and the methods of inquiry (data collection, analysis, and interpretation). Figure 2 from my PhD thesis (Nair, 2013) summarizes the process that I used to mix the methods of inquiry at each stage.

Figure 2. Diagrammatic summary of the mixed-methods approach used in the study.
Figure

John W. Creswell and Vicki L. Plano Clark (2007) suggest that mixing should occur at all stages of the method of inquiry and not just at the stage of interpretation. The framework and research questions also guided the planning of the sequence in which the quantitative and qualitative studies will be conducted and the priority of each method.

There are two main mixed-methods approaches: sequential and parallel. In a sequential approach, a qualitative study is followed by a quantitative study, or vice versa. For example, a researcher might use focus group discussions to develop a data collection instrument or questionnaire which is then used in a quantitative survey. Sequential design can also be used to conduct in-depth interviews or focus group discussions to explain the findings of a quantitative research. I decided to employ a convergent parallel mixed-methods approach comprising equal-weighted quantitative and qualitative strands that were implemented concurrently (Creswell & Plano Clark, 2007, 2011, Doyle, Brady, & Byrne, 2009), with parallel data collection and analysis (see Figure 2). The components were given equal weighting and applied in parallel because each measured specific factors mediating the effect of MGNREGA on infant malnutrition. For example, although a quantitative study would measure food security, socioeconomic status, and anthropometric indicators of infants’ nutritional status in the MGNREGA-participating and non-participating households, a qualitative study was required to understand the knowledge, beliefs, and cultural factors that influenced infant feeding practices and to explore the perception of mothers employed through the MGNREGA about the benefits of the program and any associated compromises to infant feeding and care.

The quantitative strand for my study was through a cross-sectional study, and the qualitative component was through focus group discussions. The analyzed data from each strand were triangulated through integration and correlation to interpret the findings and finally develop empirical models linking MGNREGA and infant malnutrition (Figure 2). This process enhanced both representation and legitimation, which are key principles of validation in mixed-methods research. Anthony J. Onwuegbuzie and Charles Teddlie (2003, p. 353) define representation as “the ability to extract adequate information from the underlying data” and legitimation as “the validity of data interpretation.”

Quantitative Cross-Sectional Study
Study Population and Sample Size

Households that had an infant in the age group of 1 to <12 months (based on the records of the village nurse) were eligible for inclusion in the study. The selected households were divided into MGNREGA households and non-MGNREGA households based on participation in the MGNREGA between August 2010 and September 2011. Sample size was calculated mathematically using a standard approach and adjusted for clustering at the level of the selected 44 villages spread across the district. A cluster sampling design could have implications for the assumption on independence of observation and thus on the validity of the results, as it is likely that individuals within a cluster are more similar. To minimize this clustering effect on the study outcomes, it is important to estimate the cluster effect and adjust for this during sample size calculation, which essentially inflates the sample size.

Sampling Method

I used a single-stage cluster random sampling design to select the required number of households from the 44 villages which were in turn selected by generating a random list using the statistical software Stata, version 11 (StataCorp, TX, 2009). All households having an infant in the age group of 1 to <12 months were identified from the records of the local village nurses in each of these randomly selected 44 villages and were invited to participate in the study. A total of 1,102 participants in 551 households (551 pairs of mother and infant) were finally included in the cross-sectional study.

Tools and Methods for Data Collection

I developed a paper-based questionnaire to collect information from the mothers/caregivers about the various factors that could influence infant nutrition based on the conceptual framework. The tool was tested in a pilot study in 20 households (20 mothers and 20 infants) in a village. The questionnaire was administered verbally to the mothers of infants, who provided informed consent, by me and three trained field assistants (graduate female nurses). The interviews and the anthropometric measurements were carried out in the village health and nutrition centers.

Qualitative Focus Group Discussions

I used focus groups to generate themes and interactions through group discussion with mothers of infants aged 1 to <12 months to explore the proposed mechanisms of the effect of participation in MGNREGA on infant malnutrition. The advantage of focus group discussion, as noted by Jenny Kitzinger (1994), is that it is an interactive process of generating data, and David L. Morgan (1997) and Amber Wutich, Lant, White, Larson, and Gartin (2010) suggest that it is more effective than in-depth interviews when the topic is not sensitive but is socially relevant.

During the study, it became evident that this method was the right choice as discussions brought out various socially relevant topics, including oppression of women and how women were forced to work. It was one woman who initially spoke about women with young children being coerced by their family through verbal and physical abuse to seek employment through MGNREGA. This encouraged other women to speak up about coercion, some explicitly and others agreeing with their fellow participants by nodding and smiling (Nair, Ariana, & Webster, 2014).

Study Population and Sample Size

The study site was the same as that for the cross-sectional study, and focus group discussions were conducted in all the five administrative blocks of Dungarpur district. Unlike the cross-sectional study in which sample sizes were mathematically calculated to power the study for internal validity, the sample size for the focus group discussion was based on expert opinions, published literature, research question, within- and across-group diversity, structure of the interview guide, and point of saturation of information.

Sampling Method

A purposive sampling method was used to recruit participants for the focus group discussions. This was done to include women who were employed through MGNREGA and who were not, and women from all social class and educational backgrounds to understand whether their perceptions varied by these factors. Participants were a subset of the randomly selected participants for the cross-sectional study, known as a nested sampling approach. Eleven focus group discussions (including the pilot study) were conducted with 65 participants, 62 mothers of infants, two local village nurses, and a nutrition worker.

Tools and Method for Data Collection

I used a semi-structured guide with probes to conduct the focus group discussions. This was mainly used to direct the discussions and to focus on the factors highlighted in the red boxes in Figure 1. I tested the semi-structured guide in a pilot study conducted with six mothers of infants and adapted the probes used to capture the emergent themes as the discussions progressed. Memo and code books in the form of diary notes of tentative ideas for codes, patterns emerging during the discussions, interactions between participants, and possible themes were maintained, and all focus group discussions were recorded, based on informed consent, using a voice recorder for maximum transparency.

Practical Aspects of Study Implementation, Challenges, and How I Was Able to Overcome Them

Although from Figure 2 it may seem like I had an immaculate plan from the beginning of the DPhil project, in reality, it felt more like a jigsaw puzzle that came together at the end of the project. I concentrated on one stage at a time (design, data collection, analysis, and interpretation) as it was difficult to visualize the end product. The challenges and how I was able to address them are described in detail in the following paragraphs.

Theoretical Conceptual Framework

My research started with developing a theoretical conceptual framework by conducting a thorough review of literature to understand the risk factors for infant malnutrition, the inter-relationship between them, and pathways through which these may mediate the effect of a wage-for-employment scheme. As discussed earlier, this framework not only helped in hypothesizing the pathways of effect but also helped to segregate factors that could be measured using a quantitative method and those that would require a qualitative approach which informed the convergent equal-weighted parallel design of the mixed-methods approach (see Figures 1 and 2).

Quantitative Data Collection

Primary data collection requires extensive planning and resources. I spent more than 10 months planning my fieldwork, getting necessary approvals from the local ethics committees, recruiting field staff, organizing the logistics (printing, travel, storing data collection forms and audio tapes in a safe and secure place), and finding a place to work and stay during the field work. The process of selecting the villages was meticulously completed based on probability proportional to size before I left for fieldwork, but on arriving at Dungarpur district, I realized that this would not work. Several villages in one administrative block of the district were flooded during the period of fieldwork and were difficult to access, so fewer villages were selected from this block than originally planned. Consequently, to meet the required number (44 clusters), more villages had to be selected from the adjacent blocks.

Before starting data collection, I pilot-tested the questionnaire to check construct validity. During the pilot phase, I found that two out of the three instruments used to measure household food security were not useful for the study population. The participants were unable to respond to several questions. They found it difficult to recall the details about access to food in the past 1 month and availability of food in the household in the past 12 months. In addition, the “General Health Questionnaire-12” incorporated in the participant questionnaire to assess the mental health status of the mothers was found not to be useful. I observed that a majority of the women were not able to understand the questions despite the fact that the “General Health Questionnaire-12” has been translated into a number of Indian languages and validated. So, I removed these sections from the participant questionnaire before the start of the study.

It was difficult to conduct the anthropometric measurements, mainly measuring the length of small babies which required the baby to be still with knees properly extended, and thus several had to be repeated. Some mothers required a lot of explanation to convince them that the measurements would not harm their baby. This and other factors like floods, problems with field staff, distance, and lack of proper roads to many villages increased the time period of data collection.

Qualitative Data Collection

It was relatively easy to conduct the focus group discussions in terms of recruiting study participants because these followed the quantitative study. Women who completed the quantitative study were approached to participate. Separate informed consent was taken from each participant who agreed to participate in the focus group discussions. However, although the original plan was to segregate the participants (mothers of infants) based on their employment through MGNREGA to understand the perception of the MGNREGA-employed and non-employed mothers, it was not possible to do so. The reason was that not many mothers were employed through MGNREGA during the study period within the selected villages.

I faced a major challenge during the focus group discussions in initiating discussions among the participants: The women were reluctant to speak and had their faces covered as part of social and cultural practices. In this region, women were required to cover their faces and it was considered inappropriate for women to talk loudly or to voice their concerns. It is important to consider such social and cultural aspects when planning a study.

To overcome this problem and to make the participants comfortable, I conducted the focus group discussions in the village health and nutrition centers. The local village nurse was also allowed to be present during the discussions. This resulted in some prompting by the local nurse or community health worker present at the center. However, I noted these prompts in my memos and took them into consideration while analyzing the themes. Even after using these measures, I observed that one or two participants (usually the educated participants) dominated the focus group discussions and a few of them (mainly young and illiterate women) did not speak at all but simply nodded their heads.

Triangulation of Quantitative and Qualitative Study Results

I used the findings from the cross-sectional study and the focus group discussions to re-construct the hypothetical pathways of effect of households’ and mothers’ participation in MGNREGA on infant malnutrition. However, the major hurdle was to find an appropriate technique to triangulate the findings from the two methods. After lengthy discussions with my supervisors and searching for available methods, I was convinced that a statistical modeling would be appropriate to quantify the theoretical pathways and to triangulate the results.

The themes from the focus group discussions suggested the possibility of negative effects of mothers’ employment on infant nutrition mediated via negative effects on infant care and feeding, and the cross-sectional study suggested a possible positive effect of households employed through MGNREGA on infant nutrition mediated via increase in birth weight. This was used to re-structure the models of pathways of effect of MGNREGA on infant malnutrition. I used the data from the cross-sectional study to test these models using path analysis (Vasconcelos, Varnier, & Fonseca, 1998) and themes from the focus group discussions to explain the results.

Practical Lessons Learned

Research Questions Should Guide the Selection of the Design for Mixed Methods

It is a waste of resources to conduct mixed-methods studies if the research questions do not demand the use of this approach. Finalizing the research questions is important before selecting the design for the mixed-methods research. In my case, it was the research questions and theoretical conceptual framework that guided the design, mixing of the methods, and triangulation of the results, thus making the process systematic.

Resources Required for a Mixed-Method Study Were Much More Than I had Anticipated

As discussed in the previous section, a number of factors led to an increase in the amount of resources required for the study. Although a parallel design enabled the quantitative and qualitative studies to be conducted simultaneously, the resources required were still much higher than would have been necessary if only one method were used. Rigorous planning and a good justification for using mixed methods are therefore important for optimum use of the available resources.

Mixing Should Happen Across All Stages of the Inquiry

The theoretical conceptual framework included a mixture of factors that needed to be measured quantitatively and qualitatively, which informed the parallel convergent design. During data collection, a nested sampling approach was used such that participants for the focus group discussions could be selected from those who participated in the cross-sectional study. This helped in recruiting the appropriate participants in a convenient way. The results of quantitative and qualitative analyses were fed back into the theoretical framework to re-construct the pathways of effect of MGNREGA on infant malnutrition. The resultant pathways were quantified by conducting path analysis using the quantitative data and the results were explained using the themes generated from the qualitative study during the interpretation stage. For example, the quantitative study showed that mothers’ employment through MGNREGA did not have a significant effect on reducing infant malnutrition; this was explained by the qualitative study which suggested that providing employment to mothers could lead to compromises in feeding and care of the infants. Therefore, to have the desired impact, it was essential that child care facilities be available at MGNREGA worksites.

It is necessary to have a flexible mind-set to move between the two different paradigms of quantitative and qualitative methods. I am primarily a quantitative researcher; thus, it took me time to come to terms with the philosophy that underpins qualitative study, mainly adopting a subjective approach to derive deeper meaning of what is happening in a small sample. During analysis, I tended to quantify the themes generated from the qualitative study and struggled with the concepts of validity and generalizability which are important for quantitative research. However, I learned much more from the entire process than I would have learned by conducting a quantitative study alone. I now believe that qualitative research can provide flesh to a skeleton of numbers and that mixed methods is a pragmatic approach.

Quantitative and Qualitative Findings May Not Converge, Which Is Not a Bad Thing

It is important to understand and interpret the findings of the quantitative and qualitative studies separately on their own, as well as to compare and triangulate the results. For example, an important theme that emerged during the qualitative study was that women with young children were forced to participate in the employment scheme (more details can be found in Nair, Ariana, & Webster, 2014). However, in the cross-sectional study, 97% of the mothers who were employed in MGNREGA stated their reason for working in MGNREGA as “self-motivation.” In the absence of the qualitative study, the findings of the quantitative study would have been misleading, suggesting that women were free to choose and perhaps empowered to make their own decisions with regard to participating in MGNREGA. Although the results of the quantitative analysis showed that there was no significant effect of mothers’ employment through MGNREGA on infant malnutrition, the qualitative study and the path analysis explained the reasons behind this observation.

The Overall Research Output of a Mixed-Methods Approach Was Much More Than the Sum of Its Individual Components

The mixed-methods approach allowed me to measure the effect of MGNREGA on infant malnutrition, understand the mechanisms of effect, and delineate the pathways of effect of MGNREGA on infant malnutrition by mixing objective and subjective knowledge. Methodological triangulation also helped to overcome the innate weakness of any one of these approaches alone and to substantiate the findings of both methods.

Conclusion

My DPhil research was complex primarily because it required different methods to answer the research questions. Using mixed methods allowed triangulation of quantitative and qualitative methods for greater validity and completeness, thereby providing a comprehensive answer to the research questions. I used a systematic and logical process while mixing the quantitative and qualitative components, and the mixing was done at all stages of design, data collection, analysis, and interpretation. My DPhil research not only developed my skills in conducting quantitative and qualitative research but also enabled me to observe and understand the limitation of one method associated with the presence of the other method. It helped me gain valuable insights from mixing or integrating methods. In summary, using mixed methods can help a student learn much more than using just a classic quantitative or qualitative method, but the approach is resource-intensive, and therefore, it is important to justify the appropriateness of using mixed methods.

Exercises and Discussion Questions

  • What are the advantages and disadvantages of using mixed methods? Was mixed methods appropriate for my DPhil research? Justify your answer.
  • In the case study, I have discussed the “mixing of methods.” How should this be done?
  • Use the case study to evaluate whether your own research project requires the use of mixed methods, and if so, use one of the mixed-methods approaches to design your project.

References

Atkinson, A. B. (1996). The case for a participation income. Political Quarterly, 67, 6770.
Besley, T., & Coate, S. (1992). Workfare versus welfare: Incentive arguments for work requirements in poverty-alleviation programs. American Economic Review, 82, 249261.
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ, 337, a1655.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: SAGE.
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (
2nd ed.
). Thousand Oaks, CA: SAGE.
Doyle, L., Brady, A.-M., & Byrne, G. (2009). An overview of mixed methods research. Journal of Research in Nursing, 14, 175185.
Government of Rajasthan. (2011, January). BPL Census. Retrieved from http://bpl2002.raj.nic.in/Index.htm
Kitzinger, J. (1994). The methodology of focus groups: The importance of interaction between research participants. Sociology of Health & Illness, 16, 103121.
Morgan, D. L. (1997). Focus groups as qualitative research (
2nd ed.
). Thousand Oaks, CA: SAGE.
Nair, M. (2013). Effect of the Mahatma Gandhi national rural employment Guarantee Act (MGNREGA) on malnutrition of infants in Rajasthan, India: A mixed methods study (PhD thesis). University of Oxford, UK.
Nair, M., Ariana, P., & Webster, P. (2014). Impact of mothers’ employment on infant feeding and care: A qualitative study of the experiences of mothers employed through the Mahatma Gandhi National Rural Employment Guarantee Act. BMJ Open, 4(4), e004434.
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A.Tashakkori & C.Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 351384). Thousand Oaks, CA: SAGE.
UN World Food Programme (WFP). (2009). Food security atlas of rural Rajasthan. New Delhi, India: Institute for Human Development.
Vasconcelos, A. G. G., Varnier, R. M., & Fonseca, F. (1998). The path analysis approach for the multivariate analysis of infant mortality data. Annals of Epidemiology, 8, 262271.
Wutich, A., Lant, T., White, D. D., Larson, K. L., & Gartin, M. (2010). Comparing focus groups and individual responses on sensitive topics: A study of water decision makers in desert city. Field Methods, 22, 88110.
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles