Qualitative Election Study of Britain was a qualitative investigation into people's political attitudes before and after their vote choice for the 2010 general election. We provide a close examination of the leaders evaluation component of this much larger study to present our recommendations for good research design. By ‘research design’, we refer to the entire research process: from development of the research question through to reporting the results of the data analysis. Using the leaders' evaluation component of the Qualitative Election Study of Britain as the research question, we explain and clarify the various aspects of a research design and provide examples of good research practice.
After reading this case, students will
- Be familiar with the various considerations that go into research planning
- View research design as requiring a holistic and iterative planning process rather than a linear one
- Be able to apply the research design structure provided to a research question of their own
As with all good social research, qualitative research requires careful consideration and thoughtful planning. It does not have to be complicated, but it must be rigorous. A lot can be accomplished by taking the time to ask yourself basic questions about your research design and providing direct and clear answers to them. Below we present our recommendations for good research design using the example of the leaders' evaluation component of the Qualitative Election Study of Britain (QESB), a qualitative investigation into people's political attitudes before and after their vote choice in the 2010 general election. Part of our aim in producing the QESB was to provide full transparency as to how the research was conducted. To achieve this, documents from each stage of the QESB are made available on the project's website. (During the course of this chapter we will refer to specific documents that are available on the website under Creative Commons licence as examples of how to conduct political qualitative research.)
It is not uncommon for the terms ‘study design’ and ‘research design’ to be used interchangeably. We do not; we use the term ‘research design’ to refer to the entire research process, from development of the research question through to reporting the results of the data analysis. We use the term ‘study design’ to refer to planning for the particular needs of data collection such as interviews, focus groups or ethnographic research. The research design for each research question in the QESB can be broken down into the following planning phases:
- developing the research question
- identifying the method of data collection in conjunction with methods of data analysis
- developing the study design
- ethics and planning the data sharing
- data analysis and research output
The aim of the QESB was to record and analyse the views and concerns of British citizens before and after the 2010 General Election. To accomplish this task, we had to narrow down our research questions from all the possible political topics and identify which were of greatest research interest. One of the topics we decided to investigate was people's perceptions of the main political party leaders. None of the party leaders in the 2010 election had led the party through a national election before; therefore, we felt it was important to investigate people's perceptions of Gordon Brown (the Prime Minister of the United Kingdom and leader of the Labour Party at the time of the 2010 general election), David Cameron (the leader of the Conservative party) and Nick Clegg (the leader of the Liberal Democrats) before election day. The research question we identified to guide the development of these leaders' component was as follows: What are potential voters' perceptions (positive, negative and neutral) of the three main party leaders (plus the national party leaders in Scotland and Wales)? With a clear research question in mind, we could start developing the other elements of the research design.
Once a clear research question had been identified, we could consider which form of data collection would be most appropriate to generate the data we needed to answer that question. An important part of this decision was determining the methods by which the data would be analysed. Data collection and data analysis are two sides of the same coin, since the format the data take will determine which methods of analysis can be applied. Data collection has two components: the method of data collection and the format of the data collection instrument. To make a comparison with quantitative research, if your intention is to run a factor analysis to identify latent attitudes, you will conduct a survey and not an experiment (the method of data collection) and the relevant survey questions should be in an ordinal or interval, and not an open-ended format (the format of the collection instrument). The same logic applies to qualitative data collection. Our method of data collection was focus groups. Focus groups were most appropriate because our population of interest was adults in England, Scotland and Wales who were eligible to vote in the 2010 general election. It would have been too difficult and time-consuming to organize and interview more than 70 people in the weeks leading up to the election (and again after the election); ethnography was not an appropriate method for data collection for our research questions.
After selecting focus groups as the method of data collection, we were able to determine the best methods of collecting participants' views of the party leaders in a format suitable to our method of data analysis. We identified grounded theory as the most appropriate method of data analysis to understand people's perceptions of party leaders. Grounded theory method relies on reading, rereading and re-organising the qualitative text in a continuous process of analysis and categorization. The theories that arise as a consequence of this process are said to be ‘grounded’ in the data.
Our data collection and data analysis design were also informed by previous research. Kristi Winters and Rosie Campbell had used focus groups to investigate leader evaluations before the 2005 general election. They employed a two-part strategy which we intended to broadly replicate. Winters and Campbell provided participants with photos of the three main party leaders (Tony Blair, Michael Howard and Charles Kennedy) and, before any group discussion, people were asked to write down the words or phrases they associated with each man. They also asked the participants to note down which of the words or phrases were most important to their views of the leaders. After the brainstorming exercise, participants discussed the words they had written down. The researchers then used grounded theory to analyse both the brainstorming phrases and the discussion text. Winters and Campbell also noted that people were more likely to write positive things about the leaders on their brainstorming sheets, but the tone of the discussions about the leaders was decidedly negative.
We decided there were many benefits to replicating this data collection and analysis. First, it had already been demonstrated to be a good form of data collection that produced qualitative data suited to grounded theory. Second, by collecting leader evaluation data in the same format, we could compare our findings with theirs to determine whether the same leadership characteristics emerged across elections despite the fact that each party had a new leader. Third, we could learn from the experiences of Winters and Campbell about people's written comments being more positive than the discussion. We ensured that the silent brainstorming took place before the discussion. To create discussion space for all types of comments from the focus group participants, we modified the discussion format by asking people positive, negative and neutral comments during the general discussion for each leader. This had the benefit of ensuring that the discussions were not overly negative and provided more data for interpreting the words people had written on the paper. Finally, we could improve upon their research design. This is a distinct advantage when replicating qualitative rather than quantitative research. Survey question wording must be replicated exactly in order to be considered completely comparable. There are poorly worded questions asked in successive surveys, so researchers can analyse change in the results of a poorly worded question over time. Qualitative research provides insights, not generalizable findings, and therefore, it is not constrained to reproduce bad research questions or methods to achieve perfect replicability. This freed us to improve upon the previous research design while still replicating most of it. Winters and Campbell asked people to mark the comments they considered most important. We modified their design by asking people to mark whether the word or phrase they had written had a positive, negative or neutral connotation for them. Since we intended to use grounded theory analysis, this modification would provide more information as to how people perceived each party leader.
As we noted above, knowing which method(s) of data analysis will be used makes the form of data collection easier to select. The leader evaluation brainstorm exercise provided four sources of data for grounded theory analysis: the words people wrote on their brainstorming sheets; the terms they coded as important, positive, negative and neutral; the post-brainstorming discussions; and any spontaneous mentions of the leaders that came up during the course of the focus groups. Although we could not count on the last type of data, we made the deliberate decision to count it as data for our analysis in advance of conducting the focus groups.
Our decision to differentiate between the terms ‘research design’ and ‘study design’ is deliberate and designed to avoid confusion. By ‘study design’, we mean all the specifics of the actual data collection. While it might be tempting to focus on the specifics of the study design when you start your research project, each element of the study design will become obvious if there is a good overall research design. If you have clearly identified your research questions, research question formats and intended method of data analysis, most of the particulars of the study design will be self-evident. We consider the following to be part of the study design:
- study timetable
- budgeting costs
- preparing to run a focus group
- the interview schedule – structure and timing
- piloting the interview schedule
- sampling and participant recruitment
- data recording, transcription, anonymization and dissemination
- preparing materials
- reflection after each instance of data collection
We start with the study timetable because, along with money, it is the asset that is in shortest supply. Setting out the timetable for your study design includes participant recruitment and sending out reminders, administrative tasks such as reserving rooms and refreshments (in the case of focus groups), setting out a timeframe for transcribing the recordings of the focus groups, preparing handouts and consent forms (see ‘Ethics and Planning the Data Sharing’ section below), and other relevant components of your study that will require organization in advance. Budgeting research costs is often where one's dreams of a Cadillac study design are met with the reality of a Ford Fiesta budget. Social researchers would always like more time and money; however, both are inevitably in limited supply. Good planning is the best way to maximize both. For the QESB, we prioritized the funding of participants' compensation fee (to maximize the number of participants), facilities and refreshments (without which we could not conduct the research), hiring someone to transcribe the 14 focus group transcripts (a process which is very time-consuming) and any study handouts and travel costs (which were necessary and fixed). Instead of hiring someone to conduct the focus groups, we conducted them ourselves. We did not hire a firm to recruit participants, but this required us to develop a low-cost recruitment strategy. To save money on room rentals and refreshments for the participants, we used our university connections to get free space on campuses. It could be argued that off-campus locations would have been better; however, this is an example of the sort of trade-offs researchers must make. When weighing the cost of an off-site conference room against fewer participants, we decided to err on the side of more data rather than nicer, but more expensive, facilities.
If you have access, formal training in constructing and conducting focus groups is very useful. If formal training is not accessible or affordable, it is possible you can find focus groups online, for example, the Peter Hart focus groups that are available at http://CSPAN.com. Using this, you can observe the structure and wording of the questions asked, how the moderator interacts with the participants, and find techniques that might be relevant for your own research. Reading books on focus groups, such as the one by Krueger and Casey, is another way to prepare yourself to moderate a focus group.
Researchers must budget their time during the data collection phase. For reasons of recruitment and obtaining high-quality data, focus group cannot reasonably go longer than 90 min. A duration of 2 h is a lot of time for most people to give up without a large financial incentive; there is also the problem of participant fatigue. The interview schedule – the list of questions to be asked – must be carefully developed. As researchers, we want to get the most information out of our participants as possible; however, trying to rush 8–10 participants through 25 questions is unlikely to produce the thick, rich data that is the hallmark of qualitative research. On the other hand, if there are too few questions and the focus group ends after 60 min, you've lost 30 min of potential data collection due to poor planning. We confronted this problem when we were developing the interview schedule that took place on the nights of the Leaders' Debates. The focus group was planned to be 90-min long and the debates were 90 min. To encourage recruitment, we doubled the normal participation incentive and provided a wide array of food and drink as the groups ran from 7:30 a.m. until 10:30 p.m. in the evening. We also provided a break before and after the televised debates to prevent participant fatigue. With experience, one can develop a sense of how many questions can reasonably be completed by a focus group within a given time; however, if you have never conducted an interview or focus group before, we recommend you ask a group of friends (similar in number to the focus groups you intend to run) to help you by piloting your interview schedule. Find out what works, what doesn't and what can be improved before you start real data collection.
The study's sampling frame – the people who are relevant for your research question – will be determined by your research question. It will also be guided by your method of data collection, especially if your chosen method is ethnographic. As we were conducting focus groups, our study design specified who would be included in our sampling frame, how our participants would be recruited and a strategy to ensure that we had a group that was broadly representative in terms of sex and age. Our sampling frame included people who were eligible to vote in the upcoming British general election. We recruited participants using a three-step process: we used university-wide email lists for the three campuses (at the Universities of Essex, Strathclyde and Aberystwyth) that were hosting our research. Then we used snowballing to increase the diversity of our participants by offering those who submitted an application to participate a financial bonus if they helped us recruit someone who met our sampling requirements and was not associated with the university. Snowball sampling uses a set of initial participants who nominate other participants who are eligible to participate from their social networks, according to David L. Morgan. Finally, we recruited people off the streets in London and Aberystwyth using flyers. To ensure that our groups were broadly representative, we asked people to apply to participate by filling out a form with some basic demographic information and political questions. After reviewing all the applicants, we invited a diverse set of people to participate. In terms of timing, we started recruiting by email 2 weeks in advance of each focus group. If there was a lacklustre response after 3 or 4 days, we scheduled time for street recruitment as soon as possible. To recruit for the post-election focus groups, we informed participants at the end of each focus group that they were invited to participate in the post-election focus groups for the same rate of pay. Our retention rate from pre- to post-election participants was better than 50%. The remainder of our participants were top-ups using off-the-street recruitment.
How will your data be physically collected? Audio or video recordings? Note-taking? The answers to these questions will be guided by the research method. It is possible to record focus groups with audio and video equipment, provided you notify people in advance and get their permission. Video recording can be particularly useful since the researcher can later review the sessions and note any laughing, hand gestures, body language or any silent interactions between participants and include these as part of the data analysis. In the QESB, we used both video and audio recording devices. One of each device was set up at opposite ends of the discussion area to decrease the chances of lost data due to background noise, participants speaking over one another or comments made too softly to be picked up by a distant recording device. Researchers who conduct interviews often use audio recording in addition to note-taking, although informed consent is required and allowing people to speak ‘off the record’ can produce insights that otherwise might not be offered. Ethnographic researchers should carefully consider how to approach the issue of data collection, as Alan Bryman and David Silverman note, there are unique ethical considerations related to the practice of embedding one's self within a setting in order to conduct social research. Another important consideration is the time it will take to transcribe the recordings. A good rule of thumb is that 1 min of audio will take 3–4 min to transcribe depending upon the clarity of the speaker, how many people are speaking at the time and how fast the person is talking. It is always worth asking someone to repeat something they said during an interview or a focus group rather than spend 15 min listening and re-listening to the same piece of sound trying to figure out what they said.
Preparing materials for your study requires time and planning. As part of the recruitment preparation, we designed a brochure about the QESB explaining what it was, what participants could expect, informing them that the sessions would be recorded for research purposes, explaining how their anonymity would be protected, and providing contact information where they could obtain more information. People who were recruited for the study were sent or given a copy of the consent form. At the start of the focus groups, the moderator reviewed each point in the consent form and then asked people to sign and date the forms to indicate their consent. In addition to these materials, the brainstorming exercise for the leaders' evaluation required us to select photos of each of the three party leaders and provide those as a handout along with the instructions for the exercise. Pens were supplied, as were clip boards when the participants were not sat around a table. Finally, each person's financial compensation was put in an envelope, and each participant signed a prepared form that acknowledged receipt of payment for accounting purposes.
The final item in our list for the study design is setting aside time to reflect on how things went. After the first focus group (or interview or day in ethnographic placement), there may be some obvious things that need to be addressed, for instance, perhaps you didn't have enough pens or the battery in the audio recorder gave out halfway through the data collection. You might have found that a topic came up during the session that you want to ask in all subsequent groups or interviews. Where will that question go, and how much time will you devote to it? You might find a question bombed. Was that due to the topic or perhaps the participants misunderstood it? Should you leave the question as it is and try again, should you rework the question and try a new version, or perhaps take it out all together? Did each question take as much time as your piloting indicated? Running a focus group can be tiring, as can spending a day conducting interviews or ethnographic research, but resist the temptation to think you'll remember every observation or mental note you made for another time. Use your audio recorder to make notes or write things down as soon as you can afterwards.
While ethical social research and sharing your data might not seem to be obviously connected, how researchers conform to the ethical requirements of confidentiality and anonymity have profound implications for whether your data can be reused by others. If your research grant requires you to deposit your data with a data archive, such advance planning is particularly important. Qualitative social research has ethical implications due to its reliance upon human participants. Researchers are required to ensure that participants are provided informed consent: a participant's agreement to participate after they have been informed as to the nature of the research, what their participation entails, and that they are told they can refuse to participate or drop out of the project at any time. In addition, the British Sociological Association highlights that informed consent means participants are to be informed that they may request recording devices be turned off in order to speak ‘off the record’, that they are informed as to the procedures that will be used to ensure their anonymity and about the data protection protocols which will prevent the publication or release of identifying or personal information. This might seem like an overwhelming list of things you have to prepare, but it is not as difficult as it seems.
As with everything, ethical compliance begins with good planning. Agreeing to comply with ethical practices is common when applying for research funding grants. A researcher should get a copy of their institution's ethical guidelines and determine whether their research requires approval from an internal board. Students should determine whether their research must get ethical approval before they begin their research as well. Writing your proposal for ethical compliance is a good way to ensure that you have met all the requirements. An example of such a proposal is available from the QESB website. The Economic and Social Research Council (ESRC) Research Ethics Framework requires that research staff and subjects are informed as to the purpose, methods and intended uses of the research. Participants are to be informed what participation in the research entails and what risks, if any, are involved. They also require the confidentiality of any information supplied by research participants and their anonymity be respected. Participation in your study must be voluntary, and harm to research participants must be avoided. Finally, the independence of the research must be clear; any conflicts of interest or partiality must be made explicit.
The QESB met these ethical requirements in three easy steps. The first was an information brochure about the study itself designed in a question-and-answer format. Using a threefold template in Microsoft Word, we mocked up a brochure that provided most of the information required for informed consent in the above list. It explained what the study was, what participation entailed, how confidentiality could work and listed the purpose and funding body behind the research. Not only did this brochure help us fulfil our ethical requirements, it was a very useful tool when recruiting participants. The second step was creating a consent form. The consent form detailed the more technical aspects of informed consent. It is also how the participant officially provided informed consent to participate by signing and dating the consent form. They were also provided with an electronic copy of the consent form to keep for themselves when they were recruited or invited to participate. The final step took place at the start of each focus group. In each of the focus groups, the participants were taken through the consent form and had it explained to them. This was recorded, thereby documenting our compliance with the ethical requirement of obtaining informed consent.
The consent form is a form of protection for study participants, but it can be a very important tool to assist the qualitative social researcher. It is the agreement you make with your participants as to how you can use their contributions in your research. Most people think of participant contributions in terms of confidentiality and anonymity, and those are certainly important. But with some consideration and careful planning, the consent form allows the researcher to obtain copyright to the participants' words. This legal clarification is helpful if you quote participants in your findings and then your research gets published, if you deposit your transcripts in a data archive or publish transcripts or your findings under Creative Commons online. In addition, you can use the consent form to clarify what happens to any contributions a person makes if they leave or withdraw from the study.
Returning to the issue of confidentiality and anonymity, a common misunderstanding is that social researchers are like journalists who must protect the confidentiality of their sources at all costs. As a result, social researchers make the mistake of offering participants total confidentiality and anonymity and promise that no one else will see the transcripts of their interviews or focus groups. This is simply unnecessary. A researcher first needs to consider who is being interviewed. If you are interviewing politicians or people who routinely give interviews to newspapers or magazines, it is quite possible they will give permission to let you use their name, especially if you make it clear they can speak off the record at any time. When recruiting from the general public, offering anonymity is easy and does not present the same problems confidentiality does. Providing anonymity is easily accomplished by changing participants' names in the transcripts and eliminating identifying information. Delete direct reference to places of employment, streets or area of the city in which they live in, names of their friends or family they mention and other identifying information. Individual researchers will have to consider the degree of confidentiality they need to provide to their participants. People recruited to the QESB were fully informed that they would discuss politics; they knew in advance they would be recorded and they understood that the focus group would include several other participants. In the case of the QESB, there was a low confidentiality need since we were not asking people to reveal secret or deeply personal information. Once anonymity was provided, there was no reason to keep the transcripts confidential.
We noted above that ethical requirements and data sharing had areas of commonality. This connection may be clearer after reading through this section. Ethical guidelines protect participants in many ways, but informed consent can be used to further clarify the relationship between researcher and participants. Setting the terms of what happens if a participant drops out and obtaining copyright from participants to use their contributions benefit the researcher. Assuming that there are no confidentiality concerns and if you follow the ethical guidelines, obtain informed consent and copyright and comply with the participant anonymity, then your data can be shared and reused by other researchers, thereby increasing the impact of your work.
The final piece of the research design is your research output. What is your timeline for transcribing your interviews or focus groups? If you are conducting ethnographic work, how will you prepare your notes and other data for analysis? What are you intending to produce: a report, a thesis chapter, an article in an academic journal? How much time will you set aside for data analysis? How much time will be needed to write everything up? The answers will vary depending upon the research, but it is worth trying to estimate how many pages of data you will end up trying to analyse and develop strategies to meet your deadline. Perhaps you don't have to transcribe each interview in its entirety if the research question you are working on only relies on 10 min worth of discussion in each one. If you developed clear and direct questions rather than vague ones that could go anywhere, your transcriptions should be filled with the thick, rich data that will make your analysis both easier to do and more compelling when you present it.
We started this case study report with the observation that good social research design does not have to be complicated, but it must be rigorous. If we consider ‘research design’ as an overview of the individual elements from which social research is built rather than as a linear process of question to data collection to data analysis, then the interactions between the components can more easily and obviously work together. Data analysis informs how we structure our questions and collect our data. Ethical compliance can interact with informed consent to ensure a researcher obtains permission to publish participants' words in articles and on the web, and a well-organized study design maximizes your two most important resources: time and money. A well-prepared research design will increase the chances of getting your research funded and give you confidence that you will conduct very good social research. To learn more about the data that this research design produced and how it was analysed, read the companion piece to this article (http://www.academia.edu/3652553/The_Grounded_Theory_Method_Popular_Perceptions_of_Party_Leaders_during_the_2010_British_General_Election) and our recent publication in The Qualitative Report.
- To what extent can this framework be applied to quantitative research design?
- What would need to be added to the framework if someone wanted to use mixed methods?
- Would you consider making the data you collect freely available online for reuse? Why or why not?
- If a researcher is doing exploratory research with unstructured questions, can she identify in advance which methods of data analysis she will use? What is a potential problem with not identifying the method of data analysis in advance when preparing questions?
- Develop a social science research question suitable for focus group or interviews. Create a rough draft of a research design.