Over the past 2 years, I and some colleagues have been working closely with the Oklahoma City Boathouse Foundation to better evaluate the effectiveness of their programming in sports, exercise, and physical activity. The Oklahoma City Boathouse conducts many, many different activities, but they specialize in rowing and kayaking, and serve as the Olympic training center for canoe/kayak. They recently completed a one-of-a-kind whitewater facility which is astonishing to see firsthand. However, having great facilities does not mean much if they are not being used well and creating more active individuals and better athletes. Therefore, we were recruited to evaluate some of their programming to make sure that what they were doing was actually working. This case is a reflection of one of those research projects, and it highlights the importance of being prepared, anticipating problems, and being flexible enough to overcome unforeseen circumstances.
By the end of this case, students should be able to
- Have a better understanding of why proper planning is necessary in qualitative research
- Understand the benefits and challenges of working with community members
- Be able to better prepare for unexpected situations that may occur during data collection
- Be able to develop a qualitative design that reflects on the possible pitfalls that might occur
Perceptions of Participation in an Afterschool Indoor Rowing Program for Youth
To provide context to this focus group study, it is important to understand why it was necessary. When thinking about education, sports, or really any other field, people make decisions and spend an incredible amount of time and sums of money on ideas and suggestions that sometimes lack very little logic, planning, or research. What this usually means is that money is wasted and the project is not as successful as it could be. For example, in my local community, I might decide to start a sports league for cricket, and I invest money in advertising and promoting this league. The problem is that few people in my city play cricket, and as a result, the project falls flat and I end up poorer. If I had invested a little time, effort, and maybe a little money to determine whether the population of my city would support the project, I might have saved myself a lot! Maybe through this research, I would have found a sport that might have been well-received!
The Oklahoma City Boathouse Foundation, recognizing the benefits of doing some evaluation of their projects, asked me and some colleagues to help them evaluate their current programming and some programming that they wanted to introduce. One of their main goals was to introduce an afterschool rowing program for children in the local community before eventually expanding it to other areas of the state.
Afterschool programs are popular (Afterschool Alliance, 2014), and they have many psychological and physical benefits (e.g., Schuna, Lauersdorf, Behrens, Liguori, & Liebert, 2013). However, oftentimes the physical education teacher is expected to run such a program, placing additional burdens on them, and for an afterschool program to run effectively, it needs commitment from the entire community (Baghurst, Fink, & Adib, 2016). A community partner, therefore, was seen as an ideal entity to help run a program using their expertise and simultaneously take much of the burden away from the physical education teacher.
About the Program
We helped the Boathouse develop a 6-week, twice-weekly afterschool rowing program for two schools in Oklahoma as part of a pilot program that was to be expanded over time. The program was for anyone in second through fifth grade, and it focused primarily on using rowing machines (ergometers), but games and other physical activity stations were also developed into the program. To make sure that the participants were gaining something from the project, we required them to do a pre- and post-rowing test to 500 m and also complete a self-concept survey. This was to see whether participation in the program positively changed their physical performance as well as to determine how they felt about themselves during the program and if any improvement had occurred. Then, at the end of the program, the two school programs met up at the Boathouse and had a friendly competition to see which school and participants were fastest.
How We Collected Data
Although this program has continued expanding to other schools and communities during the past 18 months, and we have tweaked it to improve the outcomes, how do we know that the program has a lasting effect on participants? That is the problem with a lot of programs; they are good for the short term, but they do not really make too much difference long term. Or, they do have long-term benefit, but no one knows about it, and the program is not supported as much as it should be. We wanted to see whether this program changed how the participants felt about themselves, sports, and rowing, but to do that required more than just a pencil and paper test. We had to ask them!
A year after the first two groups had completed the first rowing program, we asked the Boathouse to contact the parents of all of the participants involved and ask them whether they would be willing to allow their child(ren) to participate in a focus group. Although an interview can generate richer, deeper information, we opted to collect data via a focus group. This is a difficult decision but we had to consider two main concerns.
The first concern was that interviews of up to 12 participants were going to be very lengthy, time-consuming for us and the participants. It would also require an incredible amount of interview transcription, which is also time-consuming and potentially expensive.
The second reason was the participants themselves: unless they know you well, second through fifth graders have a tendency to be quite reserved, especially in what they might consider to be an awkward and “clinical” situation. The belief was that a small group of participants who knew each other might be more willing to open up than in a one-on-one situation with a stranger.
What Happened on the Day
It was a beautiful Saturday morning as my two colleagues and I drove the 90 min to the boathouse/restaurant that the Foundation had kindly made available for data collection. We arrived in plenty of time only to find the boathouse locked! No one was answering their phone and some parents and participants had begun to arrive. However, our Boathouse contact did arrive just on time and we hurriedly tried to set up our tables, computers, mics, and video camera. We had one video camera, but we also audio recorded the focus groups on a backup computer just in case of malfunction. During this process, a colleague asked participants and parents to complete the various ethical forms necessary to take part in the study.
In the first focus group, things began well. Our interviewer quickly established a rapport with the participants and we were getting good responses from almost everyone. We had pre-prepared questions designed to take participants through their initial decision to participate in the rowing program to their experiences following its completion. This was based on the Transtheoretical Model (Prochaska & DiClemente, 2005) where individuals go through a series of stages when thinking about, beginning, and maintaining an exercise program. Examples included “Is there anything that happened that sparked your interest in rowing?” “What did a good day of rowing look like?” “Do you feel like rowing helped you get better in other physical activities?” “Why don’t you continue to row?”
Unfortunately, only a few minutes into the focus group, the massive refrigerator in the kitchen area kicked in! Everyone had to talk louder and we were very concerned that we would not be able to hear the video output. Therefore, while the focus group continued, one of my colleagues dug around underneath the kitchen counter until he was able to pull the plug to the fridge. It did not stop the noise of the soda dispenser, but it was as good as it was going to get.
The first focus group went well. Beyond the noise, there were no real issues. Participants were quite engaged and we got good data from the process. As the first group was leaving, the second group started to arrive. Again, we went through the paperwork process and got settled. Our interviewer began establishing rapport and going through her list of questions. But there was a problem! After some questions had not been answered clearly, we probed for more information and discovered that this group had just started one of the afterschool rowing programs. Not only was this group unable to provide any information about how the program had affected them over the past year, they had barely begun the program and knew little about it! The Boathouse personnel had recruited the wrong group and we had no idea!
Such situations require quick thinking, and we immediately changed our questions to ask why the participants had enrolled, what their initial first impressions were, and why they might continue in the sport. For example, we could not ask “Why don’t you continue to row” and instead had to focus on the early part of the Transtheoretical Model and ask more questions such as “What were your first impressions of the program?” “What do you feel like you gain from participation?” “What obstacles might stop you from continuing to row once you complete the program?” It was completely off-script, but it was that or cancellation of the focus group. How can you cancel the focus group when parents and participants have taken a Saturday off to be there? It was certainly a learning experience.
Data Collection Takeaways
Our experiences provide opportunities to learn what we did and did not do well, which is the primary purpose of this discourse. Here are a few things to consider when preparing a qualitative study similar to our own:
- Determine which qualitative measure will not only best answer your questions but is most practical and feasible. Consider the timeframe you have, the financial cost, and how it might affect whether your participants not only choose to participate but how much information you will get.
- Work with partners, but double check everything. In our situation, we were able to make the situation work, but we were not in control of participant recruitment (or of opening the facility). Although the Boathouse personnel have been amazing during our many projects, they made a mistake and we were unaware until data collection had begun.
- Choose your location for data collection carefully. The boathouse was a good facility; it was bright and clean and near the schools where the rowing program had originally taken place. However, our mistake was not to evaluate its effectiveness before the day of data collection. Something could have been done in advance about the noisy appliances prior to data collection and not during.
- Expect non-participation. We scheduled six participants for each group, primarily because we did not think that everyone would show up. They did not, which meant that we still had a good group of participants. Similar to airlines that overbook, do the same when collecting data from a group.
Exercises and Discussion Questions
- Consider different kinds of data collection methods such as interviews, focus groups, or surveys. What are the pros and cons of these and what kind of “hiccups” might occur from using them?
- Given the limited understanding of the study, which was very briefly described, what might you have done differently to improve the process or outcome of data collection?
- We collected data using focus groups and explained why. What factors might have affected how participants responded to questions?
Conducting Focus Groups: https://www.google.com/#q=conducting+focus+groups
Focus Groups Versus Interviews: https://www.google.com/#q=focus+groups+vs+interviews