An Evaluation of Prison-Based Dog-Training Programs in Two US States Using a Mixed-Methods Approach

Abstract

This case study describes the process and challenges of conducting a multisite, multi-state prison program evaluation. This research is part of a PhD project that began as a quasi-experimental evaluation of dog-training programs with comparisons between and within individuals before and after the intervention. As problems in gathering substantial quantitative data arose, the researchers conducted qualitative research in the form of semistructured interviews and observations to strengthen the evaluation.

Learning Outcomes

  • To understand the difficulties of conducting prisons research in multiple sites and jurisdictions (e.g. US states)
  • To understand the difficulties of navigating access to prisons in multiple US states
  • To understand the tribulations of conducting research when the researcher is based in another country and the impact this has on data collection

Dog-Training Programs and the Need for Evaluation

A dog-training program is a form of prison intervention program that uses offenders to train dogs either to work as service animals (e.g. dogs for the blind) or to increase their likelihood of being adopted. There have been over 150 prisons, jails, and other correctional institutions in the United States and worldwide (e.g. Austria, Italy, and Australia) that have implemented offender-led dog-training programs.1 Given the widespread use of these programs since their inception in 1981, research to evaluate their positive and potentially negative effects is essential. However, very little research has been conducted. Arguably, one of the important aims of prison intervention program is to reduce reoffending and reconviction. Furthermore, over the last decade, there is a growing trend toward the promotion of evidence-based programming—programming that has been empirically shown to reduce reoffending. Thus far, no large-scale or methodologically sound evaluation of the effects of dog-training programs on long-term recidivism (reoffending and reconviction) has been conducted.

The current literature on dog-training programs, both academic and otherwise, is overwhelmingly reliant on interviews and the perceptions of the people involved with the programs. This qualitative research has been useful for revealing the factors (including risk and protective factors) most affected by the programs; it has shown that dog-training programs positively affect offenders in several ways.2 One such effect is the fostering of social skills and empathy that improves behavior and relationships within the prison setting. Another perceived benefit of such programs is the promotion of an increased sense of self-efficacy (believing in one's abilities) and self-esteem: participants report feeling empowered by their work with the program. Previous studies also report increased levels of self-control. Finally, being in a dog-training program also appears to improve the psychological and emotional health of offenders.3

These benefits are all likely to help offenders stop offending. Nevertheless, as Earl Strimple has lamented, the effect that dog-training programs have on desistance has not been properly studied in methodologically sound research. Consequently, the evaluation outlined in this case study aims to fill this gap in the literature by measuring the effects of dog-training programs on risk and protective factors related to desistance as well as on reconviction rates of the study participants.

Research Practicalities

The bulk of this study took place between August 2011 and February 2013. The research is part of my PhD at the University of Cambridge Institute of Criminology. The project was not funded by any institution or research body, which limited its scope, in terms of fieldwork, quite a bit.

It is also important to mention that the research was rooted in an exploratory study of the effects of dog-training programs on offenders. This exploratory study was conducted for my MPhil and took place between October 2010 and August 2011. For this study, I contacted program coordinators using a combination of random and snowball sampling and subsequently sent a questionnaire that asked them about how they thought the program affected the offender participants. Based on this study and the existing literature, I went into this case study knowing the following:

  • dog-training programs appear to have many positive effects, including improvements in self-control, empathy, self-efficacy, and emotional intelligence.
  • these are risk/protective factors for desistance versus persistence of offending.
  • however, these perceived effects need to be established in a rigorous quantitative evaluation before firm conclusions can be drawn about the beneficial or other effects of dog-training programs.

Research Design

The Initial Plan

As I could not influence participant selection at multiple sites, I knew that conducting a randomized controlled trial would not be feasible. Consequently, under the expert guidance of my supervisor, Professor David P. Farrington, I set out to conduct a quasi-experimental evaluation using comparisons between and within individuals before and after the intervention. The plan was to recruit a minimum of 200 participants (in control and treatment conditions) from a variety of programs in at least three US states and a new program in the United Kingdom.4 This would allow us to detect the likely effect size and explore any differences between programs in their effects. We opted to have either myself or a program coordinator administer pre- and post-participation surveys to participants when they began the program and when they completed it.

Furthermore, we decided that a control group would be established at each prison either by matching on test scores and previous criminal history or by having the program coordinators select a control group based on their equal eligibility for program participation. Although not ideal because this does not systematically eliminate selection effects that result from preexisting differences between the control and treatment participants, this did seem to be the most feasible method of selecting a control group. Using my contacts with several program coordinators from my MPhil research and David's vast experience with criminological research, I set out to conduct a straightforward evaluation to address the following research questions:

  • in what factors do program participants and staff members report changes? Are these changes desirable or undesirable?
  • how are the individual factors affected by the program related to desistance or persistence of offending?
  • are changes in these factors quantifiable as well as observable?
  • what effects does the program have on reoffending?
Developing the Metrics

To measure the main risk and protective factors in the study (self-control, empathy, self-efficacy, and emotional intelligence), I decided to compile a battery of already established psychometric tests rather than develop my own, which would require additional time in order to establish reliability and validity. In selecting the psychometric tests to use in the battery, I considered their contents, the literature on them, and our experience with using them. David's experience with the Basic Empathy Scale and Grasmick's Self-Control Scale led me to select those two surveys. After consulting the literature on measuring self-efficacy and emotional intelligence, I selected the Coping Self-Efficacy Scale and the Emotional Intelligence Scale for their reliability, validity, and appropriateness for this study (in terms of the questions asked).

I also decided to use a measure of criminal thinking styles to see whether the programs had any effect on antisocial thinking. The Psychological Inventory of Criminal Thinking Styles (PICTS) and the Texas Christian University (TCU) Criminal Thinking Scales both seemed useful, so I decided to let the pilot study determine which instrument was used. Using Qualtrics, an online survey program, a pilot study was conducted with students at an American community college and an English university as well as older adults (N = 65).5 This pilot study tested the amount of time it took to complete the survey battery and the clarity of instructions and questions.

The survey using PICTS took participants an average of approximately 20 min to complete, while the survey using the TCU Criminal Thinking Scales took participants an average of approximately 14 min. Given the time saved and the lack of swear words used in the questions, the TCU Criminal Thinking Scales were selected for the final survey battery. There were no other problems with the survey battery raised by the pilot study. When necessary, I sought permission to use the scales.

Sampling

As mentioned, I initially planned to evaluate programs in at least three states (one from the west coast, one from the center, and one from the east coast of the United States) and a program that was just starting in the United Kingdom. I also sought to evaluate a mixture of male, female, juvenile, and adult programs.

The sampling began by requesting further participation from four program coordinators who were particularly helpful and enthusiastic in my MPhil study. These programs were in three states, one from each zone specified above. Using an ever-growing list of dog programs that I have compiled since October 2010 (using Internet searches, the media, and word of mouth), more programs in the identified states were contacted with requests to join the evaluation.

Accessing Prisons: Practicalities and Bureaucracy

It is important to note that during my quests to gain access to programs and prisons—the bane of all criminological researchers—being a US citizen made the process easier. This is particularly true given that I applied for access while in the United Kingdom.

The West Coast State

The program coordinator contacted in this state was enthusiastic about my project but cautioned me that as the state was going through a US Supreme Court mandated culling of the prison population, none of the facilities using dog programs could be sure that they would still be operating in 6 months. Consequently, I decided not to use this state in the evaluation.

The Central State

The first program coordinator I contacted from this state agreed to participate, so I contacted more programs from this state in the manner outlined above, and ultimately, three programs agreed to participate. These programs included one adult male, one adult female, and one juvenile female prison. No state-level access permission was necessary for this state (only permission from each prison warden), which expedited the research process. Subsequently, discussions of survey administration and program visits began; however, problems began to arise in this state as well:

  • one program canceled my visit because of wide-scale reshuffling of prison and program staff. This program was ultimately excluded from the study because a visit was unable to be rescheduled for logistical reasons.
  • when considering the overall sample, the juvenile female program was unusual because it was the only juvenile sample. This, coupled with potential issues that could arise regarding the appropriateness of the survey battery for use with adolescents, led to the postponement of the administration of the survey to this group.
The East Coast State

From this state, two program coordinators were originally contacted for participation. They indicated that permission from the state's department of corrections would be necessary for any visit or survey administration. The relevant people were contacted, and I completed an access application. This initial access application was rejected. However, David and I appealed and addressed the causes for rejection (specifically, how the department would benefit from participating). Ultimately, access was granted, but the authorities requested that I include the state's oldest dog-training program in the study. Fortunately, that program agreed to participate. The programs from this state ended up including one adult female and two adult male prisons. Once these were secured, plans for visits and research were made.

The UK Program

Conducting research in programs thousands of miles away from one's home institution can be quite difficult in terms of logistics and time/spatial constraints. Consequently, David and I were quite pleased that a program was starting in the United Kingdom. The persons involved in the founding of this program participated in my MPhil research and expressed interest in continuing participation. During negotiations and discussions about participating in this evaluation, the program requested a research proposal, and I sent one. Soon after, I was told that the program would no longer be participating as they were conducting their own evaluation. While unfortunate for this study, this incident does serve as a cautionary tale for new researchers: be very careful in sharing research proposals with other researchers.

The Fieldwork Across the Pond

Planning the Trips

Finding fieldwork dates that suit everyone involved takes time. Correspondence also takes time. It is important to remember that you are at the bottom of the proverbial pile and, even when people want to help you, not all calls can be taken and emails answered promptly. Patience and gentle reminders are an art form that must be mastered as a young social science researcher.

After months of planning, two trips were scheduled: one for August 2012 and one for November 2012. However, David and I quickly realized that with six programs which each had few participants and low turnover, sample sizes would be smaller than planned. Therefore, it would be valuable to incorporate interviews and pseudo-ethnographic program observations and make the evaluation mixed-methods rather than just a quantitative evaluation. An interview schedule was then developed based on the types of questions asked in the survey battery. This required some additional access negotiations regarding equipment (e.g. recording devices) and access (e.g. time to speak with participants), both prior to the visits and on-site. In our experience, it is best to be flexible with these negotiations but also to stress the importance of maintaining the integrity of research.

The First Fieldwork Expedition

The first trip taken to complete the fieldwork for this evaluation was to the central state in August 2012.

First Program Visit

The first program visited was a service dog program that used adult male offenders. At the time of my visit, six men were training three puppies who were at the beginning of their extensive training regime. I first visited the service dog center that sponsored the program. During this visit, I interviewed two program coordinators, three members of prison staff, and two participants (who joined in on the interview with prison staff). These two participants also completed the survey. Interviewing staff and offenders together is not ideal as it could bias responses, but that was all that was permitted.

The same afternoon, I visited the prison to see the program in action. During this visit, I was able to speak informally to program participants about their experience with the program. I also took field notes. From there, the program coordinators and I caught up with three other program participants and prison staff at a local shopping mall where they were socializing and training puppies. These participants were informally interviewed and completed the survey. As previously mentioned, in the end, only five participants were available to be interviewed and to take the survey.

Second Program Visit

The second program visited on this trip was a shelter dog program at a juvenile female correctional facility. At the time of my visit, there were eight girls and eight dogs participating in the program, and they were about halfway through their 8-week class.

A recording device was not allowed, so I took extensive field notes. I spoke to staff and participants and also observed how the program functioned. As previously mentioned, this program was not included in the survey portion of the evaluation. One of the unique features of this program is that the girls had to abide by a strict school and programming schedule in addition to training their dogs, so I took particular interest in how this program fitted into the facility and how it functioned as a whole.

Lessons learned from these visits are as follows:

  • it takes time to build up rapport with offenders in prisons, but building up rapport is important when interviewing them. Some offenders need to warm up to you before they are willing to give honest and thorough answers to your questions. The same applies to interviewing staff.
  • the prison operates regardless of your presence and needs, so sometimes participants will be called away or unavailable when you need them.
  • your survey may need adjusting when you begin to use it. I needed to adjust the interview schedule by removing redundant or unfruitful questions and expanding more fruitful questions. I devised a short-form schedule for brief staff interviews and a new schedule for participants.
The Second Fieldwork Expedition

The second fieldwork trip took place in the east coast state in November 2012. Taking into account the lessons learned from the first fieldwork expedition, 2 days were allotted for each program on this expedition. I was also armed with the three different interview schedules and was prepared for anything the prisons threw at me.

Third Program Visit

This visit was to an adult male program that trained shelter dogs. At the time of the visit, this program had 36 offenders and 12 dogs in it and was 4 weeks into its 8-week class. The first day was reserved for building up rapport with the participants and observing how the program functioned. The rapport building was achieved primarily by being present at the training sessions and taking every opportunity available to chat with the offenders and answer any questions they might have about the research. When I felt it was appropriate to do so, I started conducting interviews.

Fortunately, the use of a voice-recording device was permitted during these interviews. Unfortunately, this permission was only granted under the condition that a member of prison staff was present during the interviews. This could have affected the validity and reliability of the interview data as the interviewees may not have felt that they could be completely candid with the interviewer while a member of staff was present. However, as I had spoken with all of the interviewees individually before interviewing them, I was able to corroborate and verify much of what they said in the interview. The data may still be subject to bias, but this is unfortunately a reality of conducting prison research—not every prison will be as permissive with recording equipment as the researcher would ideally like. Extensive notes could have been taken instead of using a recorder, but this method can detract from the interview experience and lead to poorer quality data than verbatim recording.

During the first day, interviews with staff members were conducted. When they could not allow more than a few minutes for the interview, I used the short form. During the second day, interviews with staff members continued, as well as two group interviews with participants, one group interview with nonparticipants (general prison population members), and an individual participant interview. As mentioned above, the program coordinator (who was also a prison staff member) was present for the interviews with offenders.

Fortunately, during this visit, the administration of surveys was not needed. Prior to the visit, the program coordinator had administered the survey to participants (n = 42) and a control group (n = 35) of members of the general prison population who were eligible to participate in the program but were not participating.

Fourth Program Visit

This program had 12 women participants and nine dogs. Five of the dogs were being trained for service purposes and had been there for varying lengths of time. Four of the dogs were shelter dogs and they were also about halfway through their class.

Two days were also spent at this program. The first portion of the first day was reserved for building rapport, but this did not seem as necessary for the women as they appeared to be much more open and engaging than the men and girls who had been interviewed. A group interview with participants and interviews with staff were also conducted. Like the previous program, a voice recorder was allowed here but also under the condition that a member of staff should be present for all recordings. I do not feel that this had a serious negative impact on the interviews, but the data may still unfortunately be subject to bias. Two more group interviews and observations were carried out on the second day of this visit.

Surveys were administered to all 12 participants at the end of the first day, and I collected them the following morning.6 The consent form and instructions were read to them before they were given the survey; they were also asked to complete the forms individually and privately. The participants preferred this ‘overnight’ method to completing it in the presence of the researcher, so I obliged. From this program, only one participant declined to participate in the survey. Unfortunately, no control group could be gathered at this prison because of its strict eligibility guidelines; all eligible offenders are generally placed in the program (often without the consent of the offender).

Fifth Program Visit

This program used adult male offenders to train shelter dogs. At the time of my visit, the program had four dogs and five men. However, this program visit ran into a scheduling problem that could not be remedied—Veteran's Day, a federal holiday, interfered with the planned prison visit. Consequently, I could only visit the program for 1 day, which also happened to be their Graduation Day. Observing the graduation proceedings and informally speaking to the program participants, however, did provide useful insights into the workings and experience of this program. Later in the week, I interviewed the program coordinator and one of the volunteer dog-training instructors.

Lessons learned from these visits are as follows:

  • be flexible, but be prepared. You never know what is going to happen when you show up at the prison gates.
  • prison is exhausting. Be mentally geared up to handle the environment.

Data Collection from Home

I conducted several phone and Skype interviews with program coordinators and prison staff who could not be interviewed during the program visits. These interviews occurred between August 2012 and February 2013 and used the same interview schedule as in the second fieldwork expedition. When conducted from Cambridge, scheduling the interviews required flexibility on my part. It is important that the researcher should be available and prepared to conduct the interview when it best suits the interviewee.

An important part of any research project is constantly asking what would make your research and its findings stronger. In the case of this evaluation and all of the problems that arose, David and I constantly needed to ask this question and attempted to collect more and improved data to strengthen any conclusions we might draw.

Additional data collection included predicted reoffending data from the relevant departments of corrections. Having this data allows me to compare the actual reoffending when offenders are released with their scores on scales that predict their likelihood of reoffending. Many departments of corrections administer these scales to convicted offenders to help classify their security levels and make parole decisions.

Data Processing and Analyses

Qualitative Data

The first step in processing the qualitative data involved typing up field notes and transcribing the interviews (n = 54). This process is extremely tedious and time-consuming. While transcription services can transcribe interviews for you, it is valuable to transcribe them yourself if you can, because you become very familiar with their contents and can begin preliminary analyses of them and looking for trends and themes.

The addition of qualitative data necessitated the need for a theoretical framework in which to analyze the data. I elected to use Derek Layder's Adaptive Theory because my research is both inductive and deductive in nature. The above transcription process works well with Adaptive Theory as the precoding (taking preliminary notice of recurrent themes) that takes place during transcription allowed us to make connections with relevant theories and to start generating a dialogue between data and theory, which is the core feature of Adaptive Theory. From these preliminary codes, concepts related to the social realities of the prison and its inmates were identified and explored further in the data (concept-indicator linkages). As the project centered around the influence of dog-training programs on specific risk and protective factors related to desistance (impulsivity, empathy, self-efficacy, emotional intelligence, and criminal thinking), the conceptual framework was in place from the beginning but continued to grow and expand as data analyses continued. For example, while improvements in all of the above risk and protective factors were noted and expanded on in the interviews, other important themes like trust and the effects of the program on the experience of serving time also emerged. These emerging themes fed back into the concepts and linkages between the data and theory.

Quantitative Data

All of the surveys (n = 98) were completed using paper and pencil. With the intention of saving time in the analysis process, the surveys were initially entered into a survey form developed using Qualtrics. However, a mistake in the development of the online form meant that the survey scores needed to be calculated by hand anyway. Predicted reoffending scores were obtained from the east coast state's department of corrections for 88 of the 98 offenders. Reoffending data are still being collected and will be included in future analyses.

To compare the mean scores of the control and treatment conditions on each psychometric test, I performed t-tests. Because of small numbers, none of these findings were statistically significant. However, as this is an evaluation, I was more concerned with effect sizes; I used both Cohen's d and odds ratio statistics to detect any effect that the DTP had on the factors I measured. I found statistically significant Cohen's d values for two of the five factors and fairly promising odds ratios for all of the factors. I found (nonsignificant) improvements in the expected direction for all of the factors and that being in the program did have a desirable effect (although possibly small) on all of the risk and protective factors I measured.

If my sample size was larger, detecting statistically significant effects would be easier. Furthermore, more accurate evaluations of the effect of the program could be made if the control group had been selected in a randomized controlled trial or using matching rather than through a subjective assessment of appropriateness for the program (which was the method used in this study). I could adjust for selection effects (preexisting differences between DTP and control groups) using predicted reoffending scores and other data.

Making Sense of it All

Merging the qualitative and quantitative analyses was essential for understanding the effects of dog-training programs on their human participants. If only qualitative data had been used to evaluate the programs, it would have been concluded that dog-training programs have a profound desirable effect on all of the factors explicitly measured as well as on several that were not. Alternatively, if only quantitative data had been used for this evaluation, it would have been concluded that the programs had only small desirable effects on the offenders. It is possible that the subjective experience of being in or affiliated with the program influences the perception of its effects by staff and participants and artificially inflates their assessment of its value.

Take-Home Messages

David and I ventured into this case study expecting to conduct a straightforward quantitative evaluation. This was not a naïve expectation—we knew it would not be easy to get access to prisons and a sufficient sample size—but the extent to which problems would arise (and from where they would arise) and the overall impact on data collection and processing were not expected. On the bright side, these problems led to the inclusion of more qualitative data, which added a new and valuable perspective to the evaluation that might have otherwise been missing. This evaluation experience also revealed a variety of other research opportunities and areas related to the dog-training programs that needed to be studied that might otherwise have gone overlooked. Some important lessons for future prison and evaluation researchers were also learned:

  • be realistic. Do not waste time making elaborate research plans unless you are certain that you will have access to the population you are studying.
  • have patience. This type of research is long, tedious, and full of opportunities for disruption.
  • be flexible. When your plans are disrupted, your research needs to be malleable.
  • maintain focus. It can be easy to get off track when you are dealing with multiple types of data or very rich information, but it is important that you maintain focus and aim to answer your research questions before you start down other avenues of analyses.
  • do not be afraid to ask. If you need something, ask for it. The worst that can happen is that your request is denied. Some of my best data came from asking for it even though I expected rejection.
  • make the most of what you have, but still strive to improve it. Determine what the strengths and weaknesses of your research are and determine how best to improve it. Get creative!

Exercises and Questions

  • What is the purpose of a program evaluation? How is it useful to prisons and policy makers?
  • What are the relative advantages and disadvantages of conducting a qualitative evaluation (compared with quantitative or mixed-methods evaluation)?
  • What are some of the ethical problems that researchers may encounter when conducting program evaluations?
  • This case study only discussed a few forms of bias that could negatively affect the evaluation findings. Discuss the other forms of bias that can arise and how you would address them if this were your evaluation.
  • How can you design a study to deal with preexisting differences between treatment and control groups?

Notes

1 These figures are from an ongoing count conducted by the author.

2 Risk and protective factors are factors that impact future offending behavior (e.g. persistence or desistance) in either a negative or positive manner.

3 To read more about these previous studies, see Lai (1998), Strimple (2003), Furst (2006), and Fournier, Geller, and Fortney (2007).

4 To protect the identities of the people involved in the study, the names of the programs and their specific locations are not named.

5 Unfortunately, time and access constraints prevented the use of an inmate population for the pilot study, which would have been an ideal sample because the main research would be conducted on inmate populations.

6 Only 11 participants successfully completed the survey.

Further Reading

Bottoms, A. (2008). The relationship between theory and empirical observations in criminology. In R. D.King & E.Wincup (Eds.), Doing research on crime and justice (
2nd ed.
, pp. 75–116). Oxford, UK: Oxford University Press.
Bowen, E., & Brown, S. (Eds.). (2012). Perspectives on evaluating criminal justice and corrections. Bingley, UK: Emerald Group.
Lösel, F. (2008). Doing evaluation research in criminology. In R. D.King & E.Wincup (Eds.), Doing research on crime and justice (
2nd ed.
, pp. 141–170). Oxford, UK: Oxford University Press.

References

Fournier, A. K., Geller, E. S., & Fortney, E. E. (2007). Human-animal interaction in a prison setting: Impact on criminal behavior, treatment progress, and social skills. Behavior and Social Issues, 16, 89–105. http://dx.doi.org/10.5210/bsi.v16i1.385
Furst, G. (2006). Prison-based animal programs. The Prison Journal, 86, 407–430. doi: http://dx.doi.org/10.1177/0032885506293242
Lai, J. (1998). Literature review: Pet facilitated therapy in correctional institutions. Ottawa: Office of the Deputy Commissioner for Women, Correctional Service of Canada.
Layder, D. (1998). Sociological practice: Linking theory and social research. London, England: SAGE.
Strimple, E. O. (2003). A history of prison inmate-animal interaction programs. American Behavioral Scientist, 47, 70–78. doi: http://dx.doi.org/10.1177/0002764203255212
locked icon

Sign in to access this content

Get a 30 day FREE TRIAL

  • Watch videos from a variety of sources bringing classroom topics to life
  • Read modern, diverse business cases
  • Explore hundreds of books and reference titles