Case
Abstract
In 2012, I project managed a team of researchers who were commissioned to undertake an evaluation of the Sunderland Health Champions Programme. Evaluation is an activity that remains central to health promotion practice because it is concerned with assessing whether interventions are effective. Health Champions are a growing component within the British public health workforce, and their roles are emphasised within the coalition's government's public health strategy. However, there is the need for further exploration of the way in which Health Champions work and the effectiveness of programmes that use Health Champions as a mechanism to try to achieve positive health changes. Therefore, Sunderland Teaching Primary Care Trust commissioned independent researchers, staff from the Centre for Health Promotion Research at Leeds Metropolitan University, to evaluate their Health Champions Programme. This case study provides an account of the evaluation, taking the reader through the methods that were used and in particular focusing upon the use of a theory of change approach that is associated with realistic evaluation as a specific approach. The case sheds light upon the challenges of evaluating practice initiatives as well as highlighting the usefulness of theory of change as an evaluation tool.
Learning Outcomes
By the end of the case study, students should
- Have a better understanding of the methodological challenges involved in conducting evaluation research within health promotion settings
- Understand the methodology of using theory of change to assess the success of health promotion interventions
- Be able to examine the advantages of using mixed-methods approaches when conducting evaluation research
- Be able to assess the pros and cons of time-limited evaluation approaches within health promotion settings
Project Overview
The Sunderland Health Champions Programme is an initiative which aims to improve health and address inequalities in the Sunderland area via the creation of a workforce development scheme. The programme seeks to identify staff groups and volunteers who through personal face-to-face contact with clients present novel opportunities to reach individuals and communities experiencing poor health and/or health inequalities. The programme set out to deliver training to these staff groups in order to train staff and raise people's awareness so that services may then demonstrate the practical implementation of the idea that every contact is a health improvement contact. The pilot project, which started in November 2010, was overseen by Sunderland Teaching Primary Care Trust (tPCT) and delivered in partnership with a range of training providers and local employers. The Sunderland Health Champions Programme is based on a social movement approach to achieve an ‘industrial scale’ approach to addressing health inequalities and, ultimately, shifting culture. It aims to do this by utilising and expanding Health Champions' circles of influence (self, family and friends, clients, wider community) in relation to health improvement.
Existing evidence about Health Champions shows that they have become an important element of the lay public health workforce (White, South, Woodall, & Kinsella, 2010). More evidence is needed to understand the mechanisms of change that lead to improvements in health, particularly as lay health workers often focus upon working with underserved communities in relation to health inequalities (South, Branney, & Kinsella, 2012). This evaluation aimed to contribute to the evidence base by exploring the specific approach used in Sunderland.
The evaluation team examining the Sunderland Health Champions Programme wished to establish how well it was meeting its existing objectives and to quantify its impact upon teams, services, organisations and communities within the area in which the programme was operating. The evaluation team adopted a theory of change (TOC) approach, a methodology which maps the assumptions informing planned interventions within all stages of a community initiative. The TOC approach is important in evaluation research that is examining a complex network of factors which influence project outcomes.
In order to conduct a realistic evaluation, the evaluation team had the specific task of assessing the two core objectives of the programme:
- The evaluation examined whether raising health awareness and promoting lifestyle change among training participants was taking place.
- The evaluation examined how staff, volunteers and community organisations were identifying and acting upon opportunities to promote health with the service users with whom they had routine contact.
The evaluation focused especially on capturing the views and experiences of course participants and service managers in order to assess the programmes performance against its core objectives and to ultimately measure the impact that the programme has had upon participants, organisations and communities within Sunderland.
Project Context
The Health Champions Programme that is the subject of this case study emerged in the context of significant health inequalities within the Sunderland area (Sunderland Data Annex–National Health Service (NHS) South of Tyne and Wear and Sunderland City Council, 2011), and a commitment and vision articulated in the NHS Sunderland tPCT Integrated Strategic Operational Plan 2011–2015 to reduce these. Part of that vision is to shift the balance from treating illness to helping and supporting individuals to live longer and healthier. One strategy that is attempting to achieve this is the Health Champions Programme.
The strategic aim of the Sunderland Health Champions Programme is to improve the health of all disadvantaged communities in Sunderland by developing the Health Champions' role as a mechanism to support local people in positively addressing both health determinants and accessing appropriate services. The Sunderland Health Champions Programme aims to address health inequalities and ultimately shift culture in relation to health by utilising and expanding Health Champions' circles of influence (self, family and friends, clients and the wider community) as a strategy to improve health.
This Health Champions Programme takes a unique approach to developing capacity for delivery of health promotion in that the training provided is not exclusive to volunteers but is also available to front-line employees working for the local authority and within other workplaces. Health Champions undertakes five training modules, offered by different training providers, including the following:
- Understanding health improvement. This is a Royal Society for Public Health (RSPH) approved course providing individuals with knowledge and understanding of the benefits of good health and well-being. It also aims to equip people with the knowledge and understanding of the principles of promoting health and well-being and to develop the public health skills to support lifestyle changes.
- Emotional health and resilience. A course that aims to support staff or volunteers in terms of training them in how to promote emotional resilience in others.
- Financial capability. A course for staff and volunteers to enable them to support and signpost people experiencing financial difficulties.
- Smoking brief intervention. A course to train people to conduct brief interventions and to provide very brief advice in relation to smoking cessation.
- Alcohol brief intervention. A course training people to conduct brief interventions and to provide advice in relation to alcohol consumption, as well as training in relation to appropriate referral where necessary.
Once completed, these training modules lead to an individual gaining the label of a Health Champion, with the expectation that once trained the Health Champions will communicate via their existing ‘circles of influence’ as part of a team of people who educate and advise others in relation to health. For example, Health Champions should educate friends, family, clients and neighbours as the starting point for health education and improvement upon completion of their training. Therefore, the programme while implemented in workplaces has an orientation to the community settings where many staff who had undertaken the training both live and work.
Research Practicalities
The evaluation was carried out between November 2011 and April 2012. The evaluation used a mixed-methods design with quantitative and qualitative components.
- Interviews were conducted with key stakeholders from different key partner organisations, both the statutory and voluntary community sectors.
- Focus groups were conducted with Health Champions from both statutory and voluntary community sectors.
- The quantitative component comprised analysis of programme monitoring data and a questionnaire-based survey to gauge impact at an individual level.
Ethics
Ethical approval to conduct this study was obtained through the university research ethics process. Informed consent was obtained from all participants prior to digitally recording all interviews and focus groups. Confidentiality and anonymity was assured across all methods used and the participant's right to withdraw without prejudice was clearly expressed to each. All quotations used in reports and associated publications paper are anonymised, simply differentiating participants only as either Health Champions or stakeholders. Due to the nature of the project, it was not foreseen that there were any physical risks to the participants in the data collection undertaken especially as it was completed by an experienced research assistant, who has worked with a variety of populations including professional stakeholders. The data collection conformed to the Social Research Association's ethical guidelines, with participation being voluntary.
Research Design: Theory of Change
The evaluation used a mixed-methods design, combining qualitative and quantitative data collection and analysis, in an approach that has become increasingly accepted in health promotion research (Green & South, 2006). The evaluation used both qualitative and quantitative approaches in order to strengthen the findings by allowing some triangulation from different data sources. In addition to combining both quantitative and qualitative data collection, this evaluation also used a realistic evaluation approach, specifically applying the TOC (Judge & Bauld, 2001).
Theory of Change in Evaluation Research: Realistic Evaluation
The TOC approach is a way of modelling how change will happen in a programme or intervention. It arose in the work of Chen, Rossi and Weiss who focused upon complex community initiatives and was further developed by the Aspen Institute (Fulbright-Anderson, Kubisch, & Connell, 1998). Health promotion programmes and interventions are often implemented with only implicit assumptions about how they work. TOC approaches are used in evaluations of complex community interventions, such as the Sunderland Health Champions Programme, as they allow for the exploration of why and how interventions work (Weiss, 1995). Thus, the TOC approach is an attempt to bring any latent assumptions to the surface (Green & South, 2006). A TOC is used to document and describe progress made towards outcomes within any given intervention. Therefore, the TOC is an approach rather than a methodology in itself. Within this approach, the principal investigator (J.S.) produced a written explanation of how the programme was moving from its activities to its objectives, and data collection was then designed to capture evidence against the TOC (see Table 3) following discussions with key partners and stakeholders. A TOC approach can be used before, during or after a programme has been implemented. In the case of this evaluation, it was used after the programme had been implemented.
The evaluation under discussion in this case study used a TOC framework (Connell & Kubisch, 1988) to explore how health awareness was raised and how lifestyle change was then promoted among trainers, volunteers and community organisations, after individuals had completed the required training. The advantage of using a TOC approach is that it helps partners and stakeholders make explicit the links between activities delivered and programme goals (i.e. behaviour change). At the outset of the evaluation, partners and stakeholders were brought in to develop and agree on their ‘theories of change’. Facilitated by the evaluation team, a map of the preconditions required was designed to bring about the long-term goal of the Sunderland Programme. This TOC was then tested by the evaluation team.
Using this approach improves the rigour of any evaluation, especially as there are often methodological difficulties in using other localities as control sites, for example, in this instance, there are no similar programmes with which comparisons can be drawn. In this case, the evaluation team have extensive experience of applying realistic evaluation and TOC approaches for complex interventions.
When applying a TOC in any evaluation, the approach must take into account the context and needs of the local area, the mix of programme activities and the range of changes (outcomes) that may result at individual, community and population level. The evaluation also needs to produce robust evidence to inform decision-making and service improvements. The evaluation in this case study used this approach by linking the context and the ways change occurred to the resulting outcomes (CONTEXT + MECHANISM + OUTCOME) (Judge & Bauld, 2001; Pawson & Tilley, 1997).
Sampling
Purposive sampling was used throughout the data collection. This sampling approach is useful when there are a limited number of people who are appropriate for addressing the aims of the study. Thus, the research team working with Sunderland tPCT chose the sample based on who they thought was appropriate for the study. For the qualitative interviews, a list of 38 key stakeholders involved in the programme was devised by Sunderland tPCT as the sampling frame from which to select interviewees. Key stakeholders were interviewed from Sunderland tPCT, the Area Committees, the Task and Finish Group, managers of Health Champions within various statutory and voluntary community organisations, wider stakeholders as well as training providers for the programme. Thus, the criteria for identification of stakeholders were involvement in the development, design, implementation and delivery of the Health Champions Programme. Wider stakeholders were also included in order to provide broader views on the strategy, for example, local councillors who had strategically supported the development of the initiative. This initial list of key stakeholders for the programme was administered to the evaluation team in December 2011 by the tPCT, and each stakeholder was contacted and invited to take part in the evaluation. In all, 22 consented to participate in a sample, and an overview is provided in Table 1.
Table 1. Summary of the interviews from key stakeholder groups.
A database containing all of the Champions who had successfully completed their training and consented to the tPCT to take part in the evaluation (144 Health Champions) was also provided to the evaluation team by Sunderland tPCT. Consequently, all Health Champions were invited to participate in the focus groups. Of 155 Health Champions, 144 consented to the tPCT to take part in the evaluation after completing all of their training. These Champions were then contacted by the evaluation team to participate in both the focus groups and in completing the questionnaire. In all, 33 Champions participated across four focus groups, and 16 of these Champions were from the statutory sector and 17 Champions from the community and voluntary sector.
The same Health Champions were also included in the sample for the questionnaire; thus, the questionnaire sent to all 144 Health Champions. A total of 58 surveys were returned: 52 online and 6 paper submissions (40% response rate). Table 2 provides an overview of the data collection conducted within this evaluation.
Table 2. Overview of the data collection for the Sunderland Health Champion Evaluation.
Methods and Data Collection
Interviews
The qualitative component of the research began with individual semi-structured interviews being conducted with key stakeholders who had developed and delivered the programme. A total of 22 key stakeholders were involved in interviews conducted by the evaluation team in January and February 2012. Semi-structured interviews were carried out by the research team to direct discussion around a number of key themes comprising involvement in the programme, perceptions of the role, motivations for doing the training, recruitment processes, support and impacts on individual, community and public health. Interviews were usually carried out face-to-face throughout January and February 2012. However, three telephone interviews were carried out with stakeholders who were not able to meet in person due to time constraints. The interviews were run using a standardised, open-ended schedule in which the same questions were asked to all of the stakeholders. This approach ensured comparability of answers within the analysis process, while still facilitating conversation and allowing interviewees to express their perceptions and experiences. These key stakeholders were able to inform the TOC by providing evidence that value had been added to public health based upon a more informal delivery than professional services, and that a ‘critical mass’ of people had been trained (see Table 3).
Table 3. Theory of change mapped against the evidence collected within the evaluation
Focus Groups
Four focus groups were carried out during January 2012 to capture the views of the Health Champions themselves. Given the variety of contexts in which the Sunderland Health Champions were working, it was necessary to differentiate and compare views of champions within statutory and third-sector organisations and those volunteering. Thus, participants from each of these sectors were invited to separate focus groups. During the focus groups, the research team facilitated discussion around key themes including how training was used, motivations for doing the training, support received and impact of the training on themselves and others. The focus groups were designed to be interactive and engaging as well as offering a chance for Health Champions to network with each other. They allowed opportunities for group discussions and chances for people to share experiences. The focus group methodology was a dual moderator approach in which two members of the research team worked together: one ensured the session progressed smoothly, while the other monitored that all topics were covered. Again the focus groups used a standardised, open-ended schedule in which the same questions were asked to all of the Health Champions. The data from the focus groups provided rich qualitative evidence linked to the TOC, for example, Health Champions discussed their experiences of training, and the application of their learning in offering health advice and support across a range of contexts, and circles of influence (see Table 3).
Questionnaires
The main quantitative component of the evaluation was a questionnaire, administered online and sent to all Health Champions. The survey was designed to complement the monitoring data already gathered through the tPCT database and covered key variables on Health Champions, the contexts they were working in and the reported impact on training. The questionnaire was administered online using SNAP (similar to Survey Monkey but with more flexibility and scope) and was complemented by administering questionnaires to Champions within the focus groups, and paper-based questionnaires were made available upon request. The questionnaire data linked into the TOC by quantifying the ways in which the Health Champions were using their training within a specified 2-week time period, for example, documenting the numbers of brief interventions that had been delivered as a form of health promotion. Finally, Sunderland tPCT held a small amount of monitoring data, which the evaluation team were able to access and analyse to produce some descriptive statistics as part of the quantitative data set. This again fed into the TOC to evidence numbers trained, which related to a key programme objective of shifting culture via large-scale training.
Theory of Change in Action: Testing the Theory of Change
The evaluation in this case study tested a specifically developed TOC (Judge & Bauld, 2001). This TOC makes explicit the links between programme goals and the different contexts and ways in which Health Champions work. It provides a framework for mapping subsequent outcomes at individual, organisational and community level which fitted with the Sunderland approach based on ‘circles of influence’. Table 3 illustrates the TOC used within the Sunderland Health Champions evaluation, as well as the evidence collected that was used to test it.
Analysis
All qualitative data were transcribed verbatim and then initially read and reread by the research team to ensure familiarity with the content of the transcripts. Initial coding was undertaken in order to develop a coding framework using an inductive approach to identify the full range of emerging themes from the data. The coding framework was then applied to each transcript, with data subsequently organised into major thematic categories and sub-categories. Themes were discussed and agreed within the research team.
The quantitative data from the questionnaire were exported from SNAP 10 to Excel and SPSS 19. Multiple choice variables were recoded from binary codes and frequency counts were generated with the production of frequency graphs and tables; these were used to display the data.
Sunderland Evaluation: Practical Lessons Learned
Evaluation is a methodological area that has many similarities to traditional social research. As you will have learned from this case study, evaluation research uses many of the same methodologies as traditional social research. In this instance, interviews, focus groups and a questionnaire were used. However, evaluation usually takes place within a political and organisational context; therefore, it requires slightly different skills than traditional social research. For example, when conducting evaluation researchers often use management skills, group skills and need to be capable of sensitively working with a range of stakeholders.
Bearing this in mind, it is useful to consider the following tips when doing evaluation research:
- Evaluation is complex. You will need to carefully plan your evaluation, and it is often useful to work in a stepped process starting with the clarification of aims and objectives, choosing indicators, linking outcomes to methods, understanding the context, collecting data and, of course, bringing all of this together (see Green & South, 2006, chapter 4). The Sunderland Health Champions Evaluation followed this stepped process.
- Each evaluation will be different. You need to be aware of the context in which each intervention is taking place in order to understand the programme and pay attention to the need to use different research methods and approaches according to their suitability. In this case study, a specific TOC was developed based upon the way in which the Health Champions Programme was operating. Every evaluation of an intervention requires its own unique TOC.
- Evaluation can be fuzzy. You may need to measure difficult concepts or have to think carefully about what it is realistically possible to measure within an often very limited time frame. For example, in the evaluation of the Sunderland Health Champions, it was not possible to measure any individual outcomes in terms of health improvements for the individuals who received advice and brief interventions as these contacts were not recorded, tracked or indeed monitored. Therefore, data collection had to focus upon the Health Champions themselves as well as stakeholders.
- Evaluation can be political. Often when working within organisations, political agendas and complex associated issues can arise. Evaluators tend to be able to improve their data collection and access to participants if they can establish a good working relationship with the organisation responsible for the programme under scrutiny. The Sunderland Health Champions Evaluation progressed well and resulted in learning because the tPCT staff and the evaluation team established a good partnership from the outset of the process.
Conclusion
Evaluation is central to health promotion practice and is an essential activity that can address questions about what works and why it does. Despite the complexities associated with doing evaluation research, there are many guidelines available to help those who engage in this activity. This case study is just one example of a realistic evaluation approach in which a TOC was developed and then tested in order to assess the Sunderland Health Champions Programme. Given the complexity of this type of research, evaluators need to be transparent about how they collect data and make decisions within their data collection. The case study written about here is a transparent description of one realistic evaluation that worked very effectively in practice. Of course, no single piece of evaluation is perfect; ideally, in this case, it would have been more rigorous to sample participants who had received health advice yet this was not possible. That said, using realistic evaluation methodology was a positive experience in that it facilitated the evaluation teams understanding of the context, mechanisms and outcomes associated with the Sunderland Health Champions Programme.
Exercises and Discussion Questions
- What do you see as the major strengths of using a realistic evaluation approach? Can you also identify any potential weaknesses?
- In conducting evaluation research, how might the goals of the commissioners contrast with those of the researchers? What might this mean for the evaluation process?
- If you were conducting evaluation research, what factors do you think would influence your choice of methods?
- There are often power imbalances within research. Reflect upon how some of these imbalances might be addressed in evaluation research contexts.
- Lay people and those who are in receipt of the intervention may not be particularly interested in participating in evaluation research. Consider how you might motivate them to participate in your research.
- Consider some of the ways in which you might disseminate evaluation findings, given that such research has the potential to influence both policy and practice.
