Skip to main content
SAGE
Search form
  • 00:11

    GRAHAM MOORE: My name's Graham Moore.I'm a researcher at the DECIPHer Center, whichis a research center which develops and evaluatescomplex interventions for health Improvement,mostly looking at children and young people.I'm going to be talking about evaluatingcomplex interventions using process evaluationmethods within randomized controlled trials.

  • 00:32

    GRAHAM MOORE [continued]: And I'll use the National Exercise Referral Schemein Wales as a case study.Complex interventions are often defined as interventionswith multiple components, which interact to produce change.Complex interventions are also complex in terms of how theyinteract with their contexts.So the same intervention might have different effects

  • 00:53

    GRAHAM MOORE [continued]: in different contexts.Randomized controlled trials are oftenconsidered to be the best way of establishing the effectivenessof complex interventions.While they are useful in showing overallwhether an intervention does more good than harmin a specific time and place, they don't reallytell us a huge amount about why interventions work, for whom,and under what circumstances.

  • 01:14

    GRAHAM MOORE [continued]: So to get a deeper understanding of those issues,we need to embed process valuationswithin complex interventions-- within randomized trials.The Medical Research Council Frameworkfor evaluating complex interventionswas published in 2008.And this recognized for the first timethat process evaluation was vital alongside trials

  • 01:36

    GRAHAM MOORE [continued]: of complex interventions.It set out some broad aims for process evaluationto achieve, stating that process valuation nested within trialscan be used to assess fidelity and quality of implementation,clarify causal mechanisms, and identifycontextual factors associated with variation in outcomes.Some years after the development,after the original publications in 2008 guidance

  • 01:58

    GRAHAM MOORE [continued]: for evaluating complex interventions,Medical Research Council funded some researchto develop guidance for conducting process evaluationof complex interventions.We developed a new framework, whichbuilt on those core aims specifiedwithin the original guidance, arguing that, alongside trials,it's necessary to understand how complex interventions are

  • 02:19

    GRAHAM MOORE [continued]: implemented.So the process is put in place to achieve implementation,to capture what's delivered in order to understand whatit was that produced change.To capture mechanisms through which the intervention works,and also explore those mechanisms qualitativelyto build up a better picture of how the intervention works.

  • 02:42

    GRAHAM MOORE [continued]: And also to understand the role of contextual factorsin shaping whether or not an intervention is implementedfully, and the mechanisms through which the deliveredinterventions actually works.The case study I'm going to be talking aboutis a trial that we conducted a few years ago,which had an embedded process evaluation within it.And that's the National Exercise Referral Scheme.

  • 03:04

    GRAHAM MOORE [continued]: And the National Exercise Referral Scheme in Walescomprised health professional referral to a leisure facilitywhere patients attended an appointmentwhere, during a consultation, they would go through a healthcheck with a qualified exercise professional whowould go through a consultation basedon motivational interviewing, and including goal setting

  • 03:25

    GRAHAM MOORE [continued]: components.Patients would then be offered a 16-week programof physical activity, before being signposted to community based exercise opportunities to tryand maintain their activity levels beyond the program.As a randomized control trial, this interventionshowed that, for patients who were referred for low levelcoronary heart disease risk factors,the intervention was very effective in promoting change

  • 03:46

    GRAHAM MOORE [continued]: in physical activity.For patients who referred for mental health reasons, somild-to-moderate depression or anxiety,the intention was effective in improving their mental health,even though it had no effect on their physical activity levels.On the basis of the trial, the interventionwas viewed by policymakers as a success.And the trial was seen as justifying ongoing funding

  • 04:08

    GRAHAM MOORE [continued]: for delivery of the intervention throughout Wales.If we relied purely on the randomized controltrial findings, we would concludethat an intervention based on health professional referralto a leisure facility where patient'sreceive motivational interviewing, goalsetting, and a 16-week program of groupbased exercise led to improvements

  • 04:29

    GRAHAM MOORE [continued]: in physical activity levels for patientsreferred for coronary heart disease risk factors.But is that actually a valid conclusion?And is it sufficient in order to allowus to kind of make sense of how that trial works,and apply the findings from it elsewhere?We embedded a detailed mixed methods process evaluation

  • 04:52

    GRAHAM MOORE [continued]: within the trial at the National Exercise Referral Scheme.And this addressed a number of key questions.First, we explored how the new natural protocols for exercisereferrals took shape across the different local settingsin which it was implemented.It was implemented in 12 local health board areas in Wales,some of which had previous schemes, others which didn't.Some of those previous schemes looked very much

  • 05:13

    GRAHAM MOORE [continued]: like the new scheme.Others were quite different from them.But we also examined how consistent implementationof the program was with the underlying program theory.So whether the core components of the interventionwere delivered in a manner that wasconsistent with the intended theory of the program.And we also conducted a range of qualitative interviewswith patients, and with excellent professionals,

  • 05:33

    GRAHAM MOORE [continued]: and with local and national coordinatorsin order to try and understand the mechanisms through whichthe intervention worked, and someof the patterning in responses to the intervention.Our interviews with policy stakeholdersat the local and national levels indicated--gave us a lot of insights into why the scheme was implemented

  • 05:55

    GRAHAM MOORE [continued]: somewhat differently in different local areas.To some extent, the acceptability of the programand how well it was implemented seemedto vary according to what had been in place beforehand.So the scheme was accepted differentlyaccording to how invested practitionerswere in the schemes that had been in place beforehand,and the extent to which they viewed

  • 06:16

    GRAHAM MOORE [continued]: the move towards national standardizationas necessary, or a good move.There are also some quite substantial challengesin particular areas where they lost members of staffduring key points during that transitional time.Initially, there were some issueswith the structures in place to ensurecoordinated national standardization.

  • 06:36

    GRAHAM MOORE [continued]: There was no national coordinatorin place at the time the intervention went live.And that led to some quite different understandingsin what the intervention was meantto be in the different areas where it was delivered.That qualitative data gave us a lotof insights into why the implementationof the intervention varied from one site to another.We actually conducted quite a few forms of quantitative data

  • 06:58

    GRAHAM MOORE [continued]: collection in order to capture the fidelityof the intervention in terms of what was implemented.And we showed that some of the key componentsof the intervention, which were intendedto enhance patients' motivation for changing physical activity,including motivational interviewing and goal setting,were actually very poorly delivered.But our qualitative data-- we conductedqualitative interviews with patients and with excise

  • 07:21

    GRAHAM MOORE [continued]: professionals-- those seemed to pointto a number of mechanisms, which we hadn't really anticipated,through which the intervention stillseemed to be supporting patients'motivation for change.These drew our attention to a lotthe emerging social networks, a lotof the emerging kind of social supportaspects of the intervention, which we perhapsundervalued in our initial theorizing of the intervention.

  • 07:43

    GRAHAM MOORE [continued]: Our quantitative analysis showed us that actually,even though a lot of the components which were intendedto be delivered in order to enhance motivationwere delivered, the intervention stillworked via improving patients' motivationfor physical activity.So if we went back to our original conclusion,

  • 08:03

    GRAHAM MOORE [continued]: if we were to base our conclusions on this schemepurely on the randomized control trial findings-- we onlyhad outcomes data-- we would concludethat an intervention based on motivational interviewing, goalsetting, et cetera was effective in promoting physical activity.But actually, that wouldn't have been a valid conclusion,because a number of the components of the intervention

  • 08:24

    GRAHAM MOORE [continued]: were fully delivered.And actually, although the intervention did work,and it worked through promoting internal motivation,the processes through which that happenedwere actually somewhat different to whatwe'd initially anticipated.So the intervention worked for many patients,despite the fact that it looked somewhatdifferent to what we'd initially expected it to look like.Embedding a detailed mixed-method process evaluation

  • 08:44

    GRAHAM MOORE [continued]: within the trial provided vital insightsinto how the intervention worked and for whom.The National Excise Referral Schemecontinues to run throughout Wales,accepting around about 1,500 new patients each year.Since the trial was completed, much of the policy emphasishas been on emphasizing the social aspectsof the intervention, which seemedto be more important than some of the one-to-one aspects

  • 09:05

    GRAHAM MOORE [continued]: in promoting change than we'd initially anticipated.

Abstract

Dr. Graham Moore discusses a mixed-methods study he undertook to evaluate medical interventions in the National Exercise Referral Scheme. Using randomized trials, he found that the referrals had a positive outcome. But incorporating other methods revealed that the program was effective for different reasons than expected.

Looks like you do not have access to this content.

Evaluating the National Exercise Referral Scheme Using Process Evaluation Methods

Dr. Graham Moore discusses a mixed-methods study he undertook to evaluate medical interventions in the National Exercise Referral Scheme. Using randomized trials, he found that the referrals had a positive outcome. But incorporating other methods revealed that the program was effective for different reasons than expected.

Copy and paste the following HTML into your website