Skip to main content
SAGE
Search form
  • 00:10

    RAY PAWSON: I'm Ray Pawson.My position is Emeritus Professorof Social Research Methodology at the University of Leeds.And today I'm going to be talking to you about a methodcalled realist evaluation.And we ought to begin with some definitions of the term.

  • 00:34

    RAY PAWSON [continued]: So start with the evaluation bit.Now, evaluation is normally knownas program evaluation or intervention evaluation,or policy and intervention.And the basic idea is to come to some judgement of the worthor the method of some social policy.

  • 00:59

    RAY PAWSON [continued]: Now, that policy can be a relatively smallconfined thing.So you could say, a traffic calming scheme, bumpsand signage, does that reduce traffic accidents?Does that reduced traffic fatalities in an area?

  • 01:21

    RAY PAWSON [continued]: Or it could be a little bit more grand than that.So an evaluator might be called to look at somethinglike beginning reading schemes.You might know that there's controversy about howbest to teach children reading.And an evaluator might well try to assessthe different schemes-- look and say,phonetic approaches-- to find out which one works better.

  • 01:48

    RAY PAWSON [continued]: Or it could be a quite grand scheme.Much evaluation is done in the National Health Service.And the health services are constantly being reorganized.And an evaluator might be called into test the effectiveness of a particular reorganization.

  • 02:10

    RAY PAWSON [continued]: So currently, we're working under something calledclinical commissioning groups.And an evaluator could try to find outif that new method of organization was efficient.So that's evaluation research.I should add that it's the main method or the main used methodin the entire social science spectrum.

  • 02:35

    RAY PAWSON [continued]: More money, more time, more effortis spent on evaluation research than any other method.So that's the evaluation research bit.Now, realism is quite different.Basically, it's a philosophy.It's a philosophy of social science research.

  • 02:56

    RAY PAWSON [continued]: And it stands between, on one hand, constructivism,and on the other hand, positivism.And realism sits in the middle.I'll try to explain the basic ideas, someactually very complicated ideas fairly swiftly.So a constructivist would say, there isn't one reality.

  • 03:19

    RAY PAWSON [continued]: Reality is in the eye of the beholder.There are multiple realities.Positivists would say, the social worldis no different from the natural world.And if it's carefully approached through observationand measurement you can come to an understandingof the real world.Realism sits in the middle.

  • 03:42

    RAY PAWSON [continued]: It says, there's contestation.There's argument.There's disagreement about many things.But there is an underlying reality, a reality thatdoesn't shift very, very much.And if you understand that, you can understandhow the world's organized.

  • 04:05

    RAY PAWSON [continued]: Try a simple example.Imagine you're a student and you've done this essay.And it's your pride and joy.And you think your essay is wonderful.And it's examined and your professor, me, comes backand awards it a third.So are we in a world of multiple realities?Is there one true mark or what?

  • 04:28

    RAY PAWSON [continued]: A realist would say, well, you'vegot to understand what's real about the system.And the system is that there is a structure in place.There's a structure in place that depends on my experienceand power and the experience and learning of the second marker,and the experience and learning of the external examiner.

  • 04:52

    RAY PAWSON [continued]: And it's that that decides the mark of your essay.So in other words, you need to understand the underlyingstructures in order to do research.OK, we've established that our field is evaluation research.There's good news and bad news here.

  • 05:14

    RAY PAWSON [continued]: The bad news is that there are dozens and dozensof different ways of approaching evaluation research.And on this slide, there's a beginning list.And believe me, it's only a beginning listof the different methods of doing evaluation research.The good news is that I'm going to be talking about one.

  • 05:37

    RAY PAWSON [continued]: That's is realist evaluation.And I'll say a little bit more about what that entails.The basic idea is that this is called program theoryevaluation, sometimes call theory-driven evaluation.

  • 05:57

    RAY PAWSON [continued]: And the idea, the very basic fundamentalis what the evaluator tries to dois to test the program theory.Tell you what program theories are in a moment.But basically, these interventionsare many, many different things.So an intervention would be the amount of moneyspent on pursuing this policy.

  • 06:22

    RAY PAWSON [continued]: For a practitioner, an interventionwould be the daily work, the readingscheme that they deliver.For a manager, it would be a bunchof practitioners, which they have to recruit, manage,and organize.So interventions are all of those things.

  • 06:43

    RAY PAWSON [continued]: But for a realist evaluator, programs are theories.And they're theories, in this sense, that at some pointsome policymaker will say, we have a problem.We'd like to remedy that problem.We'd like to alleviate that problem.And to do so we're going to provide these resources.

  • 07:06

    RAY PAWSON [continued]: Those resources, hopefully, will make people think differently,act differently, and behave differently.And if the program's a success that theoryshould come to pass.OK, so the simplest possible illustrationof a theory-driven evaluation, this is real.

  • 07:29

    RAY PAWSON [continued]: This actually happened in the 1990sunder the auspices of a group called the Health EducationalAuthority.Basically, the problem that they were trying to deal withwas the level of fitness, the levelof physical activity of young teenagers, especially girls,was very, very low.

  • 07:52

    RAY PAWSON [continued]: And their job was to promote fitness.So they scratched their heads-- thisis their theory-- about how they couldimprove these levels of theory.And so their thinking went like this--what's the problem with young teenage girls?Basically, it's the bedroom culture.They sit in their bedrooms.

  • 08:14

    RAY PAWSON [continued]: And they come under various influences.But none of them are to do with physical activities.So in girls' teenage magazines-- which were popular at the timeand are probably defunct, as far as I know--the characters, the heroes, the role modelswere soap stars, film stars, rock stars,none of them portraying this healthy lifestylethat the Health Education Authority source after.

  • 08:46

    RAY PAWSON [continued]: So the idea was can we persuade the editor of these magazinesto use healthier role models?And they did, indeed.And the healthy role models were very famous sports stars.Hence, this theory, in the trade,is known as Dishy David Beckham theory.

  • 09:07

    RAY PAWSON [continued]: What the researcher did-- and you can see this on the slide--is basically approach the girls directly with the theory.She put the theory to the girls in question.So she asked a question like does the factthat these rather good-looking blokes are in your magazines,does it influence you to take more part in sports?

  • 09:35

    RAY PAWSON [continued]: And what the girls replied-- I'm notreally good at a 13-year-old girl's voice here.So you'll have to put up with that as well--but basically, they giggled.And said, no.It doesn't influence us to play sport.It influences us to watch sport.It influences us to watch David Beckham.

  • 09:58

    RAY PAWSON [continued]: So they giggled away, basically sayingthat these images exercised their minds ratherthan their bodies.So that's a brief illustration.There's the theory.And there's the theory breaking down under interrogation.Now, you can do a bit more work on this.You could say, OK.

  • 10:20

    RAY PAWSON [continued]: How could we improve the theory?Who would be a better role model than Dishy Dave?Or you could say, well, Dave wasn't so good at sellingfitness to girls.But how might he act as a role model?What might he be able to sell rather more efficiently?

  • 10:45

    RAY PAWSON [continued]: So that, slightly trivial and silly as it was,was the basic idea.You understand the program theory.And you test the program theory.And that's the basis of a theory-drivenor a realist evaluation.The basic concepts are a bit of a mouthful.

  • 11:05

    RAY PAWSON [continued]: Basically, realist evaluation dealsin what we call mechanism, context, outcome,configurations.So let me break down those terms.These are the terms through which we approach the theoryand test the theory.The easiest one is outcomes.

  • 11:26

    RAY PAWSON [continued]: Outcomes is what it says on the tin.You're looking to see if the program works or not.You're trying to find if it improves reading, if itreduces traffic fatalities.So you're measuring outcomes.And that's a crucial part of the evaluation.Let's flip to mechanisms.

  • 11:47

    RAY PAWSON [continued]: Mechanisms are what it is about the program, what is actuallydelivered within the program that makes peoplereason in a different way.So when you're investigating any program, whatever it is,you need to ferret out the mechanisms whichthe persuasive mechanisms that try to make peoplealter their behavior.

  • 12:13

    RAY PAWSON [continued]: Context is another important concept.And basically, what that says is that no programworks perfectly.No intervention works perfectly.They work much better in some circumstances for some peoplein some contexts.

  • 12:34

    RAY PAWSON [continued]: The catch phrase there-- that's often used in evaluationis-- what works in Wigan on a wet Wednesdaywon't work in Thurso on a thunderous Thursday.Got that?So that's what we're looking for in realist evaluation,and an understanding of those three components.The mechanisms that drive the intervention, the contexts,the locations, the places, the people, where they work,and outcomes, whether they work or not.

  • 13:05

    RAY PAWSON [continued]: I'll now want to enlighten you a little bit moreby giving a real example of research.Nick Tilley is a colleague of mine.And I should add that I co-wrote the book Realistic Evaluationwith him.And this was a very early exampleof how we discovered the importanceof the idea of mechanisms, the mechanismsthat lie within an intervention.

  • 13:40

    RAY PAWSON [continued]: Now, so to take you back in history a little bit,this was in 1990.And it was a study of a crime prevention initiative.And the particular problem was the repeat victimizationof small shops, these rows shops that occur in every suburb,in every town.

  • 14:03

    RAY PAWSON [continued]: They were highly victimized.They didn't have the money to do any fancy burglar alarm systemsand protective measures.And the local crime prevention teamset up a number of initiatives to try and reducecrime in these particular areas.

  • 14:24

    RAY PAWSON [continued]: One of them was known as a silent alarm.And I think you'd be able to guess, from the name,from the very name, how a silent alarm works.What's the mechanism?Burglar enters the premises.They aren't aware of an alarm.So they carry out their business.The alarm actually rings in the police station.

  • 14:46

    RAY PAWSON [continued]: The police clamber into their carsand catch the burglar red handed,thus preventing them burgling again.So that's the theory.And the mechanism is silence.Basically, here's a prevention measurethat the burglars are unaware of.

  • 15:08

    RAY PAWSON [continued]: The sad part of this tale was this brilliantly simple idea.It didn't seem to be working particularly well.And so part of our job, at the time,was to investigate why this was so.It seemed a simple, clever little idea.Why wasn't it working particularly well?

  • 15:29

    RAY PAWSON [continued]: And so how we did it, we asked the crime prevention officers--the people that had been responsible for mountingthe investigation-- we were interested in the programtheory.What was the idea behind silent alarms?Then I recall that they thought we were slightly mad.We kept asking them, how do these things work?

  • 15:50

    RAY PAWSON [continued]: And they responded, the alarm goes off in the police stationbut not in the premises.The burglars can't hear it.And that's how it worked.So we prodded and prodded away at the program theoryuntil we finally got a better understanding of mechanisms.

  • 16:15

    RAY PAWSON [continued]: And we got to that by asking a slightly different question.Where, in what premises, what kind of shopsdo these work better?And the crime prevention officer hadn't thoughtof telling us this before.But what he went on to say was that they workedin Asian-owned businesses.

  • 16:37

    RAY PAWSON [continued]: This research was done in Leicester, now knownas Little India.And Basically, he said, that when we've had successthey've been in this area of Little India.And we said, why should that be the case?And he said, it's to do with secrecy.

  • 17:01

    RAY PAWSON [continued]: In order for these alarms to work,it's no good if the burglar knows that one of these thingshas been installed.Because they can be in and out much faster than the copsare able to respond.So it's absolutely vital that the presence of the alarmis kept secret.

  • 17:23

    RAY PAWSON [continued]: And in this particular community,they're much better at doing it.Basically, because the family, it constitutes the labor force.And so the family is able to keepthe presence of these alarms secretmuch better than when people are hired and fired much morerandomly to act as the staff.

  • 17:51

    RAY PAWSON [continued]: So there was a refined theory.We've now got an understanding, a refined understandingof the mechanism.And what it says is, basically, this combined mechanism,silence and secrecy, allows this apparatus to work better.And we went on to check this out.Was it the case?And indeed, he was right.

  • 18:11

    RAY PAWSON [continued]: The few successes of this alarm had occurred in Little India.Then the evaluator was able to adapt the program.Any future alarms that were installed,there was this insistence, this clear set of instructionsthat the presence of the alarm must be kept secret.

  • 18:34

    RAY PAWSON [continued]: And with that revision, the programs improved.And then after awhile, as happensin most crime-prevention schemes,the criminal community got to knowof the presence of these alarms.And so their effectiveness diminished thereafter.

  • 18:56

    RAY PAWSON [continued]: So there you have a tale of what works for whom in whatcircumstances.What works was the particular combined mechanism.For whom and in what context was these particular groupsof business.

  • 19:18

    RAY PAWSON [continued]: And the outcomes were the rise and fall in crime ratesas a consequence of the program.OK, so in conclusion, just two thoughts.The first is to stress, again, the importanceof this kind of evaluation research.

  • 19:40

    RAY PAWSON [continued]: It is the main sort of research that'sfunded and practiced in the UK.And so if you want to be a social researcher,keep an eye open for the possibilities in evaluationresearch.And the second thing is just to say,once more, the realist motto that evaluation is usuallythought of asking a rather blunt question.

  • 20:04

    RAY PAWSON [continued]: Will it work?And will it not work?My experience is that it's an unsettled questionto ask in the first place.And a much more relevant question is the realist motto.What works for whom, in what circumstances?

Video Info

Publisher: SAGE Publications Ltd.

Publication Year: 2017

Video Type:Tutorial

Methods: Program evaluation, Realism

Keywords: burglary victims; context effects; crime prevention measures; effectiveness; effectiveness of crime preventn prgs; exercise and fitness; family business; magazine types; mechanisms; outcomes; policy development; role models; Secrecy and leaks; Silence, silences, and silencing; Social policy; Structural power ... Show More

Segment Info

Segment Num.: 1

Persons Discussed:

Events Discussed:

Keywords:

Abstract

Professor Ray Pawson explains realist evaluation, which examines the effectiveness of social policies. This type of research looks at the mechanisms, context, outcome, and configurations of a given policy to determine how or whether it is successful.

Looks like you do not have access to this content.

An Introduction to Realist Evaluation

Professor Ray Pawson explains realist evaluation, which examines the effectiveness of social policies. This type of research looks at the mechanisms, context, outcome, and configurations of a given policy to determine how or whether it is successful.

Copy and paste the following HTML into your website