Skip to main content
Search form
  • 00:09

    [Introduction to Measuring Stakeholder Engagement withResearch Engagement Survey Tool (REST)]

  • 00:16

    MELODY S GOODMAN: Hi, I'm Melody Goodman.[Melody S. Goodman, Associate Dean for Research,NYU School of Global Public Health]I'm the associate dean for researchand an associate professor of biostatisticsat New York University School of Global Public Health.My research focuses on racial and ethnic and socioeconomicdisparities across a broad array of health outcomes.I'm really interested in enhancing the infrastructure

  • 00:36

    MELODY S GOODMAN [continued]: for community academic partnershipsand evaluating the efficacy of community academic partnershipsto impact scientific discovery.So in this video, we're going to talkabout stakeholder engagement, and the engagement principles,and some ways that my team and I have been thinkingabout measuring engagement of non-academic partners

  • 00:60

    MELODY S GOODMAN [continued]: in research projects.[What is the role of stakeholders in public healthresearch?]So there's an African proverb thatsays, if you want to go fast, go alone.But if you want to go far, go together.And I think that engaging stakeholdersin the process of doing research is

  • 01:22

    MELODY S GOODMAN [continued]: really important for a couple of reasons.It makes sure that we're looking at research questions thatare important to the populations that they'redesigned to help or serve.It make sure that the ideas are rooted in realityand can be implemented in the real worldand it also helps us develop sustainable solutions that

  • 01:43

    MELODY S GOODMAN [continued]: can be embedded in our structures,in our institutions, and our processes beyond our researchprojects.So non-academic stakeholders can playa variety of roles in a project from minimallyengaged to sort of really fully formed partnerships.And I think the goal for the type of researchthat I'm interested in is to develop these really

  • 02:05

    MELODY S GOODMAN [continued]: fully formed partnerships where non-academic stakeholdersinvolved in everything from developingthe question, selecting the methods,and the approaches that we will use.Interpreting the findings from the results,but also helping us disseminate the resultsto appropriate audiences.[How can researchers measure stakeholder's engagement

  • 02:26

    MELODY S GOODMAN [continued]: in their study?]So we've developed a survey tool whichwe call the Research Engagement Survey Tool or the RESTfor short that allows researchers to measurenon-academic partner engagement in the research processfrom the non-academic partner's perspective

  • 02:48

    MELODY S GOODMAN [continued]: and we align our measure with eight engagement principlesfrom the literature.They've come from the community based participatory researchliterature, from the Patient Centered Outcomes Researchliterature, we borrowed some informationfrom community campus partnershipsfor health which is a major organization in this space.It also really thought about community engagement research

  • 03:09

    MELODY S GOODMAN [continued]: in general.We also use a stakeholder engaged approachin our measurement development processso we had non-academic partners workingwith us on developing this tool throughout our researchprojects[How do you define stakeholder engagement?]

  • 03:31

    MELODY S GOODMAN [continued]: Key point number one is the importanceof stakeholder engagement.And we've defined stakeholder engagementas the process of working collaborativelywith groups of people who are interested in a certain topic,have an affinity because of geographic proximity,or affected by a certain issue.It could be a disease or illness but it doesn't have to be.

  • 03:53

    MELODY S GOODMAN [continued]: It could also be things like food and security,or other social determinants of healthand these are people that have some stake in the game.There's a reason why they're interested in workingwith you because it really impacts them personally,professionally, or both.And it's really helpful to engagedifferent types of stakeholders in your process.

  • 04:16

    MELODY S GOODMAN [continued]: For example, if I'm interested in obesity,I'd want to include people who do physical activitywork, but also people who study diet,and people who study behavior.But I probably would also really importantly includesome people who need to lose some weight because those are

  • 04:37

    MELODY S GOODMAN [continued]: the people that the intervention or the project is designed for.So it's important when you're thinkingabout your stakeholders to think about the full universeof people that are impacted.And oftentimes, we think about stakeholders as individuals.But in a lot of my work, stakeholdershave often been institutions.

  • 04:58

    MELODY S GOODMAN [continued]: They could be health departments, or health systems,or health insurance providers, or payers in the processand so think broadly when you thinkabout the key stakeholders that arenecessary to help you address the problem that you'reworking on.[Why is it important to engage the right stakeholders?]

  • 05:21

    MELODY S GOODMAN [continued]: It's really important to include key stakeholdersbecause they have a sense of how things work on the ground.Oftentimes, people have an idea of howthings work theoretically but how they actuallywork in practice could be very differentand these could be simple things like scheduling an appointmentto see a doctor.How one thinks that might work could

  • 05:43

    MELODY S GOODMAN [continued]: be really different than how it worksat least from the perception of the front desk staff whoactually take the appointments.And so if I'm interested in some project thatis around appointment scheduling,it would be really important for meto have in my stakeholder group the people whomake the actual appointments because they understand

  • 06:04

    MELODY S GOODMAN [continued]: how that process works.And so for any project that you're working on,you really want to think through whois going to be involved in the implementation of that workand do you really understand their roles,and their responsibilities, and how the process is assigned.Most of the time we don't but a great way to figure that out

  • 06:25

    MELODY S GOODMAN [continued]: is just to include someone who isreally knowledgeable about that part of the practice.It's really important when you think about stakeholderengagement to think about the complete universe of allof your stakeholders.So usually you start with researchersbecause we're the ones interested in conductingthe research but your stakeholders may also include

  • 06:46

    MELODY S GOODMAN [continued]: patients, or consumers, clinicians, funders,health care providers, insurance payers, or buyers,or drug makers, policy makers those whodetermine health policy.Those in the life science industrythat make the drugs or the equipment that we need.So it's really important that you

  • 07:07

    MELODY S GOODMAN [continued]: think about all the different people thatintersect with your potential research questionand what they have at stake because thoseare your stakeholders for your project.[How was the Research Engagement Survey Tool developed?]

  • 07:27

    MELODY S GOODMAN [continued]: The REST tool was developed usingmixed methods both quantitative and qualitative researchmethods through sort of a long iterative process.We started with a Delphi approachand the Delphi approach is a way of gathering stakeholder inputfrom a variety of stakeholders whoyou deem to be experts in the field that you're interested in

  • 07:51

    MELODY S GOODMAN [continued]: and to work with that sort of group to reach consensus.So it is an iterative multi round processwhere participants get to give their opinions on something,get some feedback.So the way we did it was that our participants couldsee things like engagement principles,

  • 08:14

    MELODY S GOODMAN [continued]: definitions for those engagement principles,and the sample items that we were thinking about usingto measure those principles.And they can give us some feedbackif they thought they were good, if they thought they were bad,or if they thought that they should be modified in some way.After they give us their feedback,we combine the feedback from all the membersof our Delphi process.

  • 08:35

    MELODY S GOODMAN [continued]: So in our project, we started with 19 membersand we finished with 18 members through a five round process.So after each round, each member gets a tailored reportwhere they see overall what everyone has reportedand then they see their own response.And this allows a member of the Delphi panelto really think through what they

  • 08:56

    MELODY S GOODMAN [continued]: believe in light of what other people are sayingand gives them an opportunity to rethinkor to sort of double down on their original opinion.And any items that we can't reach consensus on,move to the next round.So if an item reaches consensus, it's settled and we take it off

  • 09:18

    MELODY S GOODMAN [continued]: but items where we could not reach consensusare moved to the next round.So in our study, we define consensus as 80% agreementamongst our set of experts.So if we had 80% of our 19 participants agreeor our 18 participants in the end,then we consider that to be consensus.I think the other thing that we did that was really important

  • 09:40

    MELODY S GOODMAN [continued]: was that we stratified our analysisby whether the respondents were academic researchersor if they were non-academic partners.So we had experts that were both,we thought this was really important to developour measure but we wanted to make sure that we're reallythinking through who's saying what because the idea isto develop a measure for the non-academic stakeholders

  • 10:03

    MELODY S GOODMAN [continued]: to take.[How did you define the different levels of partnerengagement?]Using the Delphi process, we cameto consensus on five levels of partnership that movefrom minimal engagements to fully engaged partnerships.

  • 10:25

    MELODY S GOODMAN [continued]: The first level of engagement is outreach and education.This is by far we think the most common type of engagement thathappens in health professions.We are working on a research study,we have some important findings, we'retrying to outreach to people thatmay be impacted by these findings

  • 10:45

    MELODY S GOODMAN [continued]: and provide them education on the issuethat we think is important.The thing about outreach and engagementis while there's not a lot of non-academic stakeholder inputinto the process.There is potentially a lot of benefitfor non-academic stakeholders to get from the research process

  • 11:07

    MELODY S GOODMAN [continued]: and so outreach and engagement is often the kind of workyou see in the field.The second level of engagement we define as consultation.So researchers are working with non-academic stakeholderssort of as consultants.They're asking them for advice.They're really interested in their opinions.But ultimately, the decisions and the way the project moves

  • 11:31

    MELODY S GOODMAN [continued]: is determined by the researchers.And so while non-academic partnersare included in this process, they'rereally used as consultants just as a sounding boardto provide advice.The third level of engagement is cooperation.This is when researchers are askingnon-academic partners for help, and advice, strategies

  • 11:54

    MELODY S GOODMAN [continued]: on how to address a particular part of their projects,the creation of study questionnaires,developing the actual intervention itself.They're asking community partnersto help them in some way but it's usually really specificand really defined.The fourth level of engagement is collaboration.

  • 12:16

    MELODY S GOODMAN [continued]: This is where partners are reallyengaged in the process from the study design,to the data analysis, to the dissemination.All partners are valued, they all benefit from the research,they're all involved in decision making,and they share the power and the resources

  • 12:37

    MELODY S GOODMAN [continued]: that come from the projects.And the fifth level of engagementis a partnership which is basically a long termcollaboration.And so once you've done collaboration once,most people find the benefit of itand we'll continue to work with the same set of partnersfor multiple projects.And so partnerships are long term collaborative efforts.

  • 12:60

    MELODY S GOODMAN [continued]: So when we started thinking about these levelsof engagement, we thought of it as a continuum,sort of moving from left to right so to speak.But one of the things that we saw in our data analysiswas that outreach and education seemto be at a higher level than consultation

  • 13:20

    MELODY S GOODMAN [continued]: even though we thought it had less engagement.And this caused us to rethink the waywe thought about engagement and moved from a one access modelto a two-access model.We think this is really important because engagementhas two major components.What non-academic stakeholders put into the researchbut also what they get out.

  • 13:41

    MELODY S GOODMAN [continued]: And so in our figure, you will seethat while outreach and education hasminimal non-academic partner input,there's potentially a lot that non-academic partners canget out of having researchers come outand do outreach and education on a specific topic.Similarly for consultation, there's

  • 14:02

    MELODY S GOODMAN [continued]: a little bit more non-academic inputbut potentially the non-academic partnersdon't get that much out of it.They're just being used as consultantsand they're not really involved in the process.But as we move along the spectrum,we think cooperation, collaboration, and partnershipmove so that they're both increasing and non-academic

  • 14:24

    MELODY S GOODMAN [continued]: partner input into the project but also the potential benefitsthat come out for non-academic stakeholdersas we move along this sort of double access spectrum.[How is stakeholder engagement measured in the REST survey?]So we developed the Research Engagement Survey Tool or REST

  • 14:45

    MELODY S GOODMAN [continued]: for short to align with eight engagement principles.Each principle is measured by 3 to 5 items.And so the principles are focusedon community perspectives and determinants of health.This is about centering the work so that it's community centeror patient centered depending on the type of workthat you're doing.

  • 15:05

    MELODY S GOODMAN [continued]: The second engagement principle is that partnership inputis vital, engagement only works if partnersare willing to provide the information necessary to movethe research agenda forward.The third engagement principle ispartnerships sustainability to meet the goals and objectives.You need to make sure that your partnership has

  • 15:26

    MELODY S GOODMAN [continued]: sustaining power at least to last throughout your researchprojects.And so you want to think about thisup front to develop a partnership thatwill last the length of your project and hopefully beyond.The fourth engagement principle isto foster co-learning, capacity building,and co-benefit for all partners and this is really important.

  • 15:48

    MELODY S GOODMAN [continued]: The way that you keep a partnership goingor the way you sustain a partnershipis to make sure that all partners havesomething to benefit by their participation.They're either learning or gettingsomething that they need out of the partnership.The fifth engagement principle isto build on the strengths and resources within the communityor the patient population.This is really important for researchers

  • 16:09

    MELODY S GOODMAN [continued]: because we often come from the negative limitationsperspective.But doing community engaged work has reallymade me think about coming from the strengths perspective.Instead of thinking about what we don't have,think about what we do have, and whatwe can put together to get to where the partnership needsto go.The sixth principle is to facilitatecollaborative and equitable partnerships.

  • 16:32

    MELODY S GOODMAN [continued]: And so this aligns with some of the other onesbut you really want to force on this ideathat everyone is collaborating, that the partnership isequitable in terms of power and balance,and that everyone has input and say into the partnership.The seventh engagement principle isthat you want to involve all partners in the disseminationprocess.From research dissemination is so key,

  • 16:53

    MELODY S GOODMAN [continued]: how do you get people to know what you've just discovered.And it's really important that all the partners involvedin that dissemination process because theyknow the appropriate stakeholders thatwould find that information the most useful.And the eighth engagement principleis one that really came up during our Delphi process,it was the one engagement principlethat we didn't start with that was added because our community

  • 17:16

    MELODY S GOODMAN [continued]: partners felt so strongly about itand that is to build and maintaintrust in the partnership.Trust is so hard to build but it's even harder to maintain.But good partnerships have trusting, and open, and honestcommunication in terms of what the partnership is doingand where the partnership is going.

  • 17:37

    MELODY S GOODMAN [continued]: [How can researchers use the REST tool in practice?]So we're excited for people to use the REST tool in practiceand we've developed two versions.One is a comprehensive version that has 32 itemsand one is a condensed version thathas nine items that we think people will

  • 17:57

    MELODY S GOODMAN [continued]: find really useful in practice.And the idea is to think about your engagementas part of your science so not think of it as somethingthat you're adding on to the science.So in the same way that you woulddo any other scientific endeavor,you usually want to measure engagement at the beginning,some points in the middle, and some points at the end.And we think it's important to think about when it's

  • 18:19

    MELODY S GOODMAN [continued]: appropriate to use the comprehensive measure when youwant to really look at all eight engagement principlesbut when it's OK to do the shorter condensed nine itemversion to just do a checkup.And we think that people will use this in multiple waysto sort of establish where their partnership is,to think about using the REST as a conversationstarter for partnerships.

  • 18:40

    MELODY S GOODMAN [continued]: So this is where we are, where do we want to go,and how do we get there.But also to look at how partnerships have evolvedover the course of a project, so whereare we compared to where we were three years ago.One of the major reasons the rest was developed thoughis because a lot of this partnership workhad been done using qualitative approaches.And what qualitative approaches arekey to finding important information,

  • 19:02

    MELODY S GOODMAN [continued]: it's really hard to scale up across large scale projectsand to really allow us to compareacross different studies.And so we think that this short quantitative surveytool can really be used to allow usto look at engagement across a variety of projectsor within a single project over time.

  • 19:22

    MELODY S GOODMAN [continued]: And this is really important for the field of communityengaged science which is really a lot relied on best practicesand lessons learned.But we hope that once people start using the researchengagement survey tool, we can startto develop evidence-based approaches to stakeholderengagement and understand how engagement impactsthe scientific process and ultimately

  • 19:43

    MELODY S GOODMAN [continued]: scientific discovery.[How did you test the validity of the REST tool?]So one of the things that we were really interested in doingfor the REST tool was to comprehensively validatethe measure and this is really importantbecause there's not a lot of measures that assess partner

  • 20:04

    MELODY S GOODMAN [continued]: engagement out in the literature and thereis no gold standard measure for howyou assess partner engagement.So how do you validate a tool wherethere is no gold standard?So we took the approach of using multiple forms of validationto try to look at how well our measure performs.One of the things that we wanted to look at

  • 20:25

    MELODY S GOODMAN [continued]: was internal consistency.So we have 3 to five items that are alignedwith each engagement principle.If the items are internally consistent meaning,if they measure the same engagement principle,we should see similar responses from participantsabout how they feel in all of the items relatedto an individual engagement principle.

  • 20:46

    MELODY S GOODMAN [continued]: The way we look at this is we use a statistic calledCronbach's alpha.It allows us to look at how well each individual item iscorrelated with all the other items in the scaleand if they all should be included or if one or moreof those items should be removed.Luckily, for us our tool showed really good validationstatistics.

  • 21:06

    MELODY S GOODMAN [continued]: So Cronbach's alpha is usually deemedacceptable with the level greater than 0.7.In our study, we had Cronbach's alpha greaterthan 0.8 for each of the engagement principles on boththe quantity and quality scale.The other thing we wanted to look atwas correlative validity.So how well does our measure correlate with other things

  • 21:29

    MELODY S GOODMAN [continued]: that we hypothesize it should be associated with?And also sort of discriminant validityit should not be associated with things that wethink it has no relation to.So we look through the literatureand we found several other measuresrelated to engagements, partnerships, collaborations,and coalitions.

  • 21:50

    MELODY S GOODMAN [continued]: While none of these measures were measuringwhat we wanted to measure which is partnered engagementand research, they did measure certain aspectsof what we were trying to measure.For instance, some of them measured trust, or synergy,or partnerships.And so we looked at the associationbetween how participants responded to our measure,to how they responded on these similar measures

  • 22:12

    MELODY S GOODMAN [continued]: that we pulled from the literature.What we think is really interestingis that we saw statistically significant correlationswith all of the other measures that we hypothesizedwould be correlated.For example, trust in medical researchers.But these correlations were either negligible, low,or moderate.And this is really important to us

  • 22:33

    MELODY S GOODMAN [continued]: because this was exactly what we wanted to see.We wanted our measure to be correlated with other measuresbut not highly correlated.Because a high correlation would assumethat we're measuring the same thingthat other measure is also examining.But in our case, we're looking at moderate, negligible,or low correlation.So yes, our two measures are associated

  • 22:53

    MELODY S GOODMAN [continued]: but we believe we're actually measuring something quitedifferent than those other measuresand this really helped us demonstrate this statistically.[What is the importance of the REST tool for measuringstakeholder engagement?]We believe the research engagement surveytool is really a major contribution to the literature.

  • 23:15

    MELODY S GOODMAN [continued]: There's not many survey tools that quantitativelyassess the level of partner engagementfrom the partner's perspective.And this is really important as westart to work on more engagement and participatory researchapproaches in our standard scientific process.And so understanding how engagement

  • 23:39

    MELODY S GOODMAN [continued]: is happening in a study, how it is evolving over time,and how it is impacting important research outcomes.For example, the speed of recruitmentor the diversity of the population recruitedare really important for demonstrating that engagementis really a major component of our science and not somethingthat we're adding on to the benefit of the partners

  • 24:02

    MELODY S GOODMAN [continued]: but really, I think engagement makes you a better scientist.We asked better questions, we do workthat is meaningful to the people that we'retrying to help and serve and we place it in a real contextwhere it can be implemented in meaningful and useful ways.[Conclusion]

  • 24:25

    MELODY S GOODMAN [continued]: Stakeholder engagement may be new to many peopleand you may notice that funders are startingto require engagement as part of the process for securing fundsand this may be intimidating to some people.But what I will say is that everything youneed to know about engagement, you learned in kindergarten.There's a poem from Robert Fulghum that

  • 24:46

    MELODY S GOODMAN [continued]: talks about being nice to other peopleand playing fair and sharing things and that's all true.If you are nice to other people, and share things,and play fair, engagement becomes really easy.And you will notice that in the world,it's best to hold hands and stick togetherwhich is what we learned in kindergarten.

  • 25:07

    MELODY S GOODMAN [continued]: And so while engagement takes time, more time than many of ushave, and additional resources, Ithink you will find that it will makeyou a much better scientist.So I hope you look at some of the further readingsand dig into some of the new engagement strategiesthat are being proposed.Hope this was useful to you.


Melody Goodman, Associate Dean for Research at the NYU School of Global Public Health, discusses measuring stakeholder engagement with the research engagement survey tool (REST), including the stakeholder's role, engaging the right stakeholder, and the validity and importance of the survey tool.

Looks like you do not have access to this content.

Introduction to Measuring Stakeholder Engagement With Research Engagement Survey Tool (REST)

Melody Goodman, Associate Dean for Research at the NYU School of Global Public Health, discusses measuring stakeholder engagement with the research engagement survey tool (REST), including the stakeholder's role, engaging the right stakeholder, and the validity and importance of the survey tool.

Copy and paste the following HTML into your website