Skip to main content
Search form
  • 00:17

    KRISHMA LABIB: My name is Krishma Labib,and I am a PhD Student at the Amsterdam University MedicalCenter's Department of Ethics, Law and Humanities.I have a background in neuroscience and bioethics,and I'm interested in contributingtowards fixing the problems that are currently plaguing science.I became acquainted with the field of research integritydue to this interest.

  • 00:38

    KRISHMA LABIB [continued]: And now, I'm happy to be involved in the StandardOperating Procedures for Research Integrity Projectto realize this.

  • 00:46

    JOERI TIJDINK: Hello, my name is Joeri Tijdink,and I'm a Researcher at the Department of Ethics, Lawand Humanities at the Amsterdam University Medical Center.Today, we're going to talk about the Delphi consensus studyon important topics for research integrity.And in this video, we explain to you what

  • 01:08

    JOERI TIJDINK [continued]: is the Delphi study, exactly?What are the outcomes of our studythat might be helpful for you?And what you can do with the results of the Delphi studyfor future guidance, and guidelines, and policy-making.

  • 01:36

    JOERI TIJDINK [continued]: So why a Delphi consensus study?We have to start with the beginning,because we have to start with research integrity.And as you see, research integrityis crucial for the conduct of good quality,trustworthy research.And so far, most efforts to improve research integrityfocus on the behavior of individual researchers.

  • 01:57

    JOERI TIJDINK [continued]: But, however, research practice isinfluenced by multiple factors.And it's not only in the individual research.There's, of course, the system of science.For example, the competition structures,the assessment criteria of individual researchers.There's this institutional research climate.For example, the culture at the department.

  • 02:17

    JOERI TIJDINK [continued]: And there is, of course, the individual research,the viruousness of individual researchers.And to adequately address research integrity,it's crucial to tackle not only the individual,but also the system of science in institutional researchclimates.And therefore, we have to know what is actuallyimportant for both the system of science and the institution.

  • 02:42

    JOERI TIJDINK [continued]: So for example, research performing organizationshave a direct influence on the institutional research climate,while research funding organizationshave a direct influence on some elementswithin the system of science.For example, the availability of funds.So they have, indirectly, a role and a responsibilityfor responsible research practices.

  • 03:04

    JOERI TIJDINK [continued]: And because of their ability to set requirements on RPO,Research Performing Organizations and ResearchFunding Organizations.Therefore, we have to know what are the crucial elementsthat institutions and funding bodies can do

  • 03:24

    JOERI TIJDINK [continued]: to foster research integrity?And that individual researchers are not responsible for.

  • 03:40

    KRISHMA LABIB: So considering what Joeri mentioned,to foster research integrity, research performingand research funding organizationsshould develop a comprehensive plan outliningall the institutional strategies, policies,infrastructures, and procedures that they will implementto foster research integrity.

  • 04:01

    KRISHMA LABIB [continued]: Now, existing literature, such as the Bonn PRINTEGERStatement, but also, institutional and nationalcodes of conduct, already have some suggestionson what kinds of topics should be included in this researchintegrity plan of research performing and fundingorganizations.Topics that have been previously mentioned as important

  • 04:22

    KRISHMA LABIB [continued]: are things like; research integrity education, mentoring,dealing with breaches of research integrity, et cetera.However, to date, there is no international European-levelconsensus among policymakers and research leadersabout which topics should be included in the researchintegrity plans of all research performing and research funding

  • 04:45

    KRISHMA LABIB [continued]: organizations across Europe.

  • 04:55

    JOERI TIJDINK: So thank you, Krishma, to highlight this.So just to get a good feeling of what the study aim isof this Delphi study, it is to obtain consensus from researchintegrity experts and research integrity policyexperts on which topics are important for organizationalefforts to foster research integrity,

  • 05:16

    JOERI TIJDINK [continued]: and research performing, and research funding institutions.And therefore, we use the Delphi consensus study.

  • 05:31

    KRISHMA LABIB: So you might be wondering,what is a Delphi study?There are four key features of a Delphi study.First, Delphi studies recruit a panelof experts who remain anonymous to mostof the researchers and all the other peoplein that panel of experts.

  • 05:51

    KRISHMA LABIB [continued]: In a Delphi study, you use multiple questionnairesover time, and each of these questionnairesis called a round, a survey round.And you send these to your panel of experts.What happens inbetween rounds is that you provide your expertswith feedback of the results of the previous roundwhen you send out the next survey round.

  • 06:12

    KRISHMA LABIB [continued]: And the idea there is that as you do this,you can come to a convergence of opinion,because the goal of the Delphi studyis to explore experts' views on a particular topicwith the intention to inform policy or practice.And most Delphi studies, although not all,

  • 06:33

    KRISHMA LABIB [continued]: aim to achieve consensus among experts' opinions.Beyond these four criteria, thereare no further requirements about how youshould conduct a Delphi study.And there are many different variations of the Delphiout there.In our Delphi study, we specificallyuse the modified Delphi procedure.

  • 06:54

    KRISHMA LABIB [continued]: In this kind of procedure, you firststart with a review of existing knowledge about the topicbefore you conduct your first survey round.What's important to mention also is that our Delphiused a qualitative approach.So although we gathered both quantitative and qualitativedata in our study, we strongly relied on the qualitative data

  • 07:18

    KRISHMA LABIB [continued]: to interpret the results of our quantitative data.So we use a qualitative Delphi approach.

  • 07:34

    JOERI TIJDINK: I would like to highlight what are actuallyexperts in research integrity.So we defined our experts that wewanted to include in our study as policy makers or researchleaders with a strong interest and/or experiencein research integrity.And they could work in Research Performing Organizations, alsoknown as RPOs or Research Funding Organizations,

  • 07:56

    JOERI TIJDINK [continued]: also known as RFOs.Of course, they have to come from different countriesto have a diverse perspective, different genders,and disciplines.And we explicitly did not narrow our inclusion criteriato only look for research integrity expertsbecause you then probably get too much of a diverging view.

  • 08:17

    JOERI TIJDINK [continued]: We wanted to include all perspectives from policy makersand research leaders from different countries.Of course, they highlighted already in the studythat they had some experience with research integrity.So we used a purposive recruitment strategyto find and invite experts globally.

  • 08:39

    JOERI TIJDINK [continued]: And we approached people from our network followedby snowballing, as well as looking upadditional participants of at least oneResearch Performing Organization and one Research FundingOrganization from each country in Europebecause it is a European project,and we wanted to include mostly perspectives from Europe.In total, we could include 68 Research Performing

  • 08:59

    JOERI TIJDINK [continued]: Organization experts and RPOs, and 52 experts from RFO.And they responded and agreed to take part in at least oneof our Delphi study rounds.So how does that look?So how do these rounds work?

  • 09:21

    JOERI TIJDINK [continued]: So in total, we conducted three different rounds in this surveyto obtain consensus on which topics to includein the research integrity plan.And this can be done with a survey program.In this case, we use Qualtrics.Since we expected that the topics might be

  • 09:42

    JOERI TIJDINK [continued]: different from RPOs and RFOs.We split the study into two parallel parts.So parallel part one was focusing our RPOs,and part two was focusing on RFOs.And before the first round of the survey,we conducted the document analysis of existing researchintegrity guidelines, codes of conduct,and other types of guidance documents for RPOs and RFOs

  • 10:04

    JOERI TIJDINK [continued]: to create a preliminary list of RI topicsthat we could present, that we alreadyadded a description of this specific topic.So next, we presented this list of the expertsin the first round.And we asked the experts to rate the importance of each topicas well as to provide a rationale for their responses.And the ratings could be done in a 1 to 5

  • 10:27

    JOERI TIJDINK [continued]: Likert scale on the topic importancewith one indicating that the topic was not important.And of course, five was absolutely essential.Experts could also propose new topicsto ask because we wanted to have an open survey that people feltinvited to add an additional expert because we couldhave missed one while we created the first preliminary list.

  • 10:51

    JOERI TIJDINK [continued]: After analyzing the responses of the first survey,we saw that we already obtained consensusfor most of the topics.And to explore the relative importance of the topics,we asked experts in the second roundto rank the topic's important in relation to each other.And finally, in the third round, we

  • 11:11

    JOERI TIJDINK [continued]: used to explore experts' rationales for the ranking.In between each ranking, we provided expertswith a summary of the results of the previous roundand provided them with the opportunityto add any additional comments or remarks.This is important to know what steps wetook that resulted in the end product, which Krishma

  • 11:34

    JOERI TIJDINK [continued]: is going to present shortly.But first, we go to the data analysis.

  • 11:46

    KRISHMA LABIB: We analyzed our different types of datain various ways.So first of all, regarding the quantitative data.We had data on the ratings of each topic.And for this, we looked at the percentageof experts who rated the topic as 4 to 5 on the Likert scale,indicating that the topic is very important or absolutely

  • 12:07

    KRISHMA LABIB [continued]: essential to be included in the research integrityplan or research performing or funding organizations.We set the consensus threshold level at 67% agreement,meaning that when 67% of experts rated the topic as 4 to 5on the Likert scale, we assumed that therewas consensus on the importance of that topic.

  • 12:29

    KRISHMA LABIB [continued]: Now, we also obtained a ranking datafrom the second round of the survey.And for this, per expert, we assignedtopics that were highly ranked, more pointsper expert than those who were lower ranked.So for instance, if an expert ranked the topic Educationand Training in Research Integrity really high,

  • 12:51

    KRISHMA LABIB [continued]: highest, and then just after that they rankedthe topic Responsible Supervision and Mentoringsecond highest, we assigned the first topic, Educationand Training, six points.And we assigned the next topic, Responsible Supervisionand Mentoring, five points for that expert.And then we summed the scores per topic across experts.

  • 13:14

    KRISHMA LABIB [continued]: And based on this, we were able to order the topicsby priority.So that's regarding the quantitative data.Now, when we come to the qualitative data,we also used two different approaches to analyze it.So first, per topic, we explored experts' argumentsfor and against the importance of each topic.

  • 13:37

    KRISHMA LABIB [continued]: Additionally, we conducted a thematic analysisin which we clustered the data into related themesfor all the data across topics to obtaina more general impression of experts' viewson research integrity plans.

  • 13:60

    KRISHMA LABIB [continued]: We reach consensus on the importanceof including 12 research integritytopics in the research integrity plans of Research PerformingOrganizations.And you can see these in the table here.You can also see the order in which they wereprioritized by the experts.You can see that the three topics that were mosthighly ranked included research integrity education

  • 14:21

    KRISHMA LABIB [continued]: and training, responsible supervision and mentoring,but also dealing with breaches of research integrity.So education and supervision were ranked highbased on the argument that currently, the biggestproblems with research integrity are notabout downright misconduct, but rather,relate to questionable research practices

  • 14:41

    KRISHMA LABIB [continued]: which are often done unintentionallydue to poor awareness.So you need knowledge and guidance on research integrity,and these two topics can help there.However, the experts also mentionedthat it's crucial that when cases of misconductdo occur, even if they might not be so common,they should be adequately dealt with.

  • 15:04

    KRISHMA LABIB [continued]: It's interesting, also, to point that these topics alignto a large extent with practices identifiedin the European Code of Conduct for Research Integrity.

  • 15:12

    JOERI TIJDINK: Thank you, Krishma.That was helpful.Now, the results of the RFO list of topics.We reached consensus on the importance of including11 research integrity topics in the research integrityplans for Research Funding Organizations.Of course, some of these topics overlap with those of RPOs,but there are also many differences in the lists.

  • 15:34

    JOERI TIJDINK [continued]: The most highly ranked topic for the RFOswere dealing with breaches of RI within the funding body,conflicts of interest, and funders' expectationsof research performing organizations.The differences between the RPO and the RFO listhighlights that RPOs and RFOs have different responsibilitiesregarding research integrity.

  • 15:55

    JOERI TIJDINK [continued]: And since researchers are directly dependent on RPOsfor support, infrastructure, and training,RPOs are mainly responsible for these responsibilities,while the RFO can set requirementsboth on individual researches as well as on the RPO.And this can be a powerful way to incentivize and reward

  • 16:16

    JOERI TIJDINK [continued]: research integrity, so funders have an important roleto play in fostering research integrity.

  • 16:23

    KRISHMA LABIB: Our qualitative databrought us some additional insights about experts' viewsand research integrity plans.First of all, our experts advocatedfor using preventative and positive approach to researchintegrity rather than an approachfocused on misconduct, although they alsostressed that research integrity breaches should also

  • 16:47

    KRISHMA LABIB [continued]: be appropriately dealt with when they occur.They also talked about how the topics in the Delphiare very much interrelated.So for instance, many of them mentioned that research culturewill likely be influenced by all the other topics, but alsothe specific measures used to improve the research culture.

  • 17:09

    KRISHMA LABIB [continued]: So for instance, implementing policiesrelated to fair promotion of researchers.That will also have a direct effect on the other topicsin the list.They also mentioned that Research Performingand Research Funding Organizationshave different responsibilities regarding research integrity.Researchers work at Research Performing Organizations.

  • 17:31

    KRISHMA LABIB [continued]: So it's the Research Performing Organizations thatare responsible for supporting researchers by providing themwith things like infrastructure, and support,and also, training.And funding organizations, instead,can rely on the Research Performing Organizationson these things.And what they can do is to set requirementsboth on researchers and on Research Performing

  • 17:53

    KRISHMA LABIB [continued]: Organizations.Our experts also stress that, although standardizationof research integrity practices and policiesacross various institutions in Europe is helpful,there are important differences between institutions,countries, and disciplines, and research in Europe.

  • 18:14

    KRISHMA LABIB [continued]: So it's really important that research integrityplans take into account these differences in cultureand legislation, but also, research practices.Finally, and very importantly, our expertswarned us that it's important to keep in mind that research

  • 18:34

    KRISHMA LABIB [continued]: integrity policies have the risk of adding too much oversight,increasing bureaucracy at the organizational level.But also, for researchers, and thatthis is really undesirable.So they cautioned us to keep this in mindwhen developing guidelines on the research integrity topicsmentioned in our list.

  • 19:01

    JOERI TIJDINK: Again, we have identifiedwhich topics to include for research integrity promotionplans for both the RPOs and the RFOs.A preventative approach to RI alongwith clear policy for dealing with misconduct is necessary.The other thing that I take from this studyis that research integrity is complex.

  • 19:23

    JOERI TIJDINK [continued]: It's broad, and it can be complicated.Not only are there different stakeholders, each playinga different role, but there are numerous topicsthat should be addressed to adequately addressresearch integrity, many of which are interrelated.And to optimally foster research integrityRPOs and RFOs should address each of these topics

  • 19:43

    JOERI TIJDINK [continued]: in our list to improve the institutional and systemof science factors that influence research integrity.And doing this without increasingunnecessary oversight may be a challengebecause researchers do not want more bureaucracy.That's more administration.They really want to do research.Due to institutional country and disciplinary difference,

  • 20:06

    JOERI TIJDINK [continued]: each institution might need to address the topics slightlydifferently.But overall, I think this list isquite comprehensive and useful for institutions,how to rate and to rank which topicsthey want to address in their institutional policy.

  • 20:24

    KRISHMA LABIB: If you would like further informationabout this study, you can check out our pre-print,which is provided on the slide.And we also have a study that was published in Nature, whichhighlights how our results have beenused in the SOPs4RI Project.And you can also find the reference on the slide.

  • 20:44

    KRISHMA LABIB [continued]: If you're interested in the exact methods we used,you can find our study protocol on the Open Science Framework.


Joeri Tijdink, PhD, Assistant Professor, and Krishma Labib, PhD Student, at Amsterdam University Medical Center, discuss the Delphi consensus study on research integrity, including the rationale, its aims, experts involved, data collection, and research findings.


Joeri Tijdink
Krishma Labib

Segment Info


Segment Num: 1


Segment Start Time:

Segment End Time:


Things Discussed

Organizations Discussed:

Events Discussed:

Places Discussed:

Persons Discussed:



    Further Reading

    Labib, K., Scepanovic, R., Bouter, L., Widdershoven, G., & Et al., (2020).Important topics for fostering research integrity by research performing and research funding organizations–A Delphi consensus study. Open Science Framework Preprints,

    Further Reading

    Mejlgaard, N., Bouter, L.M., Gaskell, G., Kavouras, P., & Et al., (2020).Research integrity: Nine ways to move from talk to walk. Nature 586, 358-360.

    Further Reading

    Marusic, A., Mejlgaard, N., Sørensen, M.P., & Et al., (2020).Systematic review of practices and research cultures (WP3). Open Science Framework,

Methods Map

Research ethics

The field of moral philosophy dealing with the standards by which behaviour should be regulated within research.
Research ethics
Important Topics for Research Integrity: Delphi Consensus Study

Joeri Tijdink, PhD, Assistant Professor, and Krishma Labib, PhD Student, at Amsterdam University Medical Center, discuss the Delphi consensus study on research integrity, including the rationale, its aims, experts involved, data collection, and research findings.

Copy and paste the following HTML into your website