Skip to main content
Search form
  • 00:17

    ANA MARUSIC: Hello.My name is Ana Marusic.I'm Medical Doctor and Professor of Anatomyand Research Methodology at the University of Split in Croatia.I'm the work package leader of the work packagethree in the SOPs4RI project.And here is my colleague Rea.

  • 00:34

    REA ROJE: Hi, my name is Rea Roje,I hold a Master's degree in law, and I've alsobeen working on the project, SOPs4RI,mostly in work package three.On literature review, expert interviews and Delphi study.

  • 00:47

    ANA MARUSIC: So, our work packageis important because it reforms the subsequent work packagesin our project on the state of the art knowledgefrom the existing literature and stakeholder experiencewith research integrity.As well as on identified gaps thatshould be addressed in the further workof the SOPs4RI project.

  • 01:07

    REA ROJE: Yes.So the work package three in SOPs4RI projectanswers the question, where is the evidencefor what we are going to do?which means, which is for creatingthe toolbox of standard operating proceduresand guidelines addressing important research integritytopics.And to answer this question, we firstconducted two scoping reviews.

  • 01:35

    REA ROJE [continued]: So why did we choose scoping reviews?The purpose of scoping reviews, in general,is to provide an overview of available evidenceon a certain topic, and to identify and analyzeknowledge gaps.So they are different from, for example, systematic reviews,because they have a broader scope.

  • 01:53

    ANA MARUSIC: The scoping reviews do notfocus on the single precise questions.For example, answering whether a certain interventionor treatment in practice is effective.But rather they aim to map the evidence relatedto a certain topic, to discover what already exists,what knowledge we have, and what knowledge is lacking?Where we have knowledge gaps in certain areas.

  • 02:23

    ANA MARUSIC [continued]: So what steps did we take to plan and designthe literature reviews?We started the work with planning and writingthe protocols for the two scoping reviews.A scoping review on the best practicesfor research integrity promotion in researchperforming organizations, and research funding organizations.And the scoping review on factorsthat influence the implementation of research

  • 02:45

    ANA MARUSIC [continued]: integrity practices in these organizations.In developing the protocols, we follow the guidancefor scoping reviews, these are methodological guidancefrom Joanna Briggs Institute, whichspecializes in performing systematic reviews and evidencesynthesis.

  • 03:01

    REA ROJE: Yes.And we had to define precisely; whatevidence we will search, where we will search this evidence,how we are going to do the searchand extract the information, and who will perform the searchand selection of the evidence?So first we define the objectivesof our scoping reviews that will guide us in our work.And then, we define the concept, context and information

  • 03:23

    REA ROJE [continued]: sources.So we decided that we will searchthe four bibliographic databases and eightsources of gray literature.The next step in this process was to define the eligibilitycriteria.So this means to select which documents obtainedthrough the search are relevant for our study aimand should be included or perhaps excluded

  • 03:46

    REA ROJE [continued]: from the review.We also define which information, from the includeddocuments, will be extracted and used in the final analysis.And also we define the roles of researchersworking on the scoping reviews.So, who will perform the search, who will analyze the documentsand extract information?

  • 04:03

    ANA MARUSIC: Well the important partof scoping review design is developing a search strategy.A complete search strategy for at least one databaseshould be included in a research protocol.In collaboration with the librarian specialistsand experts in database search in systematic reviews searchmethodology, we developed a search strategyfor bibliographical databases.

  • 04:24

    ANA MARUSIC [continued]: First we develop a skeleton of basic terms and conceptsthat we aim to explore.For example, research integrity, misconduct, fraud, ethics,research ethics.And then with the librarian, we developed this furtherinto a very comprehensive research strategy.So our search strategy was made to be highly sensitive.

  • 04:45

    ANA MARUSIC [continued]: This means that we included a broad approach to the field,because we wanted to identify as many relevant documents aspossible.So it wasn't a specific--because the field of research integrity is very wideand there are a lot of terms that cannot be included there.So we want to have a sensitive search, but not a veryspecific search.

  • 05:05

    ANA MARUSIC [continued]: And for the great literature sources,the search strategy was more simple.It's simpler and it consisted of a few basic terms thatwere relevant for our research.Like research integrity and research ethics.And you see here the example of a search strategy for Scopusbibliographical database.So you see all of these terms thatare connected with what we call Boolean operators, that

  • 05:26

    ANA MARUSIC [continued]: are telling them when we use a Booleanoperator 'or' that means include both of the terms.And when we use the Boolean operator 'and'it means that we have to combine these words sothat the document we are searching forhas to have both of these terms in itto be included in our further analysis.

  • 05:45

    REA ROJE: We also define, in the protocol,that for the reporting of our findings.We will use the PRISMA checklist for scoping reviews.So the PRISMA checklist is a great toolthat helps us report all the relevant data,and ensures that we did not miss somethingthat could help other researchers evaluate and buildupon our work.We also registered our scoping review protocols together

  • 06:08

    REA ROJE [continued]: with other work conducted in the SOPs4RI work packagethree at the Open Science Framework.

  • 06:22

    ANA MARUSIC: How did we collect data?Now, this is the next step in our work.So based on the developed strategythat we had to search bibliographical databases,we performed this search in several bibliographicaldatabases, such as Medline for medicine,Scopus for general research fields, Web of Science too,and PsychINFO.

  • 06:42

    ANA MARUSIC [continued]: But then, we limited our search to the documentsdating from 1990s onward, because thatwas the time when the term of research integritystarted to appear in literature.We also presumed that most of the documents and researchthat we would find would be after this date, and anythingbefore that would be obsolete and replaced

  • 07:04

    ANA MARUSIC [continued]: with new findings and more contemporary practices.

  • 07:07

    REA ROJE: And after we performed the search,we exported the results in the EndNote tool, whichhelped us manage to retrieve documents and the tagduplications easily.So the search retrieved almost 33,000 documents,and after removing duplicates, wewere left with almost 27,000 documents.And the next step in this process

  • 07:28

    REA ROJE [continued]: included two researchers independentlyscreening the titles and abstracts of these 27,000documents.And before this, we actually did a pilot screeningof the 100 records to ensure that all researchers understoodthe eligibility criteria properly.The screening of all written documentswas a very long process.

  • 07:49

    REA ROJE [continued]: After which two researchers discussed and comparetheir work and documents.They included or excluded.And the goal of this discussion wasto decide which documents will be included in our full textanalysis.And in some cases researchers could not agree,and here we also included the third researcherwho helped us make the decision process

  • 08:12

    REA ROJE [continued]: and bring the final decision.So after the full text reading weobtained the documents that will be included in the dataextraction and final analysis.And for the documents that were excluded in this test,we also provided a rationale for the exclusion.After this was finished, we searched the referencesof the documents that were included in the final analysis

  • 08:34

    REA ROJE [continued]: to identify the additional ones that we may possiblymiss during our search.And this was also a good check stepbecause there is always a chance that we missed some documentsin the initial screening.

  • 08:48

    ANA MARUSIC: So after we have hadthe final list of documents, after all this extensivesearching through the retrieved documents,we went to the next step in the process,which is data extraction.So we defined a preliminary list of items to be extracted.This was already defined in the protocol,

  • 09:08

    ANA MARUSIC [continued]: but we updated it to the new itemsthat we discovered when we saw the relevant literatureduring the data extraction procedure.

  • 09:17

    REA ROJE: And for the great literature search,we mostly included the search of different web pages containingdocuments that were related to research integrity.So for example, we searched the web pages of the United Statesoffice of Research Integrity, and the European Networkof Research Integrity offices.

  • 09:35

    ANA MARUSIC: The process from performing the searchto obtain documents for the final analysis,it was also presented in the format of a PRISMA flowdiagram.PRISMA is, as we already said, a reporting guideline whichensures that the minimal data about our studyis reported in a way so that readerscan assess the validity of our results,

  • 09:58

    ANA MARUSIC [continued]: and that they can include in further evidence synthesis.So this was a great-- this flow chartis a great way to visually represent the whole processtogether with the numbers of identified,retrieved, screened, and analyzed documents.

  • 10:21

    ANA MARUSIC [continued]: Then, the analysis of data came.So, after we extracted the data, weanalyzed them to summarize the findings.In the scoping review and practicesfor research integrity and promotion,we analyzed the documents based on different criteria.One of them was their origin, geographical,

  • 10:42

    ANA MARUSIC [continued]: disciplinary field from which theycome, and also, organizational.So whether documents and research integrity practicesthat were described in this work were more relatedto research performing or to researchfunding the organizations.

  • 10:57

    REA ROJE: We also analyzed the typeof research integrity practices addressed in these documents.For example, guidelines, codes, checklists, standard operatingprocedures and target groups to whichthese practices were directed.So these included for example, researchers, policy makers,funders or research integrity officers.The important part of our analysiswas also to map these documents based on the different research

  • 11:20

    REA ROJE [continued]: processes.So research planning, research conducting, disseminationand evaluation.And we also map the documents based on the research integritytopics related to these processes that were addressedin the documents.This actually enabled us to see for example,which research integrity related topics need more guidance

  • 11:41

    REA ROJE [continued]: documents.And we can also work on this in a --further the work of our project.So, what documents we can develop in our further work?

  • 11:51

    ANA MARUSIC: Well, in the second scoping review--so we talked about the first scoping review so far,and in the second one, we focusedon identifying factors that influencethe implementation of research integrity practices.And we analyzed some general characteristicsof these documents.This again, including the types of

  • 12:12

    ANA MARUSIC [continued]: collected documents, whether theywere journal articles, books and other types of documents,and also their origin.In this review, the most importantwas to map the different factors thatmay have a positive or negative impact on how we implementresearch integrity practices.We analyze the documents based on their relationto the level of the individual or researcher,

  • 12:33

    ANA MARUSIC [continued]: and the level of the organization.And also, at the general level of the system of science.And what did we find?And why this is important for the SOPs4RI project?In this scoping review, related to research integrity

  • 12:54

    ANA MARUSIC [continued]: practices, we found that most of the documentswere in the form of guidelines.And that they also varied very much in scope and detailsaddressing research integrity issues.Some guidelines were more general,and they contained aspirational principlesthat researchers should strive to.While others were more oriented to providea step-by-step guidance on different research integrity

  • 13:18

    ANA MARUSIC [continued]: issues.For example, how to manage data?How to handle allegations of research misconduct?

  • 13:23

    REA ROJE: Most of the documents wefound were from the United States of America,and also related to research performing organizations,while we identified a lack of documents relatedto funding organizations.Also the documents were mostly related to biomedical sciencesand research in general.We used the category 'research in general'

  • 13:44

    REA ROJE [continued]: for documents that were not relatedto any specific disciplinary field,but rather they addressed research integrity issuesin general and could be applicable universallyto different disciplinary fields.

  • 13:56

    ANA MARUSIC: And in the second scopingreview we found most of the available literature thatrelated to the factors influencingthe implementation of research integrity practices, thatwere related to research performing organizations.Showing, again, a lack of documents and evidencethat we have related to research funders.Also, most of the documents address

  • 14:17

    ANA MARUSIC [continued]: the factors that have a negative impact on research integrity.

  • 14:20

    REA ROJE: Yes, we found for example, high moral integrity,adequate supervision and mentorship,or involvement in research integrity educationmay have a positive impact on the implementation of researchintegrity practices.While on the other hand, some personality traitslike, for example, laziness or egoism,situational factors like financial problems

  • 14:43

    REA ROJE [continued]: and different pressure that exist in the system of science,for example, pressure to publish or advance careermay have a negative impact.So for the organization level, wefound that the development of research integritypolicies and educational courses,as well as encouragement of open communication and discussionmay have a positive impact on fostering research integrity.

  • 15:05

    REA ROJE [continued]: While the lack of this is related to the negative impact.

  • 15:09

    ANA MARUSIC: And finally, for the system of science level,we identify the different trends and pressuresrelated to incentives provided by research organizationsoften have a negative impact, and maybe related to researchers that are involved in misconduct.Such as valuing quantity over quality of researchor awarding only researchers who published in high impact factor

  • 15:31

    ANA MARUSIC [continued]: journals.

  • 15:32

    REA ROJE: On the other hand, the workof funders and scientific publishers and journalson research integrity promotion was oftenmentioned as important for research integrityimplementation.

  • 15:50

    ANA MARUSIC: So we come now to challengesthat we face during our research and howwe overcame these challenges.One important challenge that we faced during our researchwas a very large number of documentsthat we retrieved through the literature search.Going through the titles and abstractsof more than 26,000 documents was a very demanding and time

  • 16:11

    ANA MARUSIC [continued]: consuming task.However, by having a broad approach in our search,we ensure that we captured as many as possible documents thatwere relevant to the aims of our research.Using a reference manager tool, thiswas very helpful in this stage because we had at one placeall the titles, key words and abstracts.So it was easier to go through them

  • 16:32

    ANA MARUSIC [continued]: and eliminate those that did not meet the criteria for inclusionin our evidence synthesis.

  • 16:39

    REA ROJE: And perhaps another challengewas that some documents did not have the abstracts availablein the reference manager, so we hadto search for them separately.And also, some documents did not have abstracts at all.So we had to keep them for the full text analysisand decide on their inclusion upon reading the full text.

  • 16:59

    REA ROJE [continued]: Also the last issue was the availability of the documents.So some full text documents and books were very hard to find.And in these cases we had to ask our librarian to help us.And in some cases, we even contacted the researchersdirectly, if they would be willing to send ustheir manuscripts.And in almost all cases, we did receive the positive answers

  • 17:22

    REA ROJE [continued]: and the manuscript.So although in scoping reviews the critical appraisalof the quality of evidence is notnecessary to perform as in the systematic reviews,we think that this would be a valuable addition

  • 17:43

    REA ROJE [continued]: to some scoping reviews.For example, in our scoping review and factorsinfluencing the implementation of research integritypractices, it would actually be interesting to seeinterventions described in these documents.And conduct an appraisal of the quality of evidenceto conclude about the effectivenessof certain factors on, for example,making positive changes in the research integrity

  • 18:05

    REA ROJE [continued]: implementation.

  • 18:06

    ANA MARUSIC: So in the end, the results from the work packagethree, which besides literature reviews,also included explorative interviews study, Delphiconsultation and the development of a multi-level modelof research culture, were there to inform the next workpackages for the selection of topics that would be includedin the toolbox box of the research integrity guidance

  • 18:29

    ANA MARUSIC [continued]: that we are developing in the project.And upon which the first version of these operating proceduresand guidance was made.Finally, the results also helped usto identify issues that are related

  • 18:50

    ANA MARUSIC [continued]: to the implementation of research integrity practicesthat can be addressed in future on different levels.And to help incentivize positive changesand avoid involvement in research misconduct.

  • 19:02

    REA ROJE: We hope this will help you in conducting your scopingreviews.Thanks for watching.

  • 19:08

    ANA MARUSIC: Thanks, bye.


Dr. Ana Marusic, MD, PhD, and Rea Roje discuss literature reviews and research integrity, including planning and design, data collection and analysis, research findings and importance, and challenges and lessons learned.

Video Info

Publication Info

SAGE Publications, Ltd.
Publication Year:
SAGE Research Methods Video: Research Ethics and Integrity
Publication Place:
United Kingdom
SAGE Original Production Type:
SAGE Case Studies
Copyright Statement:
©SAGE Publications Ltd., 2022


Ana Marusic
Rea Roje

Segment Info


Segment Num: 1


Segment Start Time:

Segment End Time:


Things Discussed

Organizations Discussed:

Events Discussed:

Places Discussed:

Persons Discussed:



    Further Reading

    Šcepanovic, R., Buljan, I., Marušic, A., & Et al., (2019).Protocol for the literature review, the expert interviews and the Delphi procedure. SOPs4RI,

    Further Reading

    Gaskell, G., Šcepanovic, R., Buljan, I., & Et al., (2019).Standard Operating Procedures for Research Integrity. SOPs4RI,

    Further Reading

    Šcepanovic, R., Tomic, V., Buljan, I., & Marušic, A., (2019). Report on the results of explorative interviews. SOPs4RI,

Methods Map

Research ethics

The field of moral philosophy dealing with the standards by which behaviour should be regulated within research.
Research ethics
Standard Operating Procedures for Research Integrity: Literature Reviews

Dr. Ana Marusic, MD, PhD, and Rea Roje discuss literature reviews and research integrity, including planning and design, data collection and analysis, research findings and importance, and challenges and lessons learned.

Copy and paste the following HTML into your website