Introduction to Health Technology Assessment: Clinical Effectiveness

[MUSIC PLAYING][RESEARCH METHODS tutorial][Introduction to Health Technology Assessment--Clinical Effectiveness]

GJ MELENDEZ-TORRES: Hi, everyone.Welcome back.This is Introduction to Health Technology Assessment,focusing on clinical effectiveness.My name is GJ Melendez-Torres.[G.J. Melendez Torres, Professor of Clinical SocialEpidemiology, University of Exeter]And I'm professor of clinical and social epidemiologyat the University of Exeter.In this video, we're going to talkabout the clinical effectiveness aspect of health technologyassessment.

GJ MELENDEZ-TORRES [continued]: When I say clinical effectiveness, what I mean is,does a new health technology work comparedto the relevant comparators?Put otherwise, in respect of whichoutcomes that are important to whom,does this particular treatment strategy work most effectively?We're going to talk about five different topics in this video.The first thing is we're going to talk

GJ MELENDEZ-TORRES [continued]: about the goal of clinical effectiveness evidence,and we're going to talk a little bit about whatit is that we're trying to establish or showwhen we seek to understand the clinical effectivenessof a new health technology.The second thing we're going to dois just talk briefly about the different formsof clinical evidence that exist.We'll talk about randomized trials.We'll talk about so-called real-world evidence,

GJ MELENDEZ-TORRES [continued]: and then we'll also talk about meta-analyses and networkmeta-analyses as a strategy for combining different estimatesof clinical effectiveness into an overall morerobust, better-powered, more precise estimate ofwhether or not a drug works comparedto its comparators for specific outcomes.The goal of clinical effectiveness evidence,

GJ MELENDEZ-TORRES [continued]: as I said just now, is to figure out, does this work?Does it work?Is it effective?Now, it's important to understandthat inherent in that statement is a comparison.What I mean by that is that clinical effectiveness isalways a comparative question.If we're asking if a new drug works,we should be asking, compared to what?

GJ MELENDEZ-TORRES [continued]: Now if you specify your decision problem correctly,it should not only tell you the answer to that question,compared to what?It should also tell you with respect to which outcomes.So again, inherent in that question-- does it work--is not only a comparative statement but alsoan assertion about with respect to which outcomes.

GJ MELENDEZ-TORRES [continued]: The goal of clinical effectiveness evidenceshould be to present an unbiased and generalizable estimateof the clinical effectiveness of a new drugin respect of its comparators and with respectto specific outcomes.When I say unbiased, what I mean isas close to the truth of what that clinical difference isbetween two groups, one of whom got the new drug and one

GJ MELENDEZ-TORRES [continued]: of whom got the comparison.When I say generalizable, what I meanis that we can take that estimate of effectivenessfrom the clinical effectiveness evidenceand transport it to our local contextand expect it to see a similar pattern of effects.So here's an example from multiple sclerosis.You'll recall that in a previous video,

GJ MELENDEZ-TORRES [continued]: I spoke about how relapsing-remitting multiplesclerosis is the most common form of MSand how this disease is characterizedboth by the progression of neurological disabilityas well as the rate of relapses, which are episodesof neurological inflammation.Now in first-line treatments for multiple sclerosis,

GJ MELENDEZ-TORRES [continued]: we're principally interested in slowing downthe rate of disability progression,as well as in slowing down the rate of relapses.So what does that mean?With respect to the clinical effectiveness evidence, whatwe're trying to do is think about,does a beta interferon compared to other beta interferonsor glatiramer acetate, all of which

GJ MELENDEZ-TORRES [continued]: are first-line treatments for MS,slow the rate of disability progressionand slow the rate of relapses?The goal of clinical effectiveness evidence,preferably from randomized trials,is to present an unbiased and generalizable estimate,generalizable to our local context,about the ability of these drugs,

GJ MELENDEZ-TORRES [continued]: compared it to each other and compared to no treatment,to slow disability progression and to slowthe rate of relapses.[Identifying Analyzing ClinicalEffectiveness Evidence]Now the first step in analyzing clinical effectiveness evidenceis to identify it.

GJ MELENDEZ-TORRES [continued]: In HTA, when we identify clinical effectivenessevidence, we generally do that by way of a systematic review.A systematic review is the systematic, auditable,and reproducible search selection,appraisal, and synthesis of all relevant evidence with respectto your decision problem.In this particular case, what we did

GJ MELENDEZ-TORRES [continued]: was we undertook that systematic review, again, that systematicsearch, retrieval, selection, appraisal, and synthesis,of all of the randomized trials that we could find thatcompared beta interferons and glatiramer acetateagainst each other and against placeboin this population of early-stage patients

GJ MELENDEZ-TORRES [continued]: with relapsing-remitting multiple sclerosis.We looked for all of these trials.We put them together.We appraised their quality with respect to the riskthat they posed of presenting a biased estimateof effectiveness, and then we combinedthose estimates statistically using a meta-analysis.Now as I've said, the basic unit of evidence

GJ MELENDEZ-TORRES [continued]: that we use in clinical effectivenessis a randomized trial.Now what is a randomized trial?A randomized trial is a clinical trial or a clinical studywhere you randomize some people to geta treatment and other people to get a different treatment.That's it.That's all that a randomized trial is.

GJ MELENDEZ-TORRES [continued]: You are randomizing, as in selecting randomly,people to either get in treatmentor not get a treatment.Now randomized trials are the central unit of evidencethat we use in health technology assessmentbecause they are reliably least biasedway of estimating the clinical effectiveness of a drug.

GJ MELENDEZ-TORRES [continued]: Why is that?It's because the use of randomization does two things.The first and the most important thingis that it reduces selection biasin estimating the effectiveness of a given treatment.When I say selection bias, what I meanis that people selecting of themselvesor being selected to get a treatment

GJ MELENDEZ-TORRES [continued]: or get another treatment could makeit more difficult to reliably estimatethe clinical effectiveness with respect to an outcome.So the first thing that randomized trials dois reduce the risk of selection bias.The second thing that randomized trials dois that on average, generate balance between the twogroups in the trial, both on the variables that we can measure

GJ MELENDEZ-TORRES [continued]: but also in the variables that we can't measure.This is important because what itmeans is that randomization gives us a way of generatingbalanced groups without needing to worry about havingto construct that balance, having to create that balance.The variety of tools that we can use in randomizationto improve the balance, but basic randomization

GJ MELENDEZ-TORRES [continued]: does two things.It reduces and hopefully removes that risk of selection biasand then also creates balanced groups.Together, what those mean is that a randomized trialis the most direct way to estimatethe causal impact of a new drug on the outcomes thatare relevant for understanding the clinical effectiveness

GJ MELENDEZ-TORRES [continued]: of that particular drug against a comparator.Now to go back to our example about multiple sclerosis,we found five trials that compared glatiramer acetateagainst placebo.All five of those trials captured reductionsin relapse rate of glatiramer acetate compared to placebo.

GJ MELENDEZ-TORRES [continued]: For the three trials, also captured the rateof disability progression in glatiramer acetatecompared to placebo.We also found in additional two trialsthat compared one form of interferon against another,having that head-to-head evidence comparingone active drug against another is important and useful

GJ MELENDEZ-TORRES [continued]: because it can strengthen our inferenceabout the comparative effectivenessof different drugs against each other.In the majority of cases, if you see a clinical trial,we're comparing a new drug against a placebo.Comparing one drug against another active drug, as I said,strengthens our ability to concludethat one drug is more effective than another instead of having

GJ MELENDEZ-TORRES [continued]: to indirectly compare them via their improvementsover placebo.Another important category of clinical effectiveness evidencethat we often see is called real-world evidence.The term "real-world evidence" is a bit fuzzy,and we most often use real-world evidence when we can't doa randomized trial in real life.

GJ MELENDEZ-TORRES [continued]: There are a number of reasons why that might be.It may just be too difficult. We may nothave the patience to do it.In some cases, the condition mightbe too rare to recruit the number of patientsthat we need to run the trial.In other cases, it may be the casethat we just can't run a randomized trial because

GJ MELENDEZ-TORRES [continued]: of major logistical issues.Something that real-world evidence doesis take clinical effectiveness evidence from patient recordsfrom existing databases from health system recordsand use that to estimate the relative or comparativeeffectiveness of one drug or treatmentstrategy against a comparator.

GJ MELENDEZ-TORRES [continued]: One of the challenges with real-world evidenceis that it doesn't have the same benefitsthat accrue to randomized trials,namely, reducing, if not eliminating,the risk of selection bias and generating balanced groupsboth on the variables we can observe,as well as the variables that we may not be able to observe.However, one benefit of real-world evidence

GJ MELENDEZ-TORRES [continued]: is that it comes from the real world as opposedto some of the more tightly controlled clinical settingswithin which clinical trials are generally done.Something the real-world evidence will often dois try to create groups that got a treatment and groups thatgot another treatment that are balanced using either matching

GJ MELENDEZ-TORRES [continued]: by matching one patient who got one treatment to onepatient who got another treatment on a rangeof relevant characteristics or weighting,reweighting a treatment group to bevery similar to the characteristicsof another group.Matching and waiting are both very useful methods.But the problem with matching and waitingis that we still can't completely

GJ MELENDEZ-TORRES [continued]: account for the influence of variablesthat we may not be able to observe.The overriding criterion that we shouldask if we look at real-world evidenceis, how closely does this real-world evidence approximatein terms of design, in terms of patients,in terms of relevance?How closely does this real-world evidence

GJ MELENDEZ-TORRES [continued]: approximate the randomized trial that we would havedone if we had been able to.This is also called the target trial.Whenever we read real-world evidence,we should ask ourselves, how closely does this evidenceapproximate the target trial?So to provide you an example of real-world evidencefrom our work on multiple sclerosis,

GJ MELENDEZ-TORRES [continued]: one of the evidence sources we looked atcompared a natural history cohort of peoplewho hadn't received any active treatmentfor their multiple sclerosis against the cohort thathad received those beta interferons and glatirameracetate as part of normal clinical practice.Now one of the benefits of this real-world evidence

GJ MELENDEZ-TORRES [continued]: was that it was timely and recent,which is to say it was collected fairly recently in the healthsystem context that we were interested in evaluating.This was an important benefit against someof the clinical trials that we looked at for beta interferonsand glatiramer.Some of which were first publishedin the late 1980s, which was a very different time

GJ MELENDEZ-TORRES [continued]: in the treatment and management of multiple sclerosis.One drawback, however, is that real-world evidencearound multiple sclerosis.Again, we couldn't completely accountfor differences between the natural historycohort and the treatment group.We could only account for the differencesthat we were able to directly observe.

GJ MELENDEZ-TORRES [continued]: Both forms of evidence were meaningful.Both forms of evidence were important.And having that real-world evidencehelped us to assess the generalizabilityof the evidence from our clinical trialsby thinking about how different those estimatesof effectiveness might be.[Combining Clinical Evidence-- Meta AnalysesNetwork Meta Analyses]

GJ MELENDEZ-TORRES [continued]: One of the things that we often do in health technologyassessment in the conclusion of the assessmentof clinical effectiveness is poolthe results of those clinical studies together.This is called a meta-analysis.Whenever you do a meta-analysis, the key question you needto ask yourself is, would putting the studylevel effects together tell me something useful?

GJ MELENDEZ-TORRES [continued]: Would pooling these clinical trials in the same analysistell me something useful?The benefits of meta-analysis whenyou are pulling statistically the results of multiple studiesare several.The first is you can gain a better powered estimateof the effectiveness of a drug.Any one individual trial may not beable to conclusively demonstrate effectiveness.

GJ MELENDEZ-TORRES [continued]: But if you pool multiple smaller trials together,you are more likely to generate a result thatactually reflects the true clinical effectivenessof that drug.The other and related reason is one of precision.When we pool different studies together in a meta-analysis,we can generate a more precise estimateof the clinical effectiveness of any one drug.

GJ MELENDEZ-TORRES [continued]: The third and final reason is generalizability.If you were pulling the results of multiple clinical trialsundertaken in your context, you'remore likely to be able to generatea more generalizable estimate overallof the effectiveness of a drug comparedagainst a relevant comparator.Meta-analysis is pairwise in nature.

GJ MELENDEZ-TORRES [continued]: What I mean by that is that it focuseson comparing one drug against another drug using the resultsfrom clinical trials.We can extend that to what's called a network meta-analysis.And network meta-analysis incorporates evidencefrom the range of clinical trials that compare two or moreof any of the drugs in your decision problem

GJ MELENDEZ-TORRES [continued]: to better understand how effective drugs are comparedto each other.This is important because it means that we borrow strengthwhen we're comparing one drug against another,not just from the trials that directly compare those twodrugs but also from each of the trials that compare oneof those drugs against placebo.

GJ MELENDEZ-TORRES [continued]: It also means that we can compare drugs that may nothave been directly compared in a clinical trial usingthe results of that network meta-analysis.Network meta-analysis, which are also called multiple treatmentmeta-analysis, multiple treatment comparisons,or indirect treatment comparisons,have rapidly become a central part of the statistical tools

GJ MELENDEZ-TORRES [continued]: that we use in HTA precisely for the reasonthat we can develop more clearly and more robustlyestimates of the comparative effectivenessof different drugs drawing on clinical trial evidence.Going back to our network meta-analysisand our assessment of the clinical effectiveness of drugsfor first-line treatment in relapse-remitting multiple

GJ MELENDEZ-TORRES [continued]: sclerosis, we use a network meta-analysisto compare all of the interferons and glatirameracetate against placebo, the beta interferonsagainst each other, and each of themagainst glatiramer acetate.We did that in one joined up statistical modelusing network meta-analysis.What we ultimately found was that all of the drugs

GJ MELENDEZ-TORRES [continued]: were effective compared to placeboat reducing the rate of relapses and reducingthe rate of disability progression.However, even though we found that each of the drugsindividually reduced the rate of relapse between 20% and 35%,there wasn't particularly clear evidence that any one drugwas better than any of the others

GJ MELENDEZ-TORRES [continued]: at reducing either the rate of relapseor reducing the rate of disability progression.You were able to make this conclusion with more certaintybecause we used that network meta-analysis to compareeach drug against every other, integratingall of the clinical effective evidenceinto one overarching model.If you want to do further reading

GJ MELENDEZ-TORRES [continued]: on clinical effectiveness evidenceand also how we combine that evidence using meta-analysisand network meta-analysis, I wouldrecommend that you read the Cochrane Handbook, whichis freely available online.The Cochrane Handbook is widely consideredto be the authoritative source of howto undertake systematic reviews of clinical effectivenessquestions, as well as the methods that you should

GJ MELENDEZ-TORRES [continued]: use when combining those studies statisticallyin a meta-analysis or network meta-analysis.[Conclusion]In conclusion, there are three key things that we talked aboutin this video.The first is that we talked about what

GJ MELENDEZ-TORRES [continued]: clinical effectiveness evidence is trying to get us to doand how that follows on from the importance of a decisionproblem that you use in structuringyour decisions about what clinical effectivenessevidence is relevant and in respect of which things.We also talked about different forms of clinical effectivenessevidence, such as randomized trials and real-world evidence,

GJ MELENDEZ-TORRES [continued]: and the relative benefits and drawbacks of each.We also talked about meta-analysis and networkmeta-analysis and with an examplefrom multiple sclerosis, talked about howthat could be used for comparative effectivenessand to better understand the relative effectivenessof different drugs against each other in your decision problem.

GJ MELENDEZ-TORRES [continued]: Thank you for joining us for this video, Introductionto Health Technology Assessment, with a focuson clinical effectiveness.We'll see you for the next video, whichfocuses on how we assess cost effectiveness in healthtechnology assessment as well.

GJ MELENDEZ-TORRES [continued]: [MUSIC PLAYING]nbsp;