>> For the eighth time, “Monitor Healthcare Research” held its annual scientific congress, which is dedicated to the important topics of healthcare research and healthcare management and illuminates them from the perspectives of the various stakeholders. This year the focus was on the topic of “extrapolation”. This happened at the suggestion of Prof. Dr. Jürgen diaperer, head of the Institute for Quality and Efficiency in Health Care (IQWiG), who explained in the title interview with “Monitor Healthcare Research” (MVF 01/18): “RCTs are not the basis of EBM, but rather they are for the question of Use the instruments with the lowest errors – no more and no less.”
In the MVF interview,wickeler explicitly used the term “lowest error” because he knows full well that every RCT, no matter what design you choose, has its limitations. Pragmatic studies that work with as few inclusion criteria as possible in order to cover the broadest possible range of applications can be useful, but according towickeler, “they become very problematic if, for example, B. Doctors and patients do not stick to the original randomization and everything goes haywire.” Then in the end the significance goes to zero. But the same thing can also be discussed in classic RCTs, because inwickeler's view there is a sharp distinction between "ideal"
and “everyday life” does not exist, just as the often emphasized difference between efficiency and effectiveness, which he simply considers to be nonsense. Also because no one knows exactly whether a study that tries to depict an “everyday life” ignores many other “everyday lives”. He would therefore “like to know exactly what is actually happening in care.” This in turn is a question of extrapolation, which the MVF specialist congress in 2018 was dedicated to. Below you can read the editorial summary of the first part of the specialist congress, which dealt with the status quo of the transferability of study results.
Dr. Ilona Köster-Steinebach:
increased focus of research on patient preferences
“How do we design an evidence-based health system?” That was the opening question from Dr. Ilona Köster-Steinebach, patient representative in the G-BA for the Federal Association of Consumer Organizations for eight years, is now the designated full-time managing director of the Patient Safety Action Alliance (APS) and, because she has not yet been confirmed in office, in a transition phase, which is why she spoke in her opening lecture at the MVF specialist congress “Extrapolation 2018” represented her private opinion. But also because “such complicated evidence questions” have not yet been fully discussed in all facets by patient representatives.
In order to approach her core question – “What does evidence mean for patients?” – she began by using a definition that comes from quality assurance. This states that quality is the extent to which health services for individuals or populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge. What is positive about this definition is that the patients' preferences and also the term wish are actually mentioned explicitly. Köster-Steinebach: "But somehow the preferences in this definition seem to be assumed to be known." Which raises the question: "Are they actually that?" And: Can the patient's wishes only be realized as a result of treatment?" " Or are there other ways to fulfill this patient's wish?" The core questions immediately follow: "What connection does the professional level of knowledge have with the patient's wishes? Is the professional level of knowledge the standard against which patient preferences should be based? Or do patient preferences dictate what standard of knowledge should be developed through research?”
From their point of view, research focuses much more on the question of probability - which variant, which treatment result is likely to occur - than on the patients' actual preferences and the question of whether a treatment result achieved corresponds to what patients expect want from their treatment. Their clear playdoyer: “An increased focus of research on patient preferences and a greater involvement of patient representatives, whereby one should also think about new political frameworks. And last but not least, a discussion of methods in science, which should enable a better transfer of the principles of EBM, but this does not mean that “everything has to be RCT”, whereas there would be a fundamental mistrust regarding the question of whether and when a bias exists. which is why she advocates a “return to the method of rigorous falsification” instead of the arbitrariness observed with the G-BA.
Dr. Antje Behring:
Evidence transfer instead of extrapolation
There was also a vocal veto from Dr. Antje Behring, consultant in the G-BA's pharmaceuticals department, who does not believe that her employer, the Federal Joint Committee, works "so arbitrarily", although she only provides an insight into the pharmaceuticals part, the areas surrounding Section 35a SGB V – the early benefit assessment and the associated consultations. In her lecture she addressed the evidence
G-BA actually uses to make its decisions, which would also include patient preference data. And here I always find myself in the conflict of how to objectify patients' statements as best as possible. This begins with the validation of the related instruments, especially questionnaires, and their general usability for a corresponding drug therapy or indication.
But overall, the G-BA has very little experience with patient preferences, although the Federal Committee was involved in the PREFERE study in order to sharpen patient preferences. However, “there have been no good experiences with this so far,” and “there are not really good, positive examples yet” in the area of drug therapies when it comes to patient preferences.
Therefore, the gold standard in the field of pharmaceuticals remains the RCT; randomized controlled studies are still required not only for the initial evaluation, but also for limited periods. However, deviations from RCTs are only possible in exceptional cases, such as dramatic effects, where single-arm studies and registry data are “acceptable in very rare cases”. This is where the methodology begins: “As long as there are only single-arm studies, only non-quantifiable statements are possible,” explained Behring, thus addressing a conflict that is quite difficult to understand: Because the G-BA has to carry out these approval studies for orphan drugs , but also accept other types of approval such as “Conditional Approval” or “Exceptional Circumstances”, and thus also the exceptions and uncertainties that come with them.
Behind this is the divergence why, on the one hand, the approval certifies good quality in its yes/no decision, for example, and the safety and effectiveness are also sufficiently proven, but due to the fact that these are single-arm studies, the G-BA may “ “no additional benefit” is certified. Behring's answer: The sole reason for this is that the Federal Committee has to make a comparison that requires comparative data in order to be able to quantify how high the additional benefit is. However, in the case of “Conditional Marketing Authorizations” and “Exceptional Circumstances Authorizations”, evidence only has to be generated after approval. Behring's question: “What evidence can be generated and how?” Are these actually registry studies? Or is it simply the continuation of RCTs that have already begun?” This is the explanation for many of the time limits that the G-BA incorporates into its decisions: “Preliminary data at the time of the benefit assessment actually only results in a preliminary assessment.”
But what about the real quality of the data obtained in RCTs and their transferability? “We haven't done extrapolations of data very often yet,” admits Behring openly, although she excludes PUMA drugs here. And the transfer of evidence is not that unusual, as she shows with the example of ledipasvir and sofosbuvir in the area of hepatitis C. Here we looked very closely at what evidence was available for adults and what evidence was available for children in order to answer the question: “Can I transfer the considerable evidence in adults to children or not?” The matter is different when it comes to those under consideration Studies patient groups are missing. Then doctors who work in the G-BA would say: “Can I actually find the patient I want to treat in this study?” And: “How do you deal with that?”
Prof. Dr. Gerd Antes:
Risk of bias as a guiding principle
Nobody can do that better than Prof. Dr. Gerd Antes, the scientific director and deputy director of Cochrane Germany, spoke about the seemingly perfect world of RCTs ("it's not") and "Real World XXL", the big data. At the beginning of his lecture, he introduced the patient, whom he called Mr. Neumann, who, after shared decision making, followed his doctor's treatment recommendation, but ultimately asked himself: "Was that actually right?"
There is only one way to determine the alternative, in which the patient “simultaneously chooses the opposite or doing nothing,” which would be purely fictitious for two reasons: Firstly, this option does not exist except in dermatology, where, for example, one only has to left arm smeared with ointment. And secondly, the knowledge gained in this way does not help the patient himself, but perhaps the next generation. “All we’re talking about here is replacing the two halves of Neumann with groups,” Antes explains his pragmatic thought process: “Everything we’ve learned over the last 100 years is how to optimize these groups so that there is actually a The result comes out with minimal bias.”
What is important here is “maximum protection against systematic errors”, or bias in modern German, for which co-factors are balanced, randomized and blinded or not. For him, the key phrase on the path to truth is: “Counterfactual Thinking,” which means nothing other than: “What would actually happen if I did the opposite?” This is a philosophical approach that goes far beyond medicine. There are actually only two types of errors to consider, but there are plenty of them: systematic and random.
We try to keep systematic errors to a minimum in studies. You can't get rid of them, says Antes, and "if you're unlucky, chance will get you." Antes: “These two are closely related, mastering them is the great art.” Therefore, his message is: “The gold standard is not the RCT”, but: “What is best for Neumann?” This is largely that RCT, whereby one could just as easily collect information from patients in very similar situations all over the world and make it available to the individual patient as a basis for decision-making. However, except for a few regions such as Scandinavia, this is opposed to data protection, which is why studies are the most efficient way. Antes: “If we now add registers and call these observational studies, the picture is even correct.”
But the core problem is unsolved. Although an incredible number of methodological studies are being carried out, one must learn to understand how to find the important screws that need to be turned in order to improve the output. “Unfortunately, what we know very well is that 50 percent of what is started at the top does not reach the patient,” regrets Antes. This is almost like a natural constant.
But it's certainly not the sheer number of studies, says Antes, who is trying to prove that Medline has an extremely high increase of around 30,000 studies per year. Another 50 percent are not published at all, which amounts to around 60,000 new studies per year: “If you talk about big data, you can start here.”
The unpublished studies are a gigantic waste, a fraud on the study participants, because the information generated is not available to the next generation. In addition, an irreparable, horrible bias is being introduced as the basis of the three main axes – HTA, clinical guidelines and patient information. “All guideline groups, G-BA, IQWiG or IQTiG or patient information are based on this false basis,” says Antes, adding that nothing can be done about it in the short term despite all efforts. But the problem can be tackled, for example with a process that the Canadians called “Knowledge Refinery” years ago and which follows a five-step scheme:
1. Formulate the question
2. Systematic search in the literature
3. Quality assessment of the finds
4. Summary of evidence
5. Interpretation of results
Anyone who does this receives a so-called “forest plot” or “cumulative forest plot” in which the evidence is condensed. But: How far is the question, because according to statistical logic it has to stop somewhere, at the latest when the confidence interval of the collected studies - the sum of the accumulated knowledge - no longer meets 1, it would be over. Antes: “But that is mathematically wrong” because factors such as “repeated testing” and “alpha correction” also have to be taken into account. Nevertheless, to this day there is “no accepted logical, mathematical-statistical stopping rule”. But even if there were, Antes is certain, the German ethics commissions - and those of many other countries - would not be able to implement this rule.
These are all purely RCT triggered problems. But now real world observation studies would also be added, which would immediately lead to a frontline position in a purely German undesirable development. Health services research versus RCT is not a contradiction, because the best method is needed for each question. Antes: “The stumbling block is the hierarchy of evidence from randomized studies to cohort case control to non-transparent experts.” And further up to big data, digitalization, personalized medicine, translation, innovation and artificial intelligence, according to Antes “a mishmash of really strong ones Confusion of terms”, described in the two books “The End of Theory” and “Big Data”. These would describe a sheer flood of data and ultimately postulate that scientific methods would become superfluous. And even worse, that causation would emerge from correlations alone, which is “simply wrong”. Although Big Data claims to be able to analyze unstructured data - far beyond observational studies - the whole thing is based on a scientifically incorrect basis, because, according to Antes: "More data is not necessarily better, but requires disproportionately more effort to achieve this To get the right ones out.”
All of this leads him to his very personal “Evidence Hierarchy 2025”: experimental studies at the top, followed by observational studies and at the bottom big data, as of today. Antes: “Look for the term quality in Big Data. Find a confidence interval in big data. Look for the extent of uncertainty in the statements: You won't find any of this." But that's simply because a computer scientist can calculate everything that is correct if only the amount of data is correct. The reason: With an infinite amount of data, every test inevitably becomes significant. The problem: According to Antes, there is still no agreement between statistical significance and clinical relevance. Everyone who studies medicine in their first semester has to learn this. That's another reason why he was happy to come to the MVF congress, to try to bring things together again and not just talk about where things don't fit together. His conclusion to the first: “Quality at all stages of the knowledge process is the highest benchmark against which we must develop everything.” And to the second: “The risk of bias must become completely normal as a guiding principle.”
Prof. Dr. Michel Wensing:
from scale-up to scale-out transfer
Prof. Dr. Michel Wensing, who worked at Nijmegen University Hospital for 25 years before taking up the professorship for implementation science in the Department of General Medicine and Health Services Research at the University of Heidelberg, addressed the issue of the transferability of studies from abroad in his lecture. To the question “Are research results from abroad relevant for Germany?” he has three potential answers: “Yes, if the situation is similar”, “No, only research in Germany is relevant” and thirdly: “It depends. “
Incidentally, this wide spread of answers is due to the fact that there is so far little literature that would help here. One of them comes from Aarons, who described the difference between scale-up and scale-out transfer in 2017, another from Stirman, who in 2013 systematically inventoried, documented and created over 250 categories of changes that can lead to that an intervention may not ultimately be as effective as described in the original study. According to Wensing, this has to do with “program drift” or “voltage drop”.
While “program drift” means that the inclusion criteria are different in reality than in the original study, “voltage drop” describes a possible influence of less well-trained doctors or nurses. This in turn could lead to the intensity of an intervention in the
Reality is not as high as in the original study. Wensing: “For this reason, you have to learn how context mechanisms work and what influence they can have on outcomes.”
In order to research complex interventions, one always has to first build logical models that explain how an intervention program actually works. This includes, among other things, modifications and influencing context factors in order to be able to clarify the fundamental question: “What impact do these modifications and these context factors, which may be specific to Germany, have on the effects?”
This also includes the “benefit-harm ratio” (“Is this still positive after the transfer to Germany?”), the cost effectiveness and the costs after implementation (“Is the cost effectiveness after the transfer and after implementation still attractive?”) and the “budget impact” (“Because sometimes costs have to be borne primarily by certain service providers or stakeholders”). All of these are focal points of the field of implementation science he represents, which describes which measures and strategies can be used to promote implementation.
Prof. Dr. Neugebauer:
“on the value of RCT”
Anyone who expected that Prof. Dr. Prof. hc Dr. hc Edmund AM Neugebauer, the dean of the Brandenburg Medical School (MHB) in Neuruppin and as a long-time board member of the
DNVF speaking for the AG Register of the German Network for Healthcare Research e. V., would give a “burning lecture for registers and against RCT”, was disappointed. Neugebauer, who himself comes from clinical research and has carried out many randomized controlled studies, says that he does not need to be further clarified “about the value of RCT”, although - this is the content of his contribution - there are limitations, for RCT on the one hand Register on the other side.
In doing so, he certainly concedes that certain quality criteria must also be set for registries, which, however, not so many registries would meet at this point in time. There are also positive exceptions such as the cancer, trauma or StuDoQ register of the German Society for General and Visceral Surgery. In general, as with RCTs, certain quality criteria must be applied and met for registries, as already defined in the DNVF register memorandum in 2010:
1. Systematics/appropriateness of a register
2. Standardization
3. Validity of sampling
4. Validity of data collection
5. Validity of statistical analyzes and reports
6. Comprehensive quality requirements

“All of this has to be fulfilled in order to be able to speak of a good register,” explained Neugebauer. That's why, similar to a clinical study where there is a study protocol, there is of course also a registry protocol.
An important area for registries and registry research based on them is to be able to evaluate populations that do not or cannot participate in studies. You can compare a larger population with the subpopulation in a study if you run a register parallel to the study with all those who were not included in a study. Registers can also be used to record rare or delayed events, which is hardly possible with study durations of perhaps one, two or three years in randomized studies, even with appropriate follow-up times.
The point at which opinions differ, however, is the use of registers in the evaluation of effectiveness and benefits in everyday care. That's why Neugebauer compared the different characteristics of clinical RCTs on the one hand and prospective registries on the other.
In clinical studies, due to the number of cases, only a limited number of subjects or patients could be included, which would also be “mostly very, very homogeneous”. Especially in drug studies, these would be so homogeneous that an almost equal group of patients would be included in such a study. But the institutes carrying out studies are also heavily selected and are therefore just as homogeneous with often narrow inclusion and exclusion criteria.
With registers, on the other hand, this is the opposite: an unlimited number of heterogeneous test subjects are observed in heterogeneous settings, which means: normal environment, open inclusion, different types of observation levels, etc. On the other hand, with RCTs the comparability is much better than with registries, but " in any case always associated with a certain bias. This is despite the fact that the amount of data in randomized studies is generally much greater than in registries, at least that is still the case today. But this is also due to the often lack of support for registries, even though the effort and costs for RCTs are significantly greater than for registries. Neugebauer sees the strengths of registries in quality assurance, epidemiology and the description of changes in care.
The question of proof of benefit still remains. This is still the big question with registries today, but with RCTs it is assumed that this is where they have their greatest strength. Because with RCT there are definitely problems with a lot of distortions. This starts with the narrow, homogeneous study population with low co-morbidity, minimal co-medication, narrow age window and many other associated exclusion criteria. However, this also includes the patient's willingness to take part in studies, which entails further patient selection. But care bias due to standardized, quality-tested treatment by experts should also be taken into account. Finally, we should not forget the observation effect, which may lead to unrealistic results.
But what does the external validity of clinical studies look like? Neugebauer's answer: "In clinical studies, you can only draw conclusions from the study patients for the target population." Statistical conclusions are only possible for the patients included in the studies and therefore only for the study population. And quite categorically: “If the population is not the target population, a conclusion from the study population to the target population is not easily possible.” However, in purely practical terms, you cannot do so many studies so that all possible differences in patients are actually depicted . However, external validity is less about whether the patients in studies are different than those in later practice, but rather about whether the therapy effects are different and how their effect is modified. Neugebauer: “The goal of health services research should be to identify and analyze these effect-modifying parameters and thus to be able to better explain the factors influencing external validity.” <<

from:
MVF editor-in-chief Peter Stegmaier