Insights DKVF 2024: Preconference and State of the Art Session

While Dr. Roland Staples (FAU Nuremberg) raised the fundamental question at this year's DKVF preconference seminar as to whether the initiators of the Innovation Fund were using an incorrect definition of the term innovation, Prof. Dr. Holger Pfaff (IMVF Cologne) and Prof. Dr. med. Jochen Schmitt (ZEGV Dresden) put forward a proposal for a new evidence-based approach to health policy in the "State of the Art" session by clarifying how one can move from the "theoretically best to practically best evidence."

> In his presentation, Dr. Roland Staples referred to the term "innovation" commonly used in economics, which stands for the (complex) innovations associated with technical, social, and economic change that are—and here comes the crux of the matter: "discovered/invented, introduced, utilized, applied, and institutionalized." This means: The Innovation Fund, as it has been implanted into the healthcare system so far, falls short because the ideas identified as useful or not through projects and evaluations only become innovations when they are implemented—the main topic of the DKVF 2024—and thus available to as many, if not all, insured persons and patients as possible within the framework of so-called, but also not precisely defined, standard care.
Staples, a research associate at the Institute of Sociology at FAU Erlangen-Nuremberg in the Chair of Work and Industrial Sociology (Prof. Dr. Rainer Trinczek), also addressed a second, equally fundamental misunderstanding – at least from the perspective of organizational and innovation sociology: Not all innovations are created equal. According to his remarks, there are incremental innovations and disruptive innovations. Incremental innovations are usually product innovations (whether mobile phones or pharmaceuticals) that are relatively easy to research and implement. Disruptive innovations, on the other hand, are those whose implementation requires the "figure of a creative destroyer." This person must be able to "introduce something truly new from the perspective of a company's tradition, but also be aware that such disruption can also fail. The core message of the concept of "creative destruction," which goes back to Schumpeter, is: "Every economic development is based on the process of creative destruction. Through a new combination of production factors that successfully prevails, old structures are displaced and ultimately destroyed. Destruction is therefore necessary—and not a systemic error—for reorganization to take place." (1)
This excursion into the economic concept of innovation, largely shaped by Joseph Alois Schumpeter (* 1883 / † 1950), makes it clear that the Innovation Committee at the Federal Joint Committee (G-BA), in which the benches active in the Federal Committee primarily act and determine what is funded and, above all, what is approved and recommended for implementation in standard care, has a fundamental problem. One that culminates in the following question: Are the stakeholders of the healthcare system, who depend on its cash flow, creative destroyers?

Pfaff: “What is truth?”
Prof. Dr. Holger Pfaff, Chair of the Brückeninstitut and Director of the Institute for Medical Sociology, Health Services Research and Rehabilitation Science (IMVR) at the University of Cologne, cited Luhmann, not Schumpeter, to demonstrate why innovations take so long to develop from a positively evaluated idea into reality. According to Luhmann's systems theory, every society consists of different systems: politics, the economy, civil society, and finally, science. Pfaff: "All of these are self-referential, so-called autopoietic systems with their own code." Politics is about power, the healthcare industry is about money, civil society is about satisfying needs, and science is about nothing less than the truth. According to systems theory, however, the four systems do not listen to each other. Pfaff: "They hardly influence each other—that is Luhmann's prediction. That's a bit sobering." Furthermore, according to Pfaff's own thesis, all systems want to preserve their structures: the professionals want to continue as before, politicians want to keep everything as it is, and self-government even more so. This means, according to Pfaff: "The systems fundamentally resist influence from science." And if we ourselves also introduce doubts, this leads to a double structural conservatism. This leads to either a complete abandonment of evaluations or politicians saying, "If I don't hear from you, I'll decide over a glass of wine tonight." "At least, that's what a health minister said," Pfaff added.
And with a view to the DKVF 2024 congress theme, he explains: "I'm quite critical: Implementation science develops techniques so that civil society and the healthcare industry finally adopt the things for which we have discovered the truth. And then we are disappointed when that doesn't happen because the systems say: 'I'm not interested in that.'" And even more critically: "The idea is actually to force the evidence-based form into society. One might also ask: Where does participation come into play?"
What makes matters worse is that even the truth is a matter of perspective, meaning there can be multiple perspectives. For example, when it comes to the question of whether colonoscopy can reduce mortality, there are two truths: the relative and the absolute reduction. Pfaff: "This alone shows that there can be multiple truths, even where we have data."
Moreover, according to Pfaff, there are multiple levels of truth. This becomes particularly apparent when basic scientists—whether statisticians or epidemiologists—who are concerned with nothing less than the total truth, evaluate health services research studies. Using the much-discussed WiZen study (2) as an example, Pfaff argues that, from their respective basic scientific perspectives, all those who say that on the basis of this study, one cannot yet conclude whether the intervention is truly good are right. This apparent contradiction is primarily due to the fact that health services research is an "applied science" whose task is "to ensure that reasonably well-evaluated interventions are put into practice." Pfaff continues: "Because this is the case, we health services researchers do not have a view of the total, pure truth."
Although health services research is always about achieving the highest possible truth, according to Pfaff, this conceals the difficult issue of levels of evidence—more specifically, the difference between the best theoretical evidence and the best practical evidence—and the problems and misunderstandings caused by this discrepancy.
According to Pfaff, the stated goal of health services research, to promote rational health policy based on evidence-based knowledge, has so far been achieved through the application of the principles and methods of evidence-based medicine. This, in turn, is based on randomized controlled trials (RCTs) and meta-analyses of as many RCTs as possible; ideally, at level 1a or 1b, which is theoretically the best evidence. Anything below this level is rejected by the IQWiG, although there are now movements to relax this approach.
Or they are publicly criticized, as happened with the WiZen study (3). In his editorial in the German Medical Journal entitled "Evaluability of the Effect of Certification of Oncology Centers," Andreas Stang argued that the study under discussion (4) falls below the level that could normally be accepted.
Here, Pfaff explained, similarly to Staples, it is important to know that not all innovations are the same and that three types of innovation must be distinguished, and this already when an evaluation is planned:
• Product innovations (medicines, pharmaceuticals)
• Process innovations (guidelines, etc.)
• Structural innovations

The EbM system, which Pfaff stated has its justification, is not suitable for every type of innovation. While evaluation via RTC is certainly possible for product innovations, it is only possible to a very limited extent for structural innovations. Using the example of innovation fund projects, he explained that a pilot should actually be conducted three years before the project starts to test the basic principle to be introduced. Ideally, a four-year study should then be conducted if the MASI framework is taken seriously, and only then should the project application follow, which takes a total of eight years. After that, the results must be published and reviewed, which could take another seven or eight years. Pfaff: "That means we're 15 years too late." But only if there is no requirement to replicate the study results, which is virtually impossible in the case of structural innovations because all established structures have different contexts and therefore different outcomes. Pfaff's initial interim conclusion: "We have a real problem: We need a solution for structural innovations." The only choice is: "We either leave it as it is or wait 20 years." The latter, however, is not an option because new technologies have developed in the meantime, others have become obsolete, and the structures themselves are also changing. In other words: "There are several problems that hinder the evaluability of structural innovations."
Firstly, there is also what was already described in the 1948 essay "Resistance to Change." The reason for this is that people simply don't always do what we as scientists envision, which at the very least contaminates the experiment. Secondly, the considerable time required in an ideal scenario leads to the abandonment of the pilot study, thus integrating "teething problems" into the proposed project study and testing them, which in turn can distort the results. Since such complex package tests always test a complete package of interventions, one doesn't know afterwards which components are truly effective. This, in turn, also complicates replication.
All of this leads to high innovation and evaluation costs in structures, which are a deterrent. This is different with medications, where the trial and error principle is simply applicable. Pfaff: "That's precisely the great advantage of RCTs: I don't need a theory. You can test an idea by using trial and error and see until you eventually find the solution. That's not possible with structures, partly because it takes far too long." RCT results are also still valid after 20 years because they require a stable environment, an unchanging medication, and an equally unchanging person. However, given dynamic environments and rapidly obsolete structural innovations, the EbM system does not work.
All of this, but especially the limited applicability of the criteria of evidence-based medicine, according to Pfaff, brings structural innovations to the "limits of evaluability." Pfaff: "For this reason alone, our evaluation results, no matter how hard we try, are fundamentally always open to criticism." There will always be critics who want the pure truth. They don't do this out of malice, but "because they have learned that the truth" is paramount. Even if—another excursion to Luhmann—the absolute top truth doesn't exist.
The fatal aspect of this is that "this sows the seeds of doubt." He quoted from the book "Merchants of Doubt" (5) by Erik M. Conway and Naomi Oreskes: "It only takes one or two people to express doubts and the majority no longer believes in scientific findings." According to Pfaff, this, in turn, affects implementation: "If some people start not to do something based on even the slightest doubt—which may also be genetically implanted—healthcare research either offers only cautious advice or no advice at all, because we can't be sure whether an efficacy has truly been fully proven."
This, in his view, leads to scientifically induced structural conservatism. The definition of this conservatism, which can be found in the forthcoming article "Give innovation a chance: Overcoming structural conservatism of evidence-based medicine and health policy decision-making by returning to Sackett," which will be published in the journal "Health Care Research & Implementation," is: "A practice that makes decisions in a way that systematically—intentionally or unintentionally—obstructs the development and implementation of new structures and thus indirectly preserves old ones."
Pfaff's (self-)admonition: "We share the blame for the fact that nothing is moving forward because we express too much doubt about our own studies. This is our contribution to structural conservatism, which contributes to the sluggish development and implementation of new structures and indirectly preserves old ones." And this against the backdrop of the fact that the old structures have never been tested because they evolved from their respective times and their possibilities. Pfaff: "Yet we impose an evaluation quality on new things that the old structures didn't even have to pass, because they simply arose that way."
This means, says Pfaff: "We are under pressure to act." Health services research cannot allow gut instinct to ultimately decide because "we are not delivering." This also means, Pfaff continues: "Before we no longer provide any evidence at all, we must find a middle ground between the highest level of evidence and none at all." Therefore, in the future, a distinction must be made between three types of evidence:
a) the theoretically best evidence, which EBM aims for with its implicit freedom from doubt memo.
b) the best available evidence, which David Sackett describes in his definition
c) the best practical evidence, which under the given circumstances achieves the highest level of evidence, which Pfaff and his colleague Schmitt bring into play.

This simply means that, first and foremost, the maximum evidence possible for each condition must be defined using the latest methods. Conversely, the definition of maximum evidence also requires the research community's willingness to adhere to this maximum: "If it's possible, we'll conduct the study; anything below that, we won't accept it."
This is not about pragmatic evaluation or even a realistic evaluation.
tion. It is nothing less than striving for the "highest achievable evaluation design under the given context, based on the latest evaluation research and influenced by mechanistic studies." For this work, Pfaff and Schmitt propose a structured consensus-building process modeled on the S3 guideline development: "This process must determine what constitutes the highest practical achievable evidence under the given context."
The only thing that is clear so far is that the best available level of evidence will certainly vary depending on the context and the respective political, economic, social, ethical, psychological, organizational, regional, cultural, technical, and structural (framework) conditions. It may be that an RCT can indeed be conducted in context A, but not in other contexts, but perhaps a controlled, quasi-experimental, quantitative, or correlational study.
Nevertheless, according to Pfaff's statements, it remains undisputed that an uncertainty gap will always exist between the best theoretical evidence and the best practical evidence. However, this uncertainty gap must be accepted by science, politics, and society, knowing full well that it can be exploited by critics at any time. According to Pfaff, politics must also accept having to make decisions under uncertainty. Pfaff: "EBM is – to put it bluntly – something for cowards." In the healthcare system, as in business, one must learn to make decisions under uncertainty. However, the uncertainties must be minimized to the extreme by scientists trying the maximum possible and pushing the methodological boundaries."
Referring to the WiZen study, which he considers to be the best available evidence for the tested constellation, he added: "It may be possible to do even better, but we don't yet know how."

Schmitt: the question of “how”
Prof. Dr. med. Jochen Schmitt, Director of the Center for Evidence-Based Healthcare (ZEGV) at the Carl Gustav Carus University Hospital and Medical Faculty in Dresden, and currently Deputy Chairman of the German Society of Medical Sciences (DNVF), explained the "how" in his lecture "EbMplus - International Perspectives and Developments." Schmitt's remarks should be preceded by the observation that the definition of the "EbMplus" model, developed by the two German health services researchers Pfaff and Schmitt, emerged parallel to the international "EbM+" development. Both "EbM+" and "EbMplus"—and this is very important, according to Schmitt—emerge from the actual EbM movement. This is not a countermovement, but rather an expansion of the traditional EbM concept through the integration of additional dimensions and a broader and more diverse evidence base.
According to Schmitt, EbM+ (international) is significantly characterized by two strands of development:
Prof. Jon Williamson, Professor of Reasoning, Inference and Scientific Method, Leeds, with his central theme of causality: "Evidence of mechanisms should be integrated with evidence of correlation to better assess causal claims." He postulates that a causal inference requires a "correlation claim" and a "mechanistic claim," with one type of evidence not being sufficient to demonstrate causality. Likewise, Williamson, echoing Bradford Hill, introduces "Evidencial Pluralism" - a model which, according to Schmitt, ultimately amounts to a combination of quantitative, qualitative, and mechanistic studies. According to Schmitt, he is saying that evidence based on one study and one design is not sufficient to demonstrate causality. Therefore, it is always necessary to combine qualitative studies with quantitative and mechanistic ones to overcome the efficacy-effectiveness gap. According to Williamson, effectiveness is "efficacy plus external validity." Schmitt: "This is extremely interesting because this external validity arises from mechanisms that are as robust as possible and always work the same way under different conditions."
Prof. Trisha Greenhalgh, Professor of Primary Health Care, Oxford, with her approach of "real evidence-based medicine." According to Schmitt, she calls for a reform and re-evaluation of traditional EBM to adapt it to the challenges of a complex and dynamic healthcare landscape, which necessitates interdisciplinary approaches: medicine, social sciences, computer science, and the exchange of knowledge and experience.
It's important to note that the international understanding of "EBM+" includes, firstly, only empirically proven mechanisms and only empirical, data-supported theories. Secondly, the "+" primarily stands for biomedical contexts. This is different in the German "EbMplus," where the "plus" primarily refers to the application and inclusion of social science theories.
This distinction is important because, according to Schmitt, “causality in complex interventions can be context-dependent” and thus, in addition to deterministic mechanisms, probabilistic mechanisms with their technical, interpersonal and psychological components always play a role, which will always influence the outcome.
Small note: Pfaff and Schmitt initially also distinguished between deterministic and probabilistic theories and then placed greater emphasis on probabilistic theories because they believe that these are crucial, especially for everything beyond the scenario of “simple interventions”.
Another important difference is that international "EbM+" development has so far primarily focused on "micro-level interventions," as Schmitt put it. The German "EbMplus" model, in contrast, focuses primarily on process and structural innovations. According to Schmitt, this means: "Although German health services research has so far made few connections to international "EbM developments," we can also make a relevant contribution internationally!"
To achieve this, German health services research must devote more attention to basic research in addition to pure applied research. According to Schmitt, fundamental and central principles and mechanisms must be developed and tested in order to work on understanding mechanisms in the health care and interpersonal context.

Links to the lectures given at the DKVF:
Pfaff: https://t1p.de/zfhms
Schmitt: https://t1p.de/61sgt

The basis of Pfaff and Schmitt’s statements are their successive works:
doi.org/10.3389/fpubh.2021.727427
doi.org/10.1111/jep.13890
DOI: 10.1055/a-2350-6435