>> „Golden Lamb, Sacred Cow, or Simply Common Sense?“ This was the title of the opening lecture given by PD Dr. Stefan Lange, Deputy Director of the Institute for Quality and Efficiency in Health Care (IQWiG), at the Scientific Forum „RCT – Bias Control?“ in Berlin. The forum was the first in a planned series of such methodological forums organized by the Institute of Clinical Economics (ICE). Lange, who described himself as having a „penchant for rationality,“ which is why he works at IQWiG, began his lecture by quoting Section 2, Paragraph 3 of the German Ordinance on the Benefit of Medicinal Products (AM-NutzenV): „The benefit of a medicinal product within the meaning of this ordinance is the patient-relevant therapeutic effect, in particular with regard to…“. He emphasized the concept of „effect,“ around which the entire early benefit assessment ultimately revolves.
This effect, according to Lange, who cited Wikipedia as evidence, is nothing more than "an effect caused by a specific cause," based on causality. And this causality can be demonstrated with a randomized experiment, whereby randomization, in his words, means "not more, but (sometimes considerably) less effort" to obtain meaningful results when it comes to benefits (or harms).
To those who frequently argue against randomized controlled trials (RCTs) that blinding ("often used by medical device manufacturers") is impossible, or who cite the rarity of diseases or events, he countered: "Forgoing randomization does not make a rare disease more common or a rare event more relevant." But precisely for this reason, he argued, "it is not reasonable to compromise a second component (quality) when one component of outcome reliability (precision) is already compromised." His interim conclusion was therefore: "Forgoing randomization is just as irrational for rare diseases, small target populations, or rare events as it is for common diseases or common events."„
Lange also spoke out against the use of real-world data as a supplementary tool, not because it is inherently inferior, but because the crucial factor is the justification and presentation of an effect. According to Lange, comparing the results of an RCT with data from non-randomized everyday life to address a specific research question constitutes a "variation of at least two possible effect modifiers," leading him to the core question: "How should differences in effect estimates be interpreted?"„
Hence his advice, because otherwise no clear interpretation is possible: "Anyone who wants to identify a potential effect modifier ('everyday life' vs. 'ideal conditions') would be ill-advised to simultaneously modify another (study) characteristic." This is also because, in his view, data from observational studies—the so-called "real-world evidence"—are "inherently subject to a selection bias that cannot be controlled by any statistical method in the world." And this leads him, in turn, to his absolutely unambiguous, almost categorical, concluding statement: "There is (virtually) no rational argument for resorting to non-randomized studies for a benefit assessment."„
Reviewing Lange's presentation, one cannot help but get the impression that RCTs are sensible, but also a kind of "sacred cow", because the deputy head of the IQWiG did not address a single point that could call into question the "gold standard RCT".
This only happened after a lecture by Prof. Dr. rer. pol. Jürgen Wasem, Chair of Medical Management at the University of Duisburg-Essen, who, as an economist, stated that he would work with all available instruments—provided they were appropriate to the issue at hand—following the motto: "The health economist is frugal and takes what he can get and processes it for his own purposes." For this highly respected economist, who holds many important positions within the healthcare system, a statement still holds true that originated from an expert group (Jöckel, Kohlmann, Raspe, Wasem) appointed by the Federal Ministry of Health (BMG) in 2007 under then-Minister of Health Ulla Schmidt. It reads: "Benefit presupposes proof of efficacy. Randomized clinical trials (RCTs) represent the gold standard for demonstrating efficacy and benefit. However, it may become necessary to use other types of studies to provide further evidence of benefit. When evaluating and interpreting studies with lower levels of evidence, a particularly careful examination for systematic bias is necessary."„
In his opinion, however, if reimbursement and pricing are based on willingness to pay for added benefit, it would be preferable if the "real money" were matched by "real added benefit," meaning that "effectiveness has been demonstrated not only in the artificial RCT design but also in real-world healthcare practice." Due to the complexity and heterogeneity of healthcare provision and disease processes, he believes this "also requires the use of non-RCTs.".
However, he also mentions the general problem that both non-randomized (at least) two-arm studies, in which the subjects in the control arm are prospectively included, and single-arm studies with retrospective matching of control arms (e.g. generated from routine data) would suffer from the problem of self-selection bias, so that in his opinion the possible extent of the bias would only have to be qualitatively estimated and critically discussed.
Prof. Dr. med. Franz Porzsolt agrees.,
Chairman of the Institute of Clinical Economics eV (ICE), which he founded, he said that his lecture was intended to initiate an "open discussion about the risks of bias." In his opinion, this discussion is urgently needed because it is the only way to answer three core questions posed by Archie Cochrane and Austin Bredford-Hill: "Can it work?" (Efficacy), "Does it work?" (Effectiveness), and "Is it worth it?" (Value).
To approach these questions, it is essential to openly discuss the strengths, but above all the limitations, of RCTs. Among the weaknesses commonly cited is that RCTs cannot claim to describe "effectiveness" under everyday conditions, but that their study design allows for a "similar distribution of known and unknown risk factors in the studied groups" and a "valid description of effectiveness" under ideal conditions (exclusion of confounding factors).
Porzsolt immediately raises serious questions about this, turning several previously accepted doctrines on their head. In his opinion, not only is the external validity questionable, because exclusion criteria and the doctors' preferences for studies are being considered, but the studies themselves also raise concerns.
The study considers not only the likelihood of participation and the strength of participants' preferences, but also the rarely questioned internal validity. For Porzsolt, an internist, oncologist, and clinical economist, this includes both the disregard of "known" risk factors and the effects of unknown risk factors, which randomization is intended to eliminate.
But is that correct? His answer came immediately: "No, that's wrong!" This is primarily because the so-called "Table One" (in which all inclusion and exclusion criteria must be listed for all RCTs) only includes known risk factors, as the randomized study design creates a false sense of security. This is based on the assumption that, due to the random distribution itself, unknown risk factors are equally distributed between both study arms. However, Porzsolt argues: "The smaller the study, the greater the risk posed by unknown risk factors."„
So far, this is completely uncharted territory because no one knows the threshold values that can be derived solely from statistics. Therefore, Porzsolt commissioned mathematician Christel Weiß from Mannheim to calculate the required study size to ensure that, assuming only 10 unknown risk factors, all unknown factors are equally distributed across the study arms. The purely mathematical-statistical answer, based on extremely conservative assumptions about the risk factor categories—dichotomous rather than continuously distributed variables, and independent rather than dependent variables—and a permissible threshold of 5% (instead of 1%) deviation for an assumed even distribution, yielded the desired result.
The result: A tolerated error probability of 5% is only exceeded at 0.00328, which is only the case with "n over 1,000". In other words: All RCTs with fewer than 1,000 participants cannot, for purely mathematical and statistical reasons, distribute unknown risk factors equally. Since the vast majority of published RCTs have fewer than 1,000 participants, Porzsolt argues that "it simply cannot be guaranteed that unknown risk factors are distributed equally".
To be able to discuss a possible solution, scientist Porzsolt advocates a step-by-step model. In what he considers the "very first and most important stage," RCT, it must first be demonstrated that the effectiveness of an innovation can be proven under ideal conditions. If this is not the case, any further engagement with the topic is pointless.
If an innovation can demonstrate effectiveness ("can it work" – "efficacy") under ideal conditions in hurdle 1 (RCT), the next stage, stage 2, must follow, which answers the question "does it work – effectiveness" under everyday conditions. According to Porzsolt, this second demonstration is extremely important because not everything that works under laboratory conditions works under everyday conditions, and vice versa (see Table 1): "Some things even work better in everyday life than in the lab." Therefore, stage 2 must investigate the question "does it work": "Does the innovation deliver in everyday life what it promises in the lab?"„
According to Porzsolt, efficiency and effectiveness should not be understood as two sides of the same coin, but rather as two distinct dimensions. The reason: one cannot infer the result of one dimension (e.g., efficiency) from the result of the other (e.g., effectiveness) – and vice versa. Porzsolt asserts: "If you measure efficiency, the effectiveness result is unpredictable: it can be better, but also worse, which in turn is related to the unpredictable distribution of risks."„
If the effectiveness result is positive, Porzsolt believes a third stage must follow, answering the question: If an intervention works under ideal conditions (stage 1, i.e., when all confounders are eliminated) and shows a similar effect under everyday conditions (stage 2, i.e., when none of the confounders are eliminated), then the question arises whether this (statistically significant effect) is also perceived as added value by the patient and society (is it worth it – „value“)!
The second question („does it work?“) addresses everyday conditions and, according to Porzsolt, therefore cannot be answered „in the laboratory.“ He thus calls the study design available for this purpose a „pragmatic trial“; and because this trial includes a control group, it is called a „pragmatic controlled trial“ (PCT). This PCT is a descriptive procedure used to confirm effects that have been statistically demonstrated as efficacy under ideal conditions in an RCT, also under everyday conditions – but again using statistical methods – as effectiveness.
The third step, however, requires answering the question of whether the effect actually observed in everyday life (e.g., a significant increase in sleep duration with a new sleep aid) is also perceived as "clinically relevant" or "valuable" ("is it worth it?"). According to Porzsolt, this proof cannot be provided with either an RCT or a PCT: "Whether something is worth the effort is something every individual patient will assess for themselves, without using a computer. Likewise, every experienced manager will be able to answer this question for their company without pen and paper." Nevertheless, there are scientists who do not accept this "assessment" and demand a computational algorithm whose accuracy is virtually impossible to verify.
Overall, the three evaluation steps he presented together form a clear concept:
1. the RCT,
2. the PCT and
3. an economic valuation method or an estimate.
These three different, sequential (but not mutually cannibalizing) methods could answer the three core questions of health research that have been around for 40 years. Porzsolt: "These consecutive methods initially describe the principle of what steps a preliminary benefit assessment could look like. However, this already puts us a considerable step further than we would have been without them." Whether this is truly the case still needs to be discussed openly and, above all, reached a consensus. Here, Porzsolt asks that we remain firmly grounded in science, instead of too quickly slipping into a politically evaluative level of discussion. Therefore, everyone should decide for themselves which of the presented statements are based on scientific principles and which on political assessments, which can clash again at the 2nd ICE Discussion Forum in autumn 2018.
by: MVF editor-in-chief Peter Stegmaier