>> Prof. Porzolt, once you have tried RCT and Pragmatic Trials...
…Stop. It's like a craftsman trying to alternately use a screwdriver and a pair of pliers for the same job. No reasonably trained craftsman will say: hand me a screwdriver or a pair of pliers, I'll get by somehow.

What are the differences between the two instruments?
At the? The basis of evidence-based medicine is clinical epidemiology, which includes a whole bundle of scientific rules or - to stay in the picture - provides a fairly extensive toolbox whose instruments you have to use when planning and carrying out a study. However, you should know the advantages, disadvantages and limitations of the respective instruments very well. The big problem of the last 25 years - because that's how old evidence-based medicine is now - is that many scientists basically reduce the modular system of control measures to randomization. I'm not saying anything against randomization as such, but rather against the standard opinion that almost inevitably goes hand in hand with it: that everything that is a randomized study is good and, conversely, that everything that is not randomized must therefore be bad.

Isn't that also something like an implicit bias?
He still comes on top. This bias arises simply from the way we perceive things. The implicit bias actually comes from US sociology, which in studies often had the problem that bad characteristics are almost automatically attributed to black people, while good ones are attributed to white people, which is of course complete nonsense. This implicit bias now also exists in many other areas of health research because, for example, an endocrinologist - when a patient reports symptoms - almost automatically attributes these to diabetes, while a rheumatologist will probably explain the same reported symptoms with rheumatic problems. The same thing can also be observed in tumor councils that I have held for years: If there was the possibility of irradiating a tumor, the radiation therapist was always in favor of it, while the surgeon preferred to operate on the patient straight away and the internist preferred chemotherapy. But that is completely understandable, because every doctor simply knows the advantages and disadvantages of a therapy from their own area of expertise better than those from others.

And what do you think as an oncologist and internist?
In such tumor councils I have warned against certain types of chemotherapy simply because they often do more harm than good.

This means you can quickly be seen as a polluter in the relevant specialist community, right?
Secure. But only because many people have not realized that every subject and every discipline needs internal criticism in order to survive and also to be taken seriously. I never tried to destroy my own discipline, but always had exactly the opposite in mind, because internists - like representatives of all other disciplines - can only be trusted if they carefully consider which therapies - including non-specialist ones - are suitable bring the best possible outcome.

What would have to be proven.
That's exactly where the rub lies. Because: Is it even clear which therapies bring the best possible outcome? At first glance that may be the case, but then you quickly come across the cardinal problem of any therapeutic-medical reality and therefore of any clinical study: Almost no one knows the initial risks of the patients to be treated or included in a study. By the way, not in disease management programs either.

That's why it's so difficult to evaluate whether they work or not.
But only because no one had previously thought of recording the initial risks. Therefore, all the results published on DMP are actually of little use, because one can predict from the outset that the patient with a high risk will probably have a worse result than the one with a lower one. The following applies: In such studies, only patients with the same risk may be compared with one another. That's the be-all and end-all, it's actually a pretty simple idea that unfortunately all too often isn't thought of.

The problem of the output load also arises with minimum quantities.
Naturally. The problem of minimum quantities can only be addressed if one is able to create comparable risks. If a small hospital receives an extremely high-risk patient, you can quickly tell whether a good doctor is making the admission. To quote Sauerbruch: “The good doctor differs from the bungler in that the good one recognizes the difficult patient.” On the other hand, this is the case the inexperienced doctor will treat patients with a poor prognosis in the same way as those with a good prognosis and this alone will produce a worse result. This leads to the next bias: A good doctor in a small hospital will sort out the high risks in the interests of the patient and pass them on to a better, often larger one. There are a whole host of factors that are quite logical in themselves, but which hardly anyone thinks about in the current quality debate.

And why don't you think about that?
Many researchers and especially institutions have been seduced by the apparent security that Randomized Controlled Trials, or RCT for short, promise.

Like that?
This is very simple: In an RCT you no longer have to think much about risks because the randomization alone ensures that the risks are equally distributed in the intervention and control groups.

But that also means they are hidden.
That's exactly the problem, because it has the sad side effect that no one actually thinks about the risks as such anymore.

An example?
Let's imagine that we carry out two studies, one in Bonn and one in Rosenheim, for example, just to take two cities of approximately the same size. Both studies are randomized and both treat essentially the same disease. But it may be that the Bonn clinic mainly treated seriously ill patients, and in Rosenheim mainly milder cases, because the serious ones had already all been sent to Munich anyway - by the previously assumed good doctor. Although there was sufficient randomization, the average risk in the Bonn cohort is significantly higher than in the Rosenheim cohort. This means nothing other than that it is extremely difficult or almost impossible to compare the results of both studies. And only because the reliable parameter of the existing, and very easy to determine, initial average risk is missing.

And what if the Bonn clinic – again assuming the good doctor – sends the more difficult cases to Cologne?
Even if the mild and severe ones are treated elsewhere, they will be relatively homogeneous groups of moderate risks. This would make the groups comparable again. But the real problem lies somewhere else: through randomization, the person conducting the study no longer has to worry about the initial risk because they only have the internal evidence - i.e. the evidence that exists within the study - in front of them. The randomization leads him to believe that there is a similar or equal risk distribution in both initial groups, which is not true.

Now, at least in the early benefit assessment, no studies other than RCTs are approved because they have the highest internal evidence class. Point.
IQWiG is right about that. What the institute is wrong about, however, is the assumption that internal evidence is more important than external evidence. In fact, I firmly believe that external evidence is at least as important, if not perhaps more important.

But this opinion is not proven.
This is the problem of error estimation.

What do you mean?
Let's take another simple example: If the engine of a car suddenly stops working or a tire bursts, the driver probably has a problem while driving. He doesn't care at all whether the internal evidence (the engine) is to blame or whether the external evidence (the tires) is wrong. If he wants to continue the journey, he must have both in working condition.

Does that mean applied to studies?
In principle, this is a killing argument for almost any study: if the internal evidence is correct, you can believe or disbelieve the difference found. But if the associated external evidence is not correct, the result found cannot be applied under any circumstances outside the study population. This means that the effect found may be extremely interesting for the usually very narrow study population, but it does not apply to the rest of the world.

This seems to be a big problem that is rarely discussed.

I wonder why? That would mean losing faith in RCT, right?

However, RCTs are now also characterized by their inclusion and exclusion criteria.
True, there are plenty of them. Studies published in the “New England Journal” often come up with 10 to 20 inclusion and exclusion criteria.

Let's start with the exclusion criteria, i.e. everything that is not allowed in a study.
The primary purpose of exclusion criteria is to create a population that is as narrowly defined as possible and is just large enough to be suitable for the desired measurement. If you strictly lo-
gisch thinks, one has to come to the conclusion that all the findings found in this study are only applicable to the population under consideration - but that is nothing new, we all have that with the father of EBM, who has unfortunately now passed away, David Sackett, learned.

If, for example, I wanted to set up a study in the area of diabetic foot syndrome, it would make sense to only use those inclusion and exclusion criteria that have been agreed upon by the specialist societies.
Would it be. However, each person carrying out the study can determine the inclusion and exclusion criteria for their own study, which in some cases makes sense because, for example, comorbidities can be extremely dangerous for patients - so these should be excluded, even if this distorts the results. But this also means that the exclusion criteria are based on the risks and are not 100 percent related to the innovation or therapy being tested.

However, there will certainly be inclusion and exclusion criteria that make the most sense for a specific population that is to be treated with a specific drug or form of therapy.
In a certain sense, one can certainly imagine that the inclusion and exclusion criteria can be standardized. But unfortunately it's not like that. There are a thousand reasons to formulate exclusion criteria, some perhaps due to the fear of the company behind a study that the drug being tested will no longer work at all; or that intensive care patients are excluded entirely because there could be suspicion that the drug being tested would drive the patients to the intensive care unit. This is all nonsense, of course, but it is the reality that can only exist because neither inclusion nor exclusion criteria have to be justified.

Are the intervention and control groups at least the same?
Unfortunately, they are only approximately the same, as is the case with all studies in the so-called “Table No. 1” reveals. I recently dug up a doctoral thesis that I supervised in 2015 and that was “Table No. 1” examined.

Sounds pretty banal.
But it is not. In every randomized study there is this table number 1, which says, for example: Age, group a = 62 years on average. Group b = 62.5. Proportion of men to women 1:1 in group a, in group b 1:1.5. Pre-existing heart attack = 22 % in a, 23 % in b.

That's a good thing, because this table no. 1 describes the comparability of the initial groups.
At least that's the case in 98 and 99 percent of all randomized and published studies, the rest simply forget it. But now my former doctoral student conducted an experiment in which he conducted an expert survey on that same table number 1. All he did was ask the fundamental question: Which criteria should appear in Table No. 1 and which should not?

Now, the inclusions and exclusions.
Only secondarily. First of all, the endpoint of the study must be determined, although a randomized study always only has a single endpoint - the primary endpoint. Of course, you can also examine other secondary endpoints and describe them, but you cannot test them clearly. A randomized study can only do this with the primary endpoint, because that is exactly what it is powered for. It is essential to define this primary endpoint first.

Let's say: death. This is very easy to notice.
Ok, let's take the final end point, death. Only now does the person conducting the study know which study population they need and must then include in Table No. 1 all the factors that can influence this endpoint.

For example, this eliminates common colds because they are never fatal. But a malignant pre-existing condition, if it is permitted at all, has to go in.
In general, if you want to examine heart attacks, for example, everything that could contribute to influencing this end point must be included. At this point I often ask my students: Does it also include whether the patient is Catholic or Protestant?

They will say: No.
True. Nobody can imagine that creed or religious affiliation has to be pure. A student once interjected that when it comes to trichinae, yes. That makes sense, right?

But that just means that everything must be listed in Table 1 that can influence endpoint 1 through a risk factor of whatever kind.
True, but this is often forgotten. Table number 1, which seems so simple, reflects nothing other than the equal distribution of risks and this is one of the most important things in every study.

And then there is the small “p” behind it, which indicates the statistical difference.
If that were the case, that would be good. But I know of many studies in which the “p” was not statistically different in any case. You can quickly use a kind of abacus method and note for each row in Table 1 which side the risk is a little higher or lower, although it is usually never completely identical.

As an example, let's assume an average age of 60.0 in group a to 60.1 in group b.
Everyone would say: That’s the same “p” and so it’s never significant, right?

True.
But only if you only compare one value in a single line. But if you take the trouble to compare all the rows in Table 1, you will find that there are studies with - let's say - eight points in group a and only one in group b. This means nothing other than that the negligible difference accumulates in one line and creates a clearly asymmetrical relationship when all risk factors are taken together. This shows how important it is to examine studies closely, and even this seemingly simple Table 1 ultimately reveals why a study produced a significant result or not.

This leads to the question of whether RCTs – at least as they are currently often done – can be the basis of EBM?
But now it's getting political. Because you can't go that far. I would like to restrict this question somewhat and say: They can no longer be the sole and almost unquestioned basis! One shouldn't throw the baby out with the bathwater, because randomized controlled trials are a necessary, but not sufficient, condition for deriving recommendations for clinical guidelines or for benefit assessment.

In the early benefit assessment, if you follow your argument, the results should only be applied to those populations that were included?
Not even that. Only those patients who don't have a strong preference are included in randomized studies.

Preference for what or who?
To everything and everyone. Let's take a study of keyhole surgery versus large incision. A patient might say: “My neighbor said to me, don’t have the big incision made, it looks stupid if you wear a bikini.” Then that same patient comes into the clinic and the doctor says: “We’ll do it Now a very careful operation with the large incision is better than a less clear operation with the keyhole.”

The doctor always knows exactly what he wants to “sell”, right?
Secure. The doctors can get it worked out so that the patients sign everything they need. But that means nothing more than that the patient is being persuaded to do something by the doctor, even though he may have a completely different preference. Then the operation takes place and the whole thing goes wrong. What's the point then? About the surgical method? Or perhaps the doctor's preference or value judgment?
What happens with that in the study setting?
You are usually dealing with patient cohorts that are almost always similarly distributed: 80 to 20. 80 percent of all patients would like the keyhole operation because it is more socially acceptable than the large incision, while 20 percent prefer the large operation because it seems safer to them. Now the person conducting the study neatly randomizes the patients into two groups and dutifully puts 80 percent in each group with a preference for keyhole surgery and 20 percent with a large incision. Now the operation is carried out, 100 percent in group 1 with a keyhole and 100 percent in group 2 with a conventional large incision.

And what happens?
Something completely logical. In group 1, with an 80 percent preference for keyhole surgery, they received exactly the therapy they wanted from the start. But in group 2, only 20 percent of patients get the large incision that they actually preferred. This means that the treatments were clearly randomized in the study, but not the patient preferences.

Can't anyone imagine that this psychological event has an influence on the outcome?
There are enough papers, including mine, that prove this exactly. However, no textbook yet explains why this is so. Not even Sackett's.

Could this trigger a paradigm shift in the almost monotheistic view of RCT?
I do not believe that. The organizations involved, especially IQWiG and the G-BA, stick far too closely to the classic RCT. As I said: rightly so, since RCT is the study approach that allows for the least possible fraud.

What would be your advice?
Anyone who has a good RCT has acquired no more, but also no less, justification to be able to carry out a pragmatic controlled trial based on it. Because the RCT - at least if the study was well designed - has confirmed nothing other than the fact that the tested principle could work under ideal conditions.

But then you have to ask the second, equally important or perhaps even more important question: Does the principle also work under everyday conditions?
Unfortunately, it's the case that we didn't want to know that until now. My credo is therefore that we can only regard the results that we have achieved in good randomized studies as a hypothesis for reality. Because in my opinion, if something works under idealized conditions, that doesn't mean that it will also work under everyday conditions: But: With RCT we at least have a legitimate hypothesis to assume that it could work just as well under everyday conditions.

Can an effect that was found under ideal conditions also be tested under everyday conditions? When all the confounders and disruptive factors cannot be removed?
This is exactly what the so-called Pragmatic Controlled Trial, or PCT for short, is all about: While a randomized study is characterized by the definition of inclusion and exclusion criteria, a Pragmatic Controlled Trial only has inclusion criteria. This is actually quite logical, because under everyday conditions there are no exclusions.

Are such pragmatic controlled trials carried out?
Hand on heart: No company wants to go through the trouble of carrying out a PCT after an RCT, always with the risk of finding out that the small effect found in the RCT is suddenly no longer detectable. That's why I thought of a little trick.

Which one then?
One that makes up the difference between a one-tailed and two-tailed test.

Like that?
Most published studies are one-sided. This means: A meaningful comparison only takes into account one of the two possible comparisons. An example: I am interested in whether my new therapy only appears to be or is actually better than placebo. But I don't care whether it only appears or is actually worse than placebo. That's why I only examine superiority, not inferiority.

Nobody who pays money for a study wants to know the latter.
Exactly. With a two-sided test, the chance of confirming a significant result is halved compared to a one-sided test, but the risk of making a first-order error also increases.

The majority of RCTs will therefore be one-sided.
Almost all of them are one-sided. But that is also quite reasonable.

But now you have published a letter to the editor in the “New England Journal of Medicine” because there was a paper in this highly ranked journal in which it was published that a new drug against diabetes was found in a so-called non-inferiority test: that it is not inferior to placebo.
But that's nonsense, isn't it? A non-inferiority test only makes sense when comparing two effective principles. For example, if you have two antihypertensive drugs that are almost equally effective and have been proven to lower blood pressure, you can prove that both work compared to placebo. But since one antihypertensive drug was twice as expensive as the other, a non-inferiority test was carried out in which it was defined that the cheaper one could be a little worse than the more expensive one if it met the limit of non-inferiority does not fall below.

Sounds somehow as sensible as short selling stocks.
Something like that, because all these people seem to be hoping that no one will find out about them. With these non-inferiority tests they want to prove that the new product is better than placebo by announcing: “We have a new drug. It's not inferior to placebo.” The total nonsense only becomes really clear when you consider that the authors write this nonsense, the reviewers tolerate this nonsense, the editor publishes this nonsense and the NEJM readers publish this nonsense also use it on their patients. America first, right?

It's good that, according to "WIdOmonitor", only 15 percent of German doctors actually use English-language journals. But how do such publications even get into such journals?
I'm not saying that here, but you can read about it in "Nature". Professor David B. Allison* described very well how cheating occurs in major journals.

How much does it cost to publish in such highly ranked journals?
Nothing. But trying to retract the paper after submission can cost up to $10,000 at some journals. It's all literally in the Allison paper. At these prices you're already thinking about spending $10,000 of your scarce research money just to get rid of your own junk. That's crazy.

And what about open access?
It doesn't cost anything, but someone without money can't get in. The companies behind the studies pay for this through advertisements, but mostly through special prints.

It's worth it?
And how. I'm friends with a senior editor of a global journal who I once asked why they publish such nonsense. His answer: "If a company orders special prints for $2.5 million, you'll think about it..."

Prof. Porzsolt, thank you for the interview. <<

The interview was conducted by MVF editor-in-chief Peter Stegmaier.