IQWiG worked for almost two years together with national and international experts on a method for assessing cost-benefit ratios. An essential part of this method is the analysis of the efficiency frontier, which - according to the then director of the institute, Prof. Dr. Peter T. Sawicki - “the method most suitable for the German framework conditions”. But this was exactly what met with harsh criticism when the method paper was presented in October 2009: 29 leading German health economists rejected the IQWiG proposal as “scientifically untenable” and “not normatively justified”. “Nowhere in the method paper is an algorithm described how a sum parameter is calculated through which the weighting of benefits and harms is carried out,” criticized Prof. Dr. med. Jürgen Fritze, senior association doctor at the PKV Association. The institute addressed this shortcoming with two general contracts in the form of pilot studies, with a “Conjoint Analysis” (CA) in the indication Hepatitis C and an “Analytic Hierarchy Process” (AHP) in the indication Major Depression - unfortunately in two different indication areas, so that a direct comparison is not possible. The goal, which now seems within reach: to scientifically describe procedures that make it possible to describe a cardinal benefit value, with the help of which all conceivable therapy alternatives in an indication area can be compared across all relevant endpoints. With the head of the “Conjoint Analysis” pilot study, Prof. Dr. Axel Mühlbacher (Neubrandenburg University, IGM Institute of Health Economics and Medical Management and co-founder of the Society for Empirical Consulting mbH), spoke about “Monitor Health Services Research”.

>> Prof. Mühlbacher, the reason why IQWiG followed a German path in the evaluation of cost-benefit ratios and the necessary analysis of the efficiency frontier, which was often dismissed as a special German path, was that Institute did not simply want to adopt the QALY system that was particularly prevalent in the Anglo-Saxon area.
At that time, the QALY was rejected primarily for ethical reasons. However, the concept is also questioned for methodological reasons. The advantages of the QALY concept are, on the one hand, its superficial plausibility and, on the other hand, the broad international wealth of experience. Health technologies are evaluated under the assumption that patient benefits can be reflected in length of life and quality of life. Evaluation criteria that are taken into account in clinical studies are left out. More specifically: studies that are used to set prices with the help of cost-benefit assessment have no specific connection to studies that are carried out for approval, i.e. the documentation of the benefit-risk or benefit-harm ratio. Both study approaches are also based on different academic disciplines. The economists and psychologists document the benefits of a health technology through the quality of life or the quality of life-adjusted life years, and the doctors specify the medical benefits using patient-relevant endpoints or clinical effect measures. The necessary study designs run in parallel and do not build on each other.

This makes the efficiency frontier different.
True. The efficiency limit, if it has a theoretical basis, would create an instrument that takes the results of clinical studies into account or transfers patient-relevant endpoints into a benefit assessment. If you want to base the cost-benefit assessment on the results of evidence-based medicine, that is, to design medical and economic studies consistently, then the logic of the efficiency frontier is a forward-looking approach.

Let's assume that a clinical trial examines the benefit-risk ratio of a particular technology...
...then it would be theoretically conceivable with the efficiency frontier to carry out an economic benefit assessment based on the clinical effects and side effects, i.e. to determine the cost-benefit ratio. Once approved, the decision maker could discuss the pricing of health technologies based on evidence-based medicine.

As is well known, the introduction of efficiency limits in 2009 was not exactly a success. Nevertheless, it seems to be a rational, reliable and comprehensible way in which benefit-cost assessments could be carried out.
The efficiency limit could then be used rationally, reliably and comprehensibly if the method of benefit assessment meets scientific standards. The critics of the method papers rightly saw a need for additional information. However, IQWiG must be given credit for the fact that the questions we are dealing with today were only raised by this method paper.

The first question was certainly which endpoints can actually be used to calculate efficiency limits.
Yes. At the beginning of every evaluation you have to decide on certain evaluation criteria. In this case, patient-relevant endpoints, which is certainly not a trivial question and has a significant influence on the result. Even more important, however, is the question of how these endpoints will ultimately be weighted when assessing the benefit. The big gap in the methods paper was the aggregation of different endpoints - i.e. the calculation of the overall benefit. This overall benefit should also underlie the G-BA’s decision. The crucial question is: Is there a rationale that relates the clinical effects under consideration to the patient benefits they generate? And not just per endpoint, but also for several endpoints, which can contradict each other.

Overall benefit means you take all effect parameters across all end points...
... and uses this to generate a benefit value, with the help of which all conceivable treatment alternatives in an indication area are compared across all relevant endpoints. This could then be used to carry out a cost-benefit assessment. So we document the overall benefit based on the underlying clinical evidence.

Is a comparison size already being discussed here that enables a comparison across all indications?
Not yet. In Germany, the patient benefit only has to be compared in a specific indication area. This simplified the task for the pilot projects. In this respect, the measure of overall benefit did not require a cross-indication benefit dimension, but only a dimensionless benefit score or index.

It was precisely against this indication-specific assessment that the economists ran amok.
Understandable. From the economist's perspective, a cost-benefit assessment is carried out when information about the optimal allocation is to be generated. If you now use the evaluation framework of efficiency limits to make an indication-specific decision, then that is a step in the right direction. From an economic perspective, however, this is not the final step.

The final step would be the cross-indication comparison.
The current interpretation of SGB V does not allow this step. Detached from the political debate, our methodological approach makes cross-indication comparison possible, i.e. an optimal decision can be made about all interventions in the healthcare system. However, this requires that one agree on a benefit dimension or a measure of the overall benefit, which can be measured across all alternatives in the healthcare system.

That is possible?
Absolutely. Well-known economists have long been discussing monetary equivalents as a solution. If this goes too far for you, you could possibly fall back on the equivalent of healthy years of life. I am optimistic that in the future we will also discuss methodological approaches in the context of preference measurement that allow a cross-indication evaluation of technologies with the efficiency limit. But this will only happen if the decision-makers demand it. In addition, this would further advance the open question of extrapolation within the framework of the efficient frontier.

Should IQWiG have known this before publishing the methods paper or can the discussion actually only be conducted on the basis of this methods paper?
To be fair: the latter. Publishing a first methods paper outside the mainstream was a very courageous act. Especially because there are national and international experts who have excellent knowledge of traditional methods, while there was almost no practical experience with the efficiency frontier. The efficiency frontier provides a framework for interesting methods of benefit assessment, such as discrete choice analysis or best-worst scaling. I am convinced that we will have very exciting alternative approaches to benefit measurement in the next four to five years.

Why only in the next four to five years?
On the one hand, it would be necessary to take a closer look at or discuss the economic foundations of the efficiency frontier. When it comes to benefit assessment, it is now important that we better understand the transfer of clinical evidence into the economic models of benefit assessment. On the other hand, it certainly takes time to critically examine one or another suggestion for determining an endpoint-based benefit value. We have only just completed the pilot project.

This means the discussion can begin again, as it did when the efficiency frontier was presented.
Critical discussions are certainly not always nice, but they are necessary in order to be able to continue on the path successfully. To put it somewhat provocatively, the QALY experts had 25 years to solve their theoretical and methodological problems. So let's give the efficiency frontier and the multi-attribute benefit analysis a little time.

However, as long as the model is not used in practice...
... I know. Until then, the QALY supporters still have the more powerful arguments. But I firmly believe that, assuming the IQWiG and the G-BA are interested, the problem of the efficiency frontier discussed above can be solved with the help of preference studies. We tried to show this in the IQWiG pilot project.

If IQWIG published the pilot studies.
I think publication and discussion are important. I simply cannot imagine how we in Germany would manage without a cost-benefit assessment. The AMNOG has now taken the pressure off. But in the medium term, the cost-benefit assessment will also be discussed again in Germany. However, that also depends on future political constellations. After the next federal election, it is certainly conceivable that, in addition to the financing debate, the cost-benefit assessment could also come back into focus.

Now that the first existing market has been called up, the topic of cost-benefit assessment will come back into focus for this reason alone.
This is conceivable. With the first benefit assessment for drugs from the so-called existing market that are approved for the treatment of type 2 diabetes mellitus, the active ingredients and the combinations of active ingredients need to be compared. Here too, the overall benefit of the available alternatives must be determined.

Are we still a long way from deriving all the information necessary for a cost-benefit assessment from the dossier?
I do not think so. In addition to naming the appropriate comparative therapy and the therapeutically significant additional benefit, the dossier already contains the costs of the therapy for the statutory health insurance. The advantage would be that a functioning efficiency limit allows a ranking, during which, for example, the QALY system follows the “winner takes it all” principle. What is still missing is a method that aggregates an overall benefit for the patient from the different endpoints considered.

That's a terrible term.
We haven't found anyone better yet. In the future, I would like to call it an endpoint-based utility score or index. But whatever you call this aggregation of endpoints into a uniform key figure, it is important for decision-making.

What's at the beginning?
The question of identifying endpoints. It makes sense that we should strengthen the role of patients here. Why not carry out a preference study before the clinical study phase to clarify the relevance of individual evaluation criteria from the patient's perspective.

What makes these endpoints relevant to patients?
These preference studies. By surveying patient preferences, we find out what priorities patients really have. We can also carry out the same study with experts and thus show differences in the assessment of individual endpoints. This information could bring another perspective - namely that of patients - into the scientific discussion of the respective committees. The basis for patient benefit should also be the evaluation criteria that the patient would use for the decision. If there is no patient-relevant evaluation criterion, then one cannot speak of patient orientation or patient benefit.

Identifying endpoints is very critical. Anyone who conducts studies for the AMNOG process today will only focus on those endpoints that are already well documented and widely recognized in clinical studies.
That is an important point. This is exactly where we had problems at the beginning. The IQWiG call for proposals said: identification, weighting and prioritization. If you look at the task very pragmatically, it was not about identifying a new endpoint, but rather taking those that are already well documented in clinical studies. So it was more about weighting different endpoints and ultimately aggregating them into a uniform measure of overall benefit. This would then make prioritization possible.

Which applies to the benefit-cost assessment as well as to the early benefit assessment.
Absolutely. It is incomprehensible to me why the benefits of a technology should be derived differently in the early benefit assessment than when calculating an efficiency frontier or in the cost-benefit assessment.

Anyone who looks at the previous decisions of the early benefit assessment will see that actually all patient-relevant endpoints that were not validated with hard endpoints on a study basis were negated. The question is: How important is it to the system what the patient reflects back to it? Or is only what the system deems important for whatever reason important?
The basis of the decision is always evidence-based medicine. Since I am not a doctor, I do not want to make any judgments about the validity of endpoints or surrogate parameters. For me as an economist, however, the following question arises: Are patients prepared to choose between different effects of a health technology? Specifically: Are patients willing to take greater risks in order to achieve greater effectiveness? Or: Would patients forego years of healthy life in order to reduce the risk of harm from side effects? Both questions are relevant, both for approval and price setting.

These are tough decisions.
I am aware of that. Likewise, the patients' priorities cannot always be supported by experts. But it would be wrong if we did not make these priorities of those affected when it comes to approval and price setting the subject of the consultation. It should be clear that both patients and experts ultimately make subjective decisions. If patients prefer certain properties, this suggests that a technology with these properties also delivers a higher value proposition, i.e. patient benefit.

This is currently not shown at IQWiG and G-BA.
If the current methods cannot reflect this value proposition, we have to think about whether there is something wrong with the patient or with the method. I'm guessing the method.

Then the federal association will probably argue: You are welcome to do that as long as you pay for it yourself and not through the solidarity system.
This is an exciting and legitimate question. It's exciting because as a citizen I delegate my decision-making rights to the responsible decision-making bodies. The reason lies in the information asymmetries between the person concerned and the respective expert. However, I delegate under the assumption that my well-being as a patient and insured person determines the maxim for action or the value system of the decision-makers. The objection is justified because this issue is being hotly debated and population preferences should now play a role in the decision.

There is a difference whether you think about it as an insured person or as a patient, especially as a chronic person.
Exactly. As an insured person, I want the optimal use of scarce resources, and as a patient, I want the maximum benefit from the therapy. Both requirements require systematic benefit assessment or cost-benefit assessment. But I know of no argument why the preferences of the affected patients should not be taken into account in the decision. We should not forget that we are talking about the preferences of a representative population, not the preferences of individual people.

You also mention the legitimacy of the G-BA here. There is certainly a difference whether legitimacy is presented on the basis of preference measurements of a patient group or only through commission from the federal government.
I firmly assume that the G-BA acts on behalf of patients and insured persons. Isn't that also the government's goal? I therefore also assume that interest in studies about patient preferences will increase in the future. Especially when these studies solve some essential problems of the efficiency limit or benefit assessment and thus the patient perspective can be taken into account in decisions about approval and price setting.

Let’s get back to determining patient benefit. Once the endpoints have been identified, they must also be weighted.
It is obvious that when three endpoints are taken into account, each endpoint does not necessarily have to be included in a third of the assessment. Some endpoints may be more important, some less important. The relative importance should be decisive for the evaluation. To do this, you must be able to answer the question of how relevant an endpoint is in comparison to the other endpoints. But that also means: We need a method that provides us with a relative weight for each of the end points.

You can imagine that there were certainly endless discussions here.
That wasn't so bad, because there are scientifically more or less valid instruments for this. The same methods are used, for example, in environmental economics and transport science to measure the preferences and benefits of environmental measures and infrastructure decisions. A health technology, a policy reform or a medical service can be broken down into its essential characteristics. Similar to our everyday decision-making processes, these characteristics are crucial for choosing an alternative. In experiments we can reproduce these decisions and evaluate them statistically. Each alternative can therefore be evaluated based on the realization of different characteristics. By aggregating these partial benefits, a utility value is obtained that allows different alternatives to be evaluated.

That still sounds very complicated...
... but in the end it's very simple. Let's take a decision about an automobile as an example. Characteristics of a car can include engine power, number of seats and equipment. The engine power levels can be 100 HP, 200 HP and 400 HP. Number of seats 2, 4 and 6, the equipment can be the car radio, C-changer or a complete audio system with telephone system. If a study succeeds in assigning a value to these characteristics, then an overall benefit can be assigned to every conceivable combination.

Does this also work in the area of health?
Secure. We know the endpoints from the clinical studies. To put it bluntly, these are nothing other than the properties of a product. A clinical intervention achieves certain effects and thus product-specific characteristics are realized. Now all you have to do is design different product variants across realistic ranges and let people choose between the alternatives. For me, this election is the most important expression of opinion and the only one that can be realistically measured. From this choice you can draw conclusions about your preferences and therefore also determine the benefits.
How can these values be determined?
We conduct surveys with the affected population groups. An essential component is asking the test subjects to choose between different scenarios or alternatives. So we describe a drug based on endpoints, such as extending life, reducing pain, duration of action and side effects. With the help of different manifestations of these characteristics, we can offer the participant of a survey two or more alternatives to choose from. By making a choice, the patient shows us which of these alternatives he or she prefers. The basis of the preference is the perceived benefit from these alternatives. This is determined through the statistical evaluation of the voting decisions based on the preferences in relation to the alternatives available to choose from. The overall benefit is derived from the calculated partial benefits of the individual properties or the underlying characteristics. The only important thing is that these weighting factors are determined using a method that can actually determine preferences.

Why is the voting decision so important?
Let's take the five-box Likert scale. 1 means not important and 5 means very important. Negotiate with your son or daughter about a new car. To do this, write down all possible criteria from horsepower, design to equipment. What do you think he or she will tick? Design: important 5. PS: important 5. Equipment: important 5. But will there be a Porsche in front of the house afterwards? Not because there are budget restrictions and everyone is used to allowing so-called trade-offs. Because in life you always have to choose between certain characteristics. Or compare two characteristics using a scale from 1 to 9. In this comparison, you must indicate how much more you prefer the engine performance to the equipment.

Can we really do that?
No. We can rank things and properties, but in reality it is not observed that we assign a specific value to properties - not even in comparison to an alternative. Can you answer the question, on a scale of 1 to 9, how much value do you assign to a 10 horsepower increase in your car compared to an additional CD changer? The study participant would have to answer a similar question in an Analytic Hierarchy Process, or AHP for short. With the choice-format conjoint analysis or a discrete choice experiment, however, only the choice decisions are modeled.

How is that supposed to work?
A simplified example: A certain active ingredient extends life by 6 months, 8 months or 10 months - these are the properties. You have to inject the active ingredient once, twice or three times a week. The side effects include severe or mild headaches and fatigue. In alternative 1, the active ingredient extends life by 10 months, the side effects are severe headaches and fatigue and injections have to be done twice a week. Alternative 2 involves injections once a week, the patient only has mild headaches and fatigue, but the effectiveness is only 6 months. The only thing we ask from a participant: Please tell us, which alternative would you prefer? If he prefers alternative 1, then we have an initial indication that the life extension criterion has a greater influence on patient benefit than the criteria of use and side effects. If he answers with alternative 2, then we have the information that this respondent is willing to accept a shorter life span in order to reduce side effects or a lower injection rate. The more questions we ask like this, the more precise the statistical analysis becomes.

The key question will be: Do you do this with the Analytic Hierarchy Process or with Conjoint Analysis?
The few proponents of the AHP argue that the AHP is cognitively easier for patients to understand. In addition, the study should be able to be carried out with fewer people, meaning that the effort should be lower and the instrument should therefore be cheaper.

That seems like a killer argument.
I wouldn't want to agree with these arguments. Let's take it apart. The AHP question format is ostensibly simpler - that's true. A participant has to compare two properties, i.e. prolongation of life with the side effect of headache. This comparison does not include whether it is 10 years or just 10 months of life extension or debilitating headache or mild headache. Not very realistic! In addition, he has to indicate how much he prefers a characteristic, on a scale from 1 to 9. “I prefer prolongation of life 9 times more than headache” - That is even less realistic. In addition, these comparisons are structured hierarchically. The entire decision-making situation never becomes obvious to the patient being interviewed. This can lead to distortions in the evaluation. Furthermore, it is discussed in the literature that so-called “rank reversals” occur. This means that under certain circumstances the ranking of the best alternatives changes. However, this does not happen because the answers of the people surveyed change, but rather because of the evaluation logic. This in turn poses the risk of incorrect judgments.

Why do AHP require fewer test subjects?
Because there is simply no information about it in the literature. An internal consistency check is carried out. However, the threshold value was set arbitrarily by the inventor of the method, the mathematician Thomas L. Saaty, without further explanation.

If that is the case, the AHP instrument cannot be used as part of the efficiency frontier analysis.
Whether yes or no will be shown by the discussion over the next few months. Today I wouldn't recommend it. What bothers me is the lack of theoretical classification and even more so the unrealistic question format.

Whether CA or AHP wins the race, in both approaches the patient is at least finally the focus.
This is the idea that drove both pilot project working groups. Ultimately, care must be based on the preferences of those affected. Once you know your preferences, it would be foolish not to take them into account. But we have already taken the first steps on this path.

Thank you for the talk. <<
The interview was conducted by MVF editor-in-chief Peter Stegmaier