Patient-weighted endpoints: Conjoint Analysis (CA) versus Analytic Hierarchy Process (AHP)

The so-called multi-criteria decision analysis (MCEA) can be used to support complex technology assessments. Multi-criteria decision analysis procedures have their origins in decision theory or operations research and are used there to make decisions under uncertainty. They assist decision makers in evaluating a finite number of alternative health interventions using a finite number of performance criteria. In addition to conjoint analysis, the elimination and choice translating reality method (ELECTRE) and the multiattribute utility theory (MAUT) (Lootsma/Schujit 1997), the most widely used multi-criteria decision analysis tools include the Analytic Hierarchy Process (AHP).

>> These instruments differ mainly in their approaches to collecting weights for attributes of the technologies under study and present different statistical models in structuring the preferences of decision makers regarding the available alternatives. In the area of application we are considering, the patients surveyed would primarily be the decision-makers and the respective endpoints within the clinical picture under consideration would be the performance criteria of the health interventions used. According to the theory, the added value of these instruments lies in the support of the clinical and (health) economic evaluation of health technologies, which are incorporated into relevant decision-making as a multifactorial process.
Nevertheless, the applicability and validity of multi-criteria decision analysis is questioned. For example, different multi-criteria decision analysis methods produce different results when applied to the same decision problem (Zanakis et al. 1998). Furthermore, their predictive validity often remains speculative. Experimental comparisons made concluded that each multi-criteria decision analysis method has its own advantages and disadvantages. In this paper we will briefly examine these for the two established instruments CA and AHP, which are also being used more and more in the health sector (de Bekker-Grob et al. 2010; Wallenius et al. 2008) and in the survey of patient preferences and contrast.
Mathematical-theoretical foundations of the AHP
AHP as a compositional process
The basic assumption of the AHP is that the calculated right eigenvector of the underlying matrix A approximately corresponds to the vector of the relative weights. It is therefore a compositional method that follows a weighted additive benefit model (Fig. 1).
Right eigenvector
The AHP offers an approach that is suitable for estimating the weights of individual decision criteria and for prioritizing available alternatives. This quantification is derived from a matrix of pairwise comparisons of two criteria or alternatives. The pairwise comparison of the decision criteria shows the relative importance of these criteria in achieving the goal, the comparison of the alternatives shows the relative preference for the alternatives considered depending on a corresponding decision criterion. Both - relative importance of the criteria and relative preference of the alternatives - are based on an ordinal scale that ranges from 1 for equal importance or preference to 9 for extreme importance or preference within a pairwise comparison. The individual numerical values of the ordinal scale represent the following values of relative importance or preference:
• 1 for indifferent or equally important statements (criteria),
• 3 for something more important,
• 5 for much more important,
• 7 for much more important and
• 9 for extremely much more important.
• 2, 4, 6 and 8 are each used as intermediate values.
The criteria considered within the pairwise comparison can be reversed if the direction of importance is reversed. A matrix of pairwise comparisons across n criteria is created, with A = (aij):
Both reflexivity (aij = 1 on the diagonal) and reciprocity (aij = 1/aji) are assumed in this matrix. For this reason, only the values across the diagonal need to be entered. This means that for n criteria only n(n-1)/2 distinct pairwise comparisons are necessary. If groups with x participants are surveyed as part of the AHP (x > 1), the individual cells of the matrix can be filled with aggregated pairwise comparisons by forming the geometric mean of the individual individual evaluations:
aij = (aij(1) * aij(2) * … * aij(x))(1/x) for all i,j = 1, 2, …, n.

If all pairwise comparisons are in perfect consistency with the remaining comparisons, then transitivity applies, formally expressed as follows:
aik * akj = aij for all i, j, k = 1, 2, …, n.

The AHP provides a consistency measure to point out possible inconsistencies when comparing criteria. This measure is based on the maximum eigenvalue λmax of the matrix with the pairwise comparisons. This eigenvalue is always greater than or equal to n for positive, reciprocal matrices and equal to n if the matrix of pairwise comparisons is consistent. By normalizing with the matrix size, Saaty defines the so-called consistency index CI in the following way:
CI = (λmax – 1)/(n – 1).

Saaty (1989) randomly generates 500 matrices for a matrix of size n in a randomization process and calculates the average consistency index CI from these matrices, which he calls the random index (RI). The consistency ratio (CR), in turn, indicates how far the pairwise ratings deviate from a completely random matrix of pairwise comparisons and is defined as the ratio between the consistency index and the random index: CR = CI/RI
According to Saaty, 10% of inconsistency of the random matrix are allowed. This means that a consistency ratio value of less than 10% (CR < 0.1) can be tolerated or accepted. The same eigenvalue of the matrix with the pairwise comparisons is now used to estimate the weights of the criteria:
wi = (Σn j=1 aij * wj) / λmax for all I = 1, 2, …, n.

This eigenvector method can be interpreted as a simple averaging process in which the final weights represent the averages of all possible comparisons of the importance of the criteria to be evaluated.

CA as a decompositional process
The CA is based on Lancaster's theory of value (Lancaster 1966). The utility of a “good” is derived from its properties and not from the “good” itself. It therefore follows the theory of stochastic utility (random utility theory), according to which the principle of utility maximization applies: Individuals choose the alternative with the expected highest utility. If the information is incomplete, the benefit cannot be fully assessed. It is composed additively of the observable deterministic component (e.g. features/attributes of the technology, personal characteristics) in the form of a linear utility function (estimation of the weighting of the attributes) and of an unobservable stochastic (random) component. Depending on the distribution assumptions, the estimate is made using a probit or logit model. It is therefore a decompositional method that follows an additive partial value model (Table 1).

Logistic regression
In the discrete choice variant, which is the primary focus here, the conjoint analysis follows a (logistic) regression model, according to which the relationship applies to the benefit to be determined:
Uij = Vij + Ɛij
(Uij = utility, Ɛij = stochastic unobservable component (including subconscious, physical condition, state of mind), Vij = deterministic component (characteristics of the good/performance and the individual), X = characteristics (attributes) of the good/ the performance j; z = characteristics of the individual i; β = measure of the strength of the weighting of the characteristics (attributes) of the good/service j and γ = measure of the strength of the weighting of the characteristics of the individual i.

The equations Vij = V (Xij, zij) and Vij = βXij + γzij then relate the deterministic component to the attributes of the good and the characteristics of the individual.

The relative importance (weighting) of an attribute (here endpoint) is then estimated as part of a logistic regression according to the following equations for the respective deterministic components, whereby the β coefficients provide the endpoint weights:
(i) V (A) = α + β1X1A + ... + βnXnA
(ii) V (B) = α + β1X1B + ... + βnXnB
and
(iii) lnp(A)/p(B) = β1(X1A - X1B) + ... + βn(XnA -XnB)

Theoretical positioning: economic utility theory (CA) versus descriptive theory (AHP)
Although both methods are based on the axioms of utility theory, the AHP does not make restrictive assumptions about the consistency and transitivity of preference expressions due to the difficulties of precise preference assessment by users. This means that even partially irrational decisions are not identified as wrong decisions because of the less strict assumptions regarding the preference structure. The AHP method therefore does not meet essential requirements of economic utility theory and is not considered to be well-founded in utility theory. Saaty (1990) describes the theory underlying the AHP as a descriptive theory - to be distinguished from the normative utility theory.

Methodical quality criteria CA versus AHP
When it comes to the methodological quality criteria established from social empiricism, psychometrics and biometrics, such as face validity, internal validity, convergent validity, predictive validity and model fit, the two methods perform differently in the respective studies, but this does not result in a trend advantage for one method. Only in various dominance tests, in which the agreement of the result is measured in relation to the alternative previously described as dominant by the respondents, does the CA seem to be somewhat inferior. CA can therefore lead to more contradictory results based on a priori assumptions. Dominance tests are a criterion for predictive validity. Two studies are cited as examples from the existing literature (Mulye 1998; Scholl et al. 2005). However, in his work, Triantafyllou (2000) defines two criteria for evaluating methodological quality: 1. Accurate methods in multidimensional decision-making should also provide successful solutions in one-dimensional questions. In most cases, this is fulfilled by AHP (as a weighted sum model). 2. The order of alternatives towards the best should not be changed when one alternative (not the best) is replaced by another, keeping the attribute weights unchanged.
However, the phenomenon of rank reversal occurs here with AHP, i.e. non-compliance with exactly this second quality criterion. The reason for the rank reversal is the determination of the relative weights using the eigenvalue method. Although the AHP method is based on the axioms of utility theory, the restrictive assumptions of consistency and transitivity of preference expressions are omitted due to the difficulties of precise preference assessment by users. Because of the less strict assumptions regarding the preference structure, even partially irrational decisions are not identified as wrong decisions (Saaty & Vargas 2001). The AHP method therefore does not meet essential requirements of economic utility theory and is not considered to be well-founded in utility theory. However, behavioral economics can also counter this argument by saying that the utility theory based on the assumptions of Homo economicus of individual behavior that maximizes utility and follows economic rationality according to the standard model of economics does not correspond to real human behavior in economic decisions (Tversky/Kahneman 1974).
In relation to rationality as a complex behavioral theory, Nobel Prize winner McFadden (1998) deduces, through appropriate experiments specifically for the aspects of preferences, perceptions and process, that most cognitive anomalies arise from errors in perception that are related to how information is stored, retrieved and are processed or from errors in the decision-making process itself, which lead to the formulation of - from a cognitive perspective - inconsistent decision-making problems, at least if one perceives rationality in its narrow definition based on utility theory. Shafir and LeBoef (2002) also show that individuals often violate rationality assumptions about logical conclusions, judgments and choices in their decisions that deviate from the normative behavioral model. They support the criticism of rationality as an axiomatic form of behavior that is now emerging in the social sciences.
Cognitive overload (CA) versus unreal decision situations (AHP)
Following a fictitious example with five attributes (N=5) and three characteristics or levels (L=3) for a good to be evaluated with alternatives, the mathematical foundations of both methods described above for the AHP result in n (n‑1 )/2 = 10 paired comparisons (trade-offs) and in the CA 5³ = 125 scenarios, which is a simplistic so-called orthogonal design, according to which the assumption is that there are no interaction effects between the attributes and therefore no interaction effects need to be measured with the consequence that each expression of an attribute occurs with one expression of another attribute in a scenario, resulting in 12 scenarios per patient surveyed. If the trade-offs within the scenarios are taken into account, the cognitive challenge and even excessive demands of CA appear self-evident compared to AHP. On the other hand, the reality of care is depicted much better through such scenarios than through abstract pairwise comparisons in the AHP. Figure 2 on the pairwise survey in the AHP and Figure 3 on the preferred scenarios in the context of a CA, both relating to patient-side outcome weightings for the disease hepatitis C, are intended to illustrate the respective challenges.
Comparative studies CA versus AHP
Ijzerman et al (2012) compare AHP and CA when evaluating alternatives in stroke rehabilitation. In the weightings of the considered attributes of these alternatives - whereby for methodological reasons the attributes of one method have to be combined for the other - they come to the conclusion that there is agreement between the respective weights and comparable mean standard deviations for three attributes (AHP versus CA: clinical Outcome 0.30 versus 0.33; ease of use 0.16 with both methods, complications 0.20 versus 0.19), but deviations in two attributes (AHP versus CA: cosmetics and comfort 0.28 versus 0.05; Impact of treatment operationalized as duration and invasiveness 0.06 versus 0.28). The comparison also shows differences in the resource consumption and cognitive demands of the respondents. CA is more complex and complicated.
Other comparisons, such as the study by Ijzerman et al. from 2008 to survey preferences in patients with neurological disorders, show relatively similar attribute weights, although non-medical factors such as social acceptance and user-friendliness of the respective supportive treatments seem to play a major role. In this study, weights differed between patients and clinical experts interviewed regardless of the method used. A phenomenon that is also encountered in other indications such as depression treatment (Hummel et al. 2012; Danner et al. 2011). In another study (Benaïm et al. 2010), the authors used CA and AHP to weight the 6 items of the London Handicap Scale, a disability assessment instrument that is often used in stroke prevention. With both methods, physical independence is given the highest weight, but differences emerge in the order of the other items. When using the so-called “10 binary choices” method according to Schoenemaker & Waid (1982), according to which 10 random fictitious patient profile pairs with similar sum scores are selected and the concordance between worse profiles with AHP & CA results is compared (a kappa value > 0.40 leads to a good agreement, a kappa value ≤ 0 indicates a very poor agreement), good agreements were achieved in CA in 3 out of 10 cases, in 1 a moderate agreement and in 6 a poor agreement, whereas in the AHP in 6 of A good agreement was achieved in 10 cases and a moderate and a poor agreement in 2 each.
Usability of CA and AHP for IQWiG
In their discussion paper on the comparison of methods when weighting multiple patient-relevant outcomes, Neidhardt et al. (2012) a catalog of criteria. This is based on (i) the legal requirements, (ii) the requirements of the IQWiG methodology (efficiency limit concept) and (iii) the general requirements for the scientific approach. The authors include transparency and an internationally established health economics process under the legal requirements. The requirements from the IQWiG methodology include the approximate cardinality of the aggregated benefit measure, flexibility and the possibility of aggregating preferences. As general requirements for the scientific approach, they define the utility theory foundation, the fulfillment of predetermined quality criteria, the cost-effectiveness and practicability of the approach and the completeness of the evaluation using the methods used. The authors conclude that the evaluation of the methods based on this catalog of criteria showed that neither CA nor AHP meet all the requirements of a method for endpoint weighting within the framework of the efficiency frontier approach, taking into account the legal requirements, the IQWiG methodology and the general requirements for scientific procedures and they recommend carrying out further empirical comparative studies that specifically examine both methods with regard to the question of their usability for outcome weighting within the framework of the IQWiG methodology. However, if one analyzes the comparative work already available and also presented here with regard to the catalog of criteria presented, one comes to the same conclusion. Ultimately, users must prioritize the criteria in the catalog of criteria presented based on their own preferences in order to decide on one or the other method (Table 3).
outlook
Multi-criteria decision-making procedures for eliciting patient preferences with outcome weighting are becoming increasingly important in healthcare systems. In Germany, they are a premise for health economic evaluation according to the efficiency frontier approach proposed by IQWiG in order to aggregate endpoint-related benefits. In its current method paper (version 4.0), IQWiG also mentions CA and AHP as a way to compare endpoint-related benefits and harm aspects (IQWiG 2011).
In its Guidance for Industry and FDA Administration Staff 03/2012 under the heading “Factors to Consider When Making Benefit-Risk Determinations,” the FDA (2012) sees patient preferences regarding the benefits and harms of medical applications as a useful source of information. But the EMA (2010) also recognized the importance of determining patient preferences using multi-criteria decision analysis procedures to weigh up benefits and harms in its Benefit-Risk Methodology Project. It remains to be seen whether decision-makers such as the G-BA will also use methods such as the CA and the AHP when making decisions on the additional benefits of new drugs as part of the early benefit assessment to balance benefit and harm aspects instead of intuitive heuristics. Patient-relevant endpoints that reflect the benefits and harms for the affected patients should be weighted by the affected patients also at the system level in order to derive an overall statement regarding any additional benefit in times of shared decision making at the doctor-patient decision-making level. This also requires that the system trust the responsible patient, not only on a purely rhetorical-verbal level, but also in the context of decision-making that directly affects the patients concerned. <<