With the entry into force of the Act on the Reorganization of the Pharmaceutical Market (AMNOG) on January 1, 2011, the benefit assessment procedure for new drugs was introduced in Germany in accordance with Section 35a of the German Social Code, Book V (SGB V). Since then, the Federal Joint Committee (G-BA) has assessed the added benefit of new drugs in over 500 procedures based on the evidence submitted by the pharmaceutical manufacturer and, with the decision, forwarded the price negotiations between the pharmaceutical manufacturer and the umbrella organization of statutory health insurance funds. The vast majority of these assessments were in the field of oncology. The procedures are documented transparently on the
The G-BA homepage has been made available to the public (www.g-ba.de).
Particularly in the area of benefit assessment, new methodological questions continually arise that have a significant impact on the evaluation and assessability of the evidence for new drugs. These include, among other things, the availability of results from randomized clinical trials, along with the question of when data were included in the statistical analyses and whether these time points were defined a priori or post-hoc, whether they were requested by regulatory authorities such as the European Medicines Agency (EMA) or the American Food and Drug Administration (FDA) during the approval process, and whether all or only some of the patient-relevant endpoints were analyzed. A second major aspect of the required event-time analyses is the assessability of the results in light of censorship, i.e., the question of how many events should be observed for an assessment to be possible.
The reason for this commentary is the fundamental idea of establishing a common foundation and basic knowledge, and of initiating a discussion about the meaning of technical terms such as "prospective" versus "post-hoc" or "censorship" within the context of benefit assessment. The following will explore which constellations of data cuts and censorship can occur, how the G-BA (Federal Joint Committee) has dealt with these in recent procedures, and what lessons can be learned from this.
Interpretation of different data sections
in ongoing studies
It is not uncommon for statistical analyses to be performed before the end of the overall study duration. The most frequent reasons are interim analyses, i.e., interim analyses for the primary endpoint, or preliminary analyses, i.e., final analyses for the primary endpoint at a point in time when, for example, an insufficient number of events for secondary endpoints have yet been observed. Particularly in oncology, a data cutoff is prospectively defined, usually based on a required number of events that must have occurred, at which the primary analysis is performed. However, data, especially for secondary endpoints, continue to be collected during the study for later analyses. At the time of the preliminary analysis of the primary endpoint, e.g., progression-free survival, the necessary number of events for the secondary endpoint, overall survival, for a robust analysis has often not yet occurred. In the approval process, the results for the data cutoff for the primary endpoint are then initially submitted for the preparation of the clinical trial report, and analyses based on later data cutoffs are submitted subsequently during the course of the process. In the post-marketing benefit assessment, the question arises as to which data set should be used for evaluating a drug. A data set refers to the point in time up to which data are included in the statistical analysis. A distinction is made between a priori and post-hoc defined data sets, the former during the approval process and the latter within the benefit assessment procedures. In the approval process, the following definition is commonly used: Prospectively or a priori planned data sets are those that are described in detail in the study protocol and are used for interim analyses, the primary analysis for presentation in the clinical trial report, and – if the observation period extends beyond the time of the study report's creation – the final analysis at the conclusion of the entire study.
The definition of terms used in the admission process a
Priori and post-hoc analyses are not directly transferable to the benefit assessment. For the benefit dossier, the manufacturer is required to conduct the statistical analysis as prospectively planned in the study's statistical analysis plan. According to the G-BA's (Federal Joint Committee) rules of procedure, further analyses must be carried out, which are requested by the G-BA independently of the G-BA's analyses. In the benefit assessment, only analyses, endpoints, and subgroups that are subsequently initiated by the manufacturer and defined and carried out with knowledge of the analyses already performed should be considered post-hoc.
Unlike the approval process, the benefit assessment process involves further data cuts based on a continuously growing database from ongoing studies, leading to increased information and a better estimation of potential added benefit. One example is the cumulative meta-analysis approach used by the Cochrane Institute, in which meta-analyses are regularly updated to ensure that treatment decisions are based on the most current information possible. Clarke et al. (Clarke et al. 2014) comment: „This large, unique collection of cumulative meta-analyses highlights how a review of the existing evidence might have helped researchers, practitioners, patients and funders make more informed decisions and choices about new trials over decades of research.” Subsequent data cuts generally increase the database and thus the reliability of the results.
One example is the evaluation of ribociclib in combination with fulvestrant for the treatment of breast cancer. In the 2019 evaluation, an interim analysis showed a numerical, but not statistically significant, benefit in overall survival (G-BA 2019e). For resubmission after the deadline for the decision (March 2020), a further planned data set was available. Here, statistical significance could be demonstrated (G-BA 2020a). Data sets requested by regulatory authorities are also generally considered, as, for example, in the procedure for lenvatinib in advanced renal cell carcinoma (G-BA 2017).
Accordingly, data slices should be used in the utility assessment if the only difference to previous data slices is the length of the observation period, but nothing changes in the analysis methods, the endpoints and the presentation of the results.
All data cutting operations have in common the required effort. The database is closed on the date of the data cutting operation, the data then contained in the database is checked for plausibility and errors, corrected if necessary, and then submitted for statistical analysis. Considering the effort involved, a data cutting operation should be carefully considered. Particularly in the case of unblinded studies, if data cutting is unplanned and not requested by regulatory authorities, it is essential to assess the relevance of the data cutting operation for the benefit assessment.
The template for Module 4 states: „Furthermore, describe whether and for what reason various data slices have been carried out or are planned. Also indicate whether these data slices were planned in advance (i.e., in the statistical analysis plan). As a rule, only the presentation of data slices planned a priori or required by regulatory authorities is necessary.” (G-BA 2019a).
The procedural rule stipulating that, as a rule, only pre-planned data slices requested by the regulatory authorities are to be reported suggests that other data slices may also be relevant. For example, in the case of pomalidomide, an interim analysis, the primary analysis, and an unplanned data slice approximately six months after the primary analysis were used to determine the added benefit (G-BA 2016). In contrast, in the apalutamide procedure, an unplanned data slice submitted with the statement was not considered in the decision (G-BA 2019f). Here, the G-BA cited the uncertainty in the validity of the results and did not take the significant overall survival benefit into account when determining the added benefit.
It remains unclear what criteria should be used to identify relevant data cuts. Furthermore, in practice, it is challenging to define data cuts for individual endpoints, such as overall survival in oncology, but also for security updates.
For endpoints, it is also necessary to perform all analyses for all other relevant endpoints. Here, it is crucial to demonstrate which endpoints are still being collected. For example, with late data cutoffs, many patients are outside the observation window for adverse events and patient-reported endpoints related to symptoms and quality of life. The available information does not change with later data cutoffs. Overall survival, however, can still be determined even after the study has ended or after the patient has ceased participation. In situations where so-called data cleaning—i.e., quality control of the entered data and correction of errors in the database—is performed only for the endpoint, problems arise in practice when analyses for all endpoints are required.
A higher number of relevant data slices does not necessarily lead to easier evaluation of the results. For example, if oncological studies have results from many data slices that are consistent but not necessarily statistically significant for every slice, this can lead to a gradation in the reliability of the results (Schwenke 2018).
Censorship in event-time analyses
in the context of benefit assessment
In the context of benefit assessment, event time analyses play a central role whenever the observation periods differ between the treatment groups under investigation (Unkel et al. 2018). An analysis that disregards these differing observation periods would lead to a bias favoring the treatment with the shorter observation period, as these patients are under treatment for a shorter time and therefore also at lower risk of reporting an adverse event. In such situations, event time analysis has become established in the benefit assessment process and has been incorporated by the G-BA (Federal Joint Committee) into the new template for the benefit dossier (G-BA 2019a).
A common approach is to use a Kaplan-Meier plot, from which the median time to event can be directly read (Zwiener et al. 2011). In addition, the hazard ratio is usually estimated for comparing the event time curves (Zwiener et al. 2011). Patients without an event are censored at the last time point for which data for the relevant endpoint are available. Censoring can be for two reasons: either the patient prematurely terminates the study, or the patient who remains under observation has not experienced an event by the data cutoff point. At later data cutoffs, the data situation changes because no further data are collected only for patients who prematurely terminate the study.
Subsequent data cutoffs generally lead to greater certainty, provided there has been no crossover, i.e., no transition of patients from control to active therapy. The procedure for abemaciclib serves as an example. In an initial assessment in 2019 (G-BA 2019b), no significant overall survival benefit was achieved in one of the evaluated subpopulations, with event rates of 201 TP3T in both arms. In the reassessment after the time limit expired in 2020, event rates of 461 TP3T and 671 TP3T, respectively, and a significant hazard ratio were reported for the later data cutoff in the active and control groups, resulting in an additional benefit (G-BA 2020b).
For event time analyses, the G-BA (Federal Joint Committee) uses not only the hazard ratio but also the difference in the median time to the event in its assessment. This difference is also important in the medical field, as it can be explained in a way that is understandable even to laypersons, whereas the hazard ratio, as a relative effect measure, requires a methodological understanding of the statistical method. If the median cannot yet be estimated, the 25% quantile or event rates at specific time points, such as in year 1 or 2, can be used as alternatives.
As Prof. Hecken explains in the 2017 Drug Report, early data cutoffs are problematic, as in the example of afatinib as a first-line therapy, while later data cutoffs are significantly more meaningful and, in the case of afatinib, led to a considerable additional benefit in the subpopulation of patients with EGFR mutation Del19 (Hecken 2017; G-BA 2015).
Censorship and its influence on the
Derivation of the added benefit
For endpoints with different observation periods in the treatment arms, the Institute for Quality and Efficiency in Health Care (IQWiG) typically derives the added benefit based on the relative effect measure hazard ratio (IQWiG 2017). The Federal Joint Committee (G-BA), however, additionally uses the absolute treatment effect in the form of the difference in median times to the event. For the latter, the timing of censoring plays a crucial role in the precision of the median estimation. For both the hazard ratio and the median estimation, early censoring leads to increased uncertainty. In this case, the median is determined by events in the few patients still under observation. Due to the expected variability, this event can result in a shorter or longer median, depending on whether the next event occurs earlier or later in the remaining patients. This effect is particularly relevant because no explicit statistical test is performed for the difference in medians; instead, the median estimators are considered without a measure of variability.
If the median is not reached in at least one treatment group, no statement can be made about the difference between the medians. If the medians are reached in both groups and there are many censored data from before the median date, then the data available to determine the median come from only a few at-risk patients. This leads to significant uncertainty and can result in a difference of several months.
In the 2019 benefit assessment of dacomitinib, the Federal Joint Committee (G-BA) stated in its decision that while a significant hazard ratio was achieved for the assessment of the added benefit in terms of overall survival, and the median time to event in the intervention arm was also extended by 7.3 months, a high number of censorships in the median range were noted, which did not allow for a quantification of the added benefit (G-BA 2019c). It should be noted that, according to the subsequently submitted data cutoff from May 19, 2019, the vast majority of patients with censorship had reached the final observation time point of week 48 as per the protocol and had completed the study as planned. Patients who reached week 48 were therefore censored on the date of study termination. The reason for the censorship was thus administrative, which makes an informative effect of the censorship on the estimation of the treatment effect unlikely. Subsequent data cuts do not resolve the issue of censorship.
A similar situation can be found in the previously mentioned resubmission of ribociclib with fulvestrant (G-BA 2020a). Here, median overall survival has so far only been reached in the control group, and there are numerous censored values around the median. In the intervention group, the median has not yet been reached, despite also numerous censored values. Here, the significant advantage in the hazard ratio was used to derive the added benefit, and the potential for bias was also classified as low, even though, in the control group, fewer than 101 TP3T of patients were still at risk when the median was reached. In the previously mentioned case of apalutamide in prostate cancer, the data cutoff for overall survival shown in the dossier was used to derive the added benefit, even though the median had again only been reached in the control arm, with numerous censored values before the median in both treatment groups (G-BA 2019f). Since only 2 out of 401 patients were at risk when the median was reached, the assessment of the median was considered questionable.
discussion

Two important aspects for assessing the added benefit of a drug in the benefit assessment according to §35a SGB V repeatedly raise questions: firstly, the terms prospective vs. post-hoc, and secondly, the questions of the influence of censorship on the assessability of event time analyses.
The conflict between prospective and post-hoc approaches is particularly relevant when assessing the relevance of new data sets in ongoing studies. According to the procedural guidelines, prospectively planned data sets should be presented in the benefit dossier, and there are limitations if an unplanned data set is presented. In such cases, a justification must be provided for why a particular data set is used or not. For assessing the relevance of data sets, the definitions of "prospective" and "post-hoc" are of paramount importance. The term "post-hoc" should not be applied to data sets in the benefit assessment if the only difference from previous data sets is the length of the observation period, while the analytical methods, endpoints, and presentation of the results remain unchanged.
In general, it is advisable to address the issue of data cuts as a question to the G-BA during (early) consultation in order to gain clarity about the relevance of unplanned data cuts in particular, insofar as this is already foreseeable at the time of the consultation.
The longer one waits, the more certain the statements usually become, as the number of events included in the analysis increases and the number of censorships decreases.
Linked to the topic of data censorship is the question of data maturity for analysis and, consequently, the proportion of patients in event-time analyses who have not yet experienced an event. Even with a smaller number of events, a significant treatment effect can be observed, even if the median time to event has not yet been reached. Thus, numerous censorships around the median are not a reason to forgo an evaluation or to quantify the added benefit. In such cases, the relative treatment effect, expressed as the hazard ratio, can be used. It remains unclear, however, what can be used to assess the absolute treatment effect when there is no difference in median times to event. Alternatives include the 25% quantile or event rates at meaningful time points, sensibly defined as time points at which a sufficient number of patients are still uncensored.
At the European level, there are also many question marks regarding the benefit assessment, including how data editing and censorship will be dealt with in the future (Juhnke 2019).
In summary, all parties should agree on a common terminology to standardize the requirements for preparing a benefit dossier and to ensure the uniformity of benefit assessments for medicinal products.

Clarke M, Brice A, Chalmers I (2014) Accumulating Research: A Systematic Account of How Cumulative Meta-Analyses Would Have Provided Knowledge, Improved Health, Reduced Harm and Saved Resources. PLoS ONE 9(7): e102670. doi:10.1371/journal.pone.0102670
Joint Federal Committee (2015): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Afatinib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/170/ (accessed on 26.06.2020)
Joint Federal Committee (2016): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Lenvatinib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/194/ (accessed on 26.06.2020)
Joint Federal Committee (2017): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Lenvatinib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/261/ (accessed on 26.06.2020)
Joint Federal Committee (2019a): Dossier on benefit assessment pursuant to Section 35a SGB V Module 4, document template, version of 21 February 2019. Available at: www.g-ba.de/download/40-268-5599/2019-02-21_VerfO_Aenderung-Anlage-I-II_Kapitel-5_Anlage-II-7_WZ.pdf (accessed on 15 May 2020)
Joint Federal Committee (2019b): Benefit assessment of medicinal products with new active substances according to § 35a SGB V Abemaciclib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/409/ (accessed on 11.09.2020)
Joint Federal Committee (2019c): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Dacomitinib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/460/ (accessed on 15.05.2020)
Joint Federal Committee (2019d): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Rucaparib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/446/ (accessed on 26.06.2020)
Joint Federal Committee (2019e): Benefit assessment of medicinal products with new active substances according to § 35a SGB V Ribociclib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/430/ (accessed on 26.06.2020)
Joint Federal Committee (2019f): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Apalutamide, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/437/ (accessed on 26.06.2020)
Joint Federal Committee (2020a): Benefit assessment of medicinal products with new active ingredients pursuant to Section 35a SGB V Ribociclib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/527/ (accessed on 11.09.2020)
Joint Federal Committee (2020b): Benefit assessment of medicinal products with new active ingredients according to § 35a SGB V Abemaciclib, Available at: www.g-ba.de/bewertungsverfahren/nutzenbewertung/535/ (accessed on 11.09.2020)
Hecken J. (2017): Experiences from the early benefit assessment of oncology drugs – the G-BA's perspective. In: Grandt, D./Schubert, I. (eds.) Drug Report 2017 (2017): 248–251
Institute for Quality and Efficiency in Health Care (2017): General Methods. Version 5.0 of 10 July 2017. Available at: www.iqwig.de/de/methoden/methodenpapier.3020.html (accessed on 15 May 2020)
Juhnke, C. (2019): Shaping European benefit assessment? In: Monitor Versorgungsforschung 2019, 04: 46-47.
Schwenke C., Schwenke S. (2018) The early benefit assessment of pharmaceuticals according to § 35a SGB V. In: Pfannstiel M., Jaeckel R., Da-Cruz P. (eds) Innovative healthcare and market access. Springer Gabler, Wiesbaden
Unkel S, Amiri M, Benda N, Beyersmann J, Knoerzer D, Kupas K, Langer F, Leverkus F, Loos A, Ose C, Proctor T, Schmoor C, Schwenke C, Skipka G, Unnebrink K, Voss F, Friede T (2018): On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies. Pharmaceutical statistics. 18: 166-183
Zwiener I, Blettner M, Hommel G: Survival time analysis. Deutsches Ärzteblatt, Vol. 108(10): 163-169