Natural experiments to assess the effectiveness of Covid-19 containment measures

Methodological principles, implementation in NPI studies and a case study

Nearly every country worldwide has implemented measures to contain infections (non-pharmaceutical interventions, NPIs) during the COVID-19 pandemic. Many countries implemented comprehensive packages of measures (lockdowns). Since most of these measures also have negative effects (e.g., learning deficits due to school closures, restrictions on individual freedoms, economic losses), assessing their actual effectiveness—that is, their impact on the overall course of the infection—is of paramount importance. However, the question of causality is by no means trivial. In medicine, the randomized controlled trial (RCT) is often considered the "gold standard" of evidence. The (statistical) conclusion regarding the causal effects of interventions is ensured by the presence of a control group (control conditions) that is comparable to the intervention group (structural equivalence), as well as randomization and, in many cases, blinding (Kabisch et al. 2011: 663 ff.). Regardless of the fact that fulfilling all the aforementioned quality criteria is already challenging in "real" experiments, it quickly becomes apparent that this principle of evidence gathering cannot be directly applied to assessing the effectiveness of pandemic containment measures. However, the effectiveness of non-pharmaceutical interventions (NPIs) – which are, after all, political interventions – can be examined using so-called natural experiments, which are then analyzed using econometric methods. This article first outlines the underlying methodological principle and then, as a first step, discusses the problems in implementing existing NPI studies. Particular emphasis is placed on the necessary control conditions for drawing causal conclusions regarding the effectiveness of measures, and on the specification of the analytical models used in these studies. In the second step, the results of a natural experiment are presented, specifically concerning five NPIs implemented in Germany during the second wave of the COVID-19 pandemic; Switzerland, where these measures – including school and daycare closures – were not implemented, serves as the control group.

Methodological problems of model-based studies for assessing the effectiveness of NPIs
Natural experiments and their statistical analysis in the assessment of the effectiveness of NPI
Natural policy interventions (NPIs), such as the introduction of a minimum wage, a temporary VAT reduction for a sector, or a ban on a specific herbicide, are political interventions. These interventions are not randomized, as a particular measure is not implemented in randomly selected sub-regions. Blinding is also not possible, since members of the intervention group know whether the measure is in place or not. However, provided that a specific intervention is implemented in one area and not in another, there is, as in a "real" experiment, an intervention group and a control group; in such a case, it is referred to as a natural experiment (or quasi-experiment). It is also a before-and-after study with a control group, which allows for a causal conclusion about the effectiveness of an intervention (Kugler et al. 2014: 114 ff.; Wooldridge 2012: 457). However, the conditions are not laboratory-like, but rather real-world. Unlike in a „real“ experiment, the variance of the independent variables is not controlled by the researchers, so it cannot be called a field experiment, but rather a field study (Eifler/Leitgöb 2019: 210 ff.).
Provided that outcome data are available for the same units of analysis at at least two time points—i.e., panel data—econometrics offers various methods for the statistical analysis of the intervention effect (Kugler et al. 2014: 114 ff.). The difference-in-differences (DiD) approach, based on a linear regression model, is particularly prominent. The derivation of causality of outcomes arises, in addition to the existence of control conditions, from the fact that the units of analysis are assigned to the intervention and control groups via the specification of the independent variables. It is not necessary for both groups to have the same baseline level; the DiD approach is even specifically designed for cases where the intervention and control groups have different levels of the outcome of interest before the intervention. The approach compares how the outcome in the treatment group would have developed without intervention (counterfactual) with the development of their actual outcome with intervention. The development of all study units before the intervention and the development of the control group from the time of the intervention serve as reference points. The basic assumption is that the intervention and control groups would have developed similarly (or not statistically significantly differently) without the intervention (Kugler et al. 2014: 120 ff.; Wooldridge 2012: 454 ff.). The DiD approach can be extended by fixed effects for the study units and/or time points, which allows for a generalization of the approach (Goodman-Bacon 2021: 255 ff.).
Regarding the numerous studies on the effectiveness of COVID-19 NPIs, it is worth noting that most do not employ a causal modeling approach as described above (exceptions include de Haas et al. 2022; Kosfeld et al. 2021; von Bismarck-Osten et al. 2022). This is less problematic, as it could be compensated for by specifying the respective analytical models differently. However, a far more serious issue is that the aforementioned prerequisites for a natural experiment—namely, the existence of an intervention group and a control group—are often not met.
The majority of NPI studies focus on the first wave of the COVID-19 pandemic (e.g., Banholzer et al. 2021; Brauner et al. 2021; Flaxman et al. 2020; Haug et al. 2020), which presents a problem regarding their explanatory power: In spring 2020, containment measures were very similar internationally, resulting in only slight variance between the countries compared. Greater differentiation only emerged from the second wave of the pandemic (autumn/winter 2020/2021) onward, for which Germany and its neighboring countries are a good example (e.g., school closures in Germany and Austria, but not in France and Switzerland). The problem of low variance between the units of analysis also applies when only subregions of a single country are included in the analysis and there were only minor differences in the implementation of measures within that country. This case is found both in peer-reviewed scientific papers, which are otherwise certainly beyond reproach (e.g., Kosfeld et al. 2021; Khazaei et al. 2023), and in publicly funded NPI studies such as the Swiss evaluation of measures (Funk et al. 2022) and the Robert Koch Institute's "Stop COVID" study on German measures (an der Heiden et al. 2023). Deriving causal effects of measures is not possible if, due to a lack of variance between the units of study, there are no control conditions, i.e., if the same measures were implemented (almost) simultaneously across the entire study area.
The control conditions are also weakened by the fact that many studies aggregate individual measures that actually vary between the countries or regions studied. For example, the closure of primary and secondary schools as well as universities is often combined into a single NPI, and then an effect is estimated for this aggregate (e.g., Brauner et al. 2021; Gavenciak et al. 2022; Haug et al. 2020; Sharma et al. 2021). The problem here is that different types of schools and universities were by no means treated equally; in many countries, universities switched to distance learning from the second wave of the pandemic onward, but schools remained open (e.g., France, Switzerland). Other studies even aggregate the closures of schools, universities, and daycare centers into a single NPI (e.g., Kosfeld et al. 2021), thereby losing another dimension for comparison. In the cases mentioned, it is difficult to determine the effect of the closures. B. can no longer separate a (possible) effect of closing a single type of school from a (possible) effect of closing other types of schools or universities.
The control conditions continue to be influenced by the choice of spatial level of analysis and the study period: A large-scale level of analysis is frequently chosen, particularly entire countries (e.g., Banholzer et al. 2021; Brauner et al. 2021; Flaxman et al. 2020; Haug et al. 2020; Gavenciak et al. 2022), which obscures regional differences in infection rates and the design of pandemic measures. For example, some non-pharmaceutical interventions (NPIs) in Germany were implemented at the district level (e.g., curfews); the subsequent federal emergency brake was also district-specific. Two studies that otherwise have control conditions limit their data to the year 2020, thus ending their analysis in the midst of the second wave of the coronavirus pandemic (Levelu/Sandkamp 2023; Sharma et al. 2021). It is difficult to derive effects in this case because only a (relatively small) part of the entire intervention period is included in the analysis.

Problems in model specification in NPI studies
Nearly all empirical NPI studies are based on regression analyses with interventions as independent variables. The effective reproduction number usually serves as the dependent variable in the regression model (e.g., an der Heiden et al. 2023; Brauner et al. 2021; Flaxman et al. 2020; Gavenciak et al. 2022; Khazaei et al. 2023; Levelu/Sandkamp 2023; Li et al. 2021; Sharma et al. 2021). In such a model, the effect of a single explanatory variable is considered in isolation, i.e., under otherwise constant conditions (ceteris paribus). The empirically estimated influence of a variable is therefore, in essence, only "valid" in combination with the other explanatory variables in the model. To estimate it without bias, other variables must also be included in the model, which, while not of interest, may influence the dependent variable. This is the principle behind control variables, which are included in such models to "factor out" their influence (Hilmert 2019: 4 ff.; Wooldridge 2012: 97). This aspect of model specification can be generalized: A regression model that omits variables which actually influence the dependent variable is underspecified. Consequently, the influences of the explanatory variables included in the model may also be overestimated or underestimated; this is referred to as omitted variable bias (Greene 2012: 96 ff.; Wooldridge 2012: 88 ff.).
A review of the analytical methodology in previous NPI literature reveals that many studies using the effective reproduction number as the dependent variable integrate only NPIs as independent variables in their regression models (e.g., Brauner et al. 2021; Flaxman et al. 2020; Li et al. 2021; Sharma et al. 2021). For example, Gavenciak et al. (2022: 4), referring to the work of Brauner et al. (2021) and Sharma et al. (2021), state: „Both models assume the time-varying reproduction number R(t) to be a product of R0, the ‘natural’ reproduction number when no NPIs are in place, multiple terms representing the effects of interventions, and noise terms modeling the influence of other, unobserved, factors on R(t).“ In such a model, therefore, no other independent variables that could influence the temporal course of Rt are included. It is therefore implicitly assumed that the effective reproduction number is constant as long as no corona measures are in effect.
However, as is well documented in the epidemiological literature, the time course of Rt depends not only on containment measures, but also on the following aspects:
• Decrease in susceptible population: An infected person is immune after their own infection – at least for a certain period of time – and therefore (temporarily) can neither be reinfected nor infect another person; the same applies to vaccination (Mercer et al. 2011: 984 f.; Nishiura/Chowell 2009: 104)
• Population heterogeneity: Not every individual has the same probability of infecting themselves or others, also due to very different contact patterns (Müller 2021: 70 f.; Viboud et al. 2016: 31 ff.)
• Voluntary behavioral changes: Individual reduction of infection risks independent of measures (Mercer et al. 2011: 984 f.; Nishiura/Chowell 2009: 104; Viboud et al. 2016: 31 ff.), which has also been demonstrated in the Corona pandemic, e.g. with regard to mobility behavior (e.g. Goolsbee/Syverson 2021; Jacobsen/Jacobsen 2020).
• Seasonality: Like other respiratory viral diseases (e.g., influenza, RSV, other coronaviruses), SARS-CoV-2 follows a seasonal cycle in temperate climates, with a focus on autumn and winter (Dbouk/Drikakis 2021; Moriyama et al. 2020: 84 ff.).

Of course, it is impossible to consider all real-world explanatory factors in a statistical model. However, the models of the aforementioned studies cannot be fully specified, which complicates the interpretation of the results regarding the NPI variables (omitted variable bias, see above). This is because if only measures are included as explanatory variables in the model, every change in the R-value is necessarily attributed to the effect of these measures, even if it occurred for other reasons. Some subsequent NPI studies have expanded their analytical models to include certain aspects mentioned above, such as model terms for seasonal fluctuations (an der Heiden et al. 2023; Gavenciak et al. 2022), the effect of COVID-19 vaccinations on the R-value (an der Heiden et al. 2023), or (possible) voluntary behavioral changes (Levelu/Sandkamp 2023). However, even in these models, no variables are found that, apart from these partial aspects, could bring about a change in the effective reproduction number. Only the work by Khazaei et al. (2023), published in November 2023, considers voluntary behavioral changes as a kind of "personal measure" and, furthermore, incorporates the decreasing probability of (re-)infection due to previous infections and vaccinations into the model equation.
Another aspect regarding the inclusion of control variables is that many studies only include a portion of the implemented COVID-19 measures in their regression models. For example, the aforementioned studies, which group the closure of schools and universities into a single NPI, do not consider the closure of childcare facilities at all, or at least do not indicate whether this measure is included in the aggregate (Brauner et al. 2021; Gavenciak et al. 2022; Sharma et al. 2021). Khazaei et al. (2023), on the other hand, deliberately select eight NPIs (including school closures, curfews, and restrictions on the number of people at events). Regardless of whether these omissions are due to deliberate selection or a lack of data, a similar problem exists here as with other conceivable control variables: The effect of a specific measure found in the model can be over- or underestimated because not all other measures are included in the model. It is possible that an effect found is actually due to other measures that do not appear in the model.
Case study: Baden-Württemberg and Switzerland in the second wave of the coronavirus
During the second wave of the COVID-19 pandemic, the intensity of government containment measures had already diverged significantly between countries, making effectiveness analyses based on natural experiments possible in many places. This is evident, for example, in the direct comparison between Germany and neighboring Switzerland: While both countries implemented measures of varying intensity (albeit with a time lag), Switzerland, unlike Germany, refrained from closing childcare facilities, primary and secondary schools, hotels, and personal services (e.g., hairdressers). This difference provides a good basis for a natural experiment, as no-treatment control conditions prevail with regard to these interventions. Therefore, this study analyzes the influence of these five non-physical interventions (NPIs) in relation to Switzerland and the neighboring state of Baden-Württemberg. The spatial level of analysis is the NUTS3 level, i.e., urban and rural districts (Baden-Württemberg) or cantons (Switzerland).
The empirical analysis builds on previous NPI studies (see above): A regression model is constructed with the effective reproduction number as the dependent variable and the COVID-19 measures as independent variables (in the form of dummy variables) (see Fig. 1). However, the model specification is modified in two respects: First, the analysis of the intervention effect is performed according to the difference-in-differences (DiD) approach. Following the formalization in Wooldridge (2012: 457 f.), the intervention group for a specific measure (e.g., closure of primary schools) and the intervention period (e.g., days on which the primary schools in the intervention group were closed not due to holidays but as a COVID-19 measure) are each labeled with a dummy variable, and the interaction of these two variables is additionally included in the model. The coefficient of this interaction term is the DiD estimator and indicates the intervention effect of the specific measure (percentage change in the R-value).
Secondly, control variables are added to the model that can influence the course of Rt independently of pandemic measures. Analogous to the model by Levelu/Sandkamp (2023: 9 f.), the R-value ten days prior is included in the model, as it is assumed that people may react to increased infection rates and engage in individual risk avoidance. Furthermore, cumulative numbers of infections and vaccinated individuals, as well as their interaction over time, are included in the model to account for the (potential) slowing of the infection wave due to (temporary) immunity (see above). A model was estimated for each of the five NPIs. All other COVID-19 measures are also included in the model as control variables to isolate the actual effect of the intervention of interest (Fig. 1).
Table 1 shows the model estimates and the associated model fit statistics. Figure 2 shows the DiD estimators from the five models and the corresponding confidence intervals based on robust standard errors. A significant DiD estimator below zero indicates the effectiveness of a measure in the form of an average reduction in the effective reproduction number. Here, it is shown that the closure of personal services causes an average reduction in the effective reproduction number of approximately 1.71 TP3T. This measure, which was implemented in Baden-Württemberg (and throughout Germany) during the second wave of the coronavirus pandemic, but not in Switzerland, can therefore be considered effective. No such effect could be found for the other four NPIs examined. The DiD estimators for the closure of schools, daycare centers, and hotels are even positive, suggesting a counterproductive effect. This cannot be attributed to the fact that the model design makes it difficult to reject the null hypothesis in terms of statistical significance, because on the one hand these point estimates are also significant (only with a different sign than expected), and on the other hand the theoretically expected effect has already been demonstrated for another investigated NPI – closure of personal services (Fig. 1, Table 1).
A look at the control variables shows that, in the vast majority of cases, they have a statistically significant influence on the dependent variable. For example, the R-value decreases significantly with increasing numbers of both infected and vaccinated individuals, which corresponds to the epidemiological basis regarding the effective reproduction number (see, e.g., Mercer et al. 2011: 984 f.; Nishiura/Chowell 2009: 104). The fact that the R-value also has a significantly decreasing effect ten days earlier may indicate that there were voluntary changes in behavior, in the sense that people reacted to increased infection rates and avoided infection risks. However, this statistical result does not allow us to determine the extent to which this actually occurred or what impact these behavioral changes had on the infection wave. In general, the control variables should not be overinterpreted. Nevertheless, it must be emphasized that they are necessarily integrated into such a model for analyzing the effectiveness of NPIs. Without them, the model would be underspecified.
Conclusion
The assessment of the effectiveness of pandemic measures, unlike the effect of a drug or therapy, is not a medical question. Non-physical interventions (NPIs) are political interventions whose effectiveness can be evaluated in the same way as other government interventions. This requires that the initial conditions of a natural experiment are met, meaning that there is an intervention group and a control group that can be compared, and that panel data are available for the units of analysis. If these conditions are met, econometrics, with its difference-of-differences approach, offers an established analytical methodology that can depict any initial differences between the units of analysis in the two groups and simultaneously account for other influences in the form of control variables.
The literature on COVID-19 measures—of which only representative examples can be mentioned here, and a complete bibliography cannot be provided—shows that many studies do not fully meet these requirements and that their results must be interpreted in light of these limitations. Under no circumstances can the causal effect of a measure be inferred if the studies that found this effect lack control conditions, i.e., if no comparison region(s) without this measure were included in the analysis. Furthermore, the effect size of measures must be questioned, at the very least, if studies only consider non-pharmacological indicators (NPIs) as explanatory variables or if important control variables are missing. This problem does not concern the researchers involved, as they have documented their methodology and its limitations, but rather the reception of such studies in the media and by policymakers. In any case, the methodology of an NPI study must be critically examined with regard to control conditions and model specification in order to answer the following question unequivocally: Can the study make a causal statement about the effect of a measure? Another aspect not addressed here is the efficiency or proportionality of a measure, which should be the subject of political or social discussions.
The case study shows that the closures of schools, childcare facilities and hotels, in a direct comparison of two neighboring regions with a "real" control group, have no dampening effect on the effective reproduction number, and are therefore ineffective in the sense of the methodology of previous NPI studies.
This reveals, on the one hand, the importance of natural experiments and their adequate statistical analysis, and on the other hand, the necessity of control variables in the model, as these have demonstrable influences on the course of the infection. Without them, the model would be misspecified, and the effects of the measures under investigation might be misjudged. The finding tends to be similar to that in the (few) studies that have examined COVID-19 measures with "real" control conditions and a DiD approach; for example, no effect was found from nighttime curfews (de Haas et al. 2022) or from school closures (von Bismarck-Osten et al. 2022).
Limits of the procedure
Of course, this also reveals the limitations of the method: A causal analysis in the sense of a natural experiment was only possible for five measures, as only these had "real" control conditions. No conclusions can be drawn about other NPIs on this basis.
Furthermore, it must be pointed out that a natural experiment like this can only ever make statements regarding an average intervention effect, i.e., an effect at the population level. The results do not reflect individual risk. For example, the fact that school closures do not show an infection-reducing effect in society does not, of course, mean that a person cannot become infected with SARS-CoV-2 in a school or that no infections whatsoever have occurred in schools. It simply means that the infections that were (possibly) prevented by school closures are so few that their aggregated effect is not reflected in the overall infection rate in society. It is also conceivable that there are offsetting effects, i.e., that infections that were (possibly) prevented by school closures were even overcompensated for by the fact that students and teachers were instead in another location where the risk of infection is (possibly) much higher.