History versus Unobserved Confounding in Causal Panel Analysis

• In Event: Challenges in Learning from Panel Data

Abstract

Political scientists have a longstanding tradition of using observational panel data to establish causality. These data, in which each unit is observed across multiple periods, allow researchers to achieve causal identification based on less stringent assumptions compared to cross-sectional scenarios. Early research attempts to accurately model outcome dynamics by integrating lagged dependent variables (LDVs) and exogenous covariates into the regression equation (Beck and Katz, 2011). Recently, the use of models accounting for unobserved confounders—such as the difference-in-differences (DID) estimator, the two-way fixed effects (TWFE) model, and the synthetic control method (SCM)—has significantly increased among practitioners.

Despite its popularity, the second approach, specifically the TWFE model, has faced various critiques. As a result, practitioners often face challenges when selecting an appropriate method. In this paper, we propose a design-based perspective for understanding causal inference with panel data. We treat the data as being generated by a hypothetical ideal experiment, in which an experimenter (e.g., Mother Nature) assigns the treatment, and a researcher later analyzes the data after choosing structural restrictions based on their knowledge. We underscore its value by providing guidance for practitioners, highlighting essential trade-offs between different ideal experiments, addressing identification problems when observed confounding is present, and emphasizing the necessary structural restrictions that must be imposed, even when the treatment assignment process cannot be precisely discerned.

To do so, we first demonstrate how the identification assumptions behind the two approaches, sequential ignorability and strict exogeneity, represent two types of ideal experiments. The former corresponds to a sequential experiment, wherein the experimenter assigns the treatment after observing the outcome from the preceding periods and exogenous factors from the current period. The latter, on the other hand, aligns with “baseline randomization,” in which case, the probability of getting treated depends solely on the subject’s long-term characteristics, whether observed or unobserved, at the outset, although the probability may evolve over time due to factors unrelated to the subject. Additionally, we delve into the possibility that treatment assignment may concurrently depend on both long-term confounding and short-term shocks, as reflected in past outcomes or covariates.

We then draw attention to an identification issue in the presence of unobserved confounding. Specifically, we demonstrate that, without imposing additional structural assumptions, it is impossible to simultaneously accommodate treatment reversal, arbitrary heterogeneous treatment effects (HTE), and arbitrary carryover effects within the fixed effect framework. We refer to this phenomenon as a trilemma. This result echoes the under-identification problem in staggered adoption settings when anticipation effects are present (Borusyak, Jaravel and Spiess, 2021). We build upon this finding by demonstrating that even in the absence of anticipation effects, treatment reversal and unstructured carryover effects introduce additional complexities to the identification of causal effects using fixed effects models.

We explore several structural restrictions that could potentially navigate around this trilemma. The most evident, and relatively easy to confirm with data and domain knowledge one, is staggered adoption, or the absence of treatment reversal. We illustrate that in this scenario, if there are many pretreatment periods, methods developed under both identification assumptions can yield similar results. This is because, by conditioning on past outcomes, both short-term observed confounding and long-term unobserved confounding are accounted for. However, in more general cases involving treatment reversal, estimating causal effects becomes significantly more challenging. This is because researchers need to restrict either the structure of HTE or the structure of carryover effects and select methods that can accommodate these restrictions. We illustrate how researchers can test the validity of these restrictions and guide informed decision-making by integrating the test results with their substantive knowledge.

We validate these results through Monte Carlo simulations and by revisiting the relationship between democracy and economic development. Our findings reveal that, regardless of the method used, when we restrict the sample to countries that never revert to autocracy (hence, staggered adoption), democratization results in a short-term decline in GDP per capita followed by a long-term increase. This pattern cannot be detected if we apply fixed effects models to the full sample, likely due to the bias introduced by HTE and carryover effects.

Authors

• Ye Wang, University of North Carolina, Chapel Hill

• Yiqing Xu, Stanford University