Spillover Effects, Causal Inference, and the Ecological Inference

• In Event: Challenges of Inference with Spillovers, Diffusion, and Interference

Abstract

The recent trajectory of social science research shows a discernible shift towards the design-based perspective of causal inference, reshaping the understanding of the relationship between independent and dependent variables as a causal connection. This paradigm, exemplified by the seminal contributions of Imbens and Rubin (2015) and Angrist and Pischke (2009), has emerged as a robust framework for scrutinizing data from evaluations of interventions, programs, and policies. Grounded in the principles of experimental designs with minimal assumptions, these methods ensure the derivation of impact estimators with reliable statistical properties.

The Stable Unit Treatment Value Assumption (SUTVA) stands out as a foundational element in design-based causal inference methods. SUTVA asserts that the treatment assigned to one unit remains independent of the outcomes of other units, preserving the stability of treatment effects within each unit. This assumption is critical for upholding the validity of causal inferences in design-based methods, and meticulous verification and adherence to SUTVA become indispensable to maintain the independence required for these methods. Any deviation may result in biased estimations and compromise the reliability of impact assessments by allowing the spillover of treatment effects across units. Consequently, meticulous verification and adherence to SUTVA emerge as fundamental elements ensuring the robustness of design-based causal inference in social science research.

In this paper, we argue that design-based perspectives in causal inference are susceptible to violations of SUTVA in the presence of spatial spillover effects in treatment, particularly when applied to ecological inferences. To address this challenge, we introduce the concept of "hierarchical spillover," aiming to investigate how the intensity of spillover at a lower level influences spillover effects at higher levels during the aggregation process. This is crucial given that many theories in the social sciences are grounded in the premise that individual actors adhere to their preferences, explicitly or implicitly allowing for potential spillover effects among them. To illustrate, suppose our treatment variable exhibits stronger spillover effects at the micro-level. However, upon aggregating the data to a higher level — a common practice when using observational data provided at an aggregated level — we discover that the observed spillover effects are less pronounced or even take a different form. While, in most cases, we cannot directly evaluate this discrepancy, it highlights the intricate interplay between micro-level dynamics and their implications for higher-level ecological inferences. Our exploration of hierarchical spillover provides valuable insights into the nuanced relationship between treatment effects and the aggregation process, offering a more comprehensive understanding of making causal inferences for ecological inferences.

In this study, we employ Monte Carlo simulations to investigate the impact of hierarchical spillover within the context of design-based perspectives in causal inference. The configuration of lower-level simulations involves establishing individual geographical units, totaling 30,000, to represent distinct characteristics or behaviors. Throughout this process, we explicitly define spatial spillover effects between units at the lower level to varying degrees, aiming to model the influence of changes in one unit on its surrounding units. Additionally, we generate a model specification in which spatial spillover in treatment explains the dependent variable of interest.
Subsequently, the data is aggregated into a higher level, enabling us to assess its effects on both the overall level of spatial dependency/clustering and the coefficients elucidating the relationships between the treatment and dependent variables at each simple cross-sectional setting and time-series and cross-sectional setting.

Through an iterative process, we demonstrate how the intensity of lower-level spillover influences spatial dependency/clustering at the higher level and impacts the estimation of the treatment's effects. This exercise illustrates how the strength of spillover at the lower level affects spillover at the upper level and the identification of the causal effects of interest. We recommend that researchers carefully consider potential violations of the Stable Unit Treatment Value Assumption (SUTVA) and subsequently test their theories concerning the appropriate unit of analysis. Our findings indicate that even moderate or weak levels of spillover in treatments at lower levels can lead to increased biases in our estimation. This way, we revisit the magic and failure of ecological inference in the new methodological context and underscore the importance of addressing SUTVA concerns and selecting an appropriate unit of analysis to ensure the robustness and accuracy of causal inferences in the presence of spillover effects.

Authors

• Jongwoo Jeong, Washington University in St.Louis

• Young-An Kim, Florida State University