The Attrition Permutation Test: ML for the Detection of Systematic Attrition

• In Event: Challenges in Learning from Panel Data

Abstract

Systematic attrition in panel data, particularly panel experiments, is a serious threat to statistical inference. While a serious problem, existing approaches for detecting and adjusting for systematic attrition are limited to simple visual tests and post-hoc regression-based approaches. Given this gap in the literature, we implement a new statistical test, the “attrition permutation test,” designed to detect arbitrarily complex systematic attrition in panel data. Given the goal of attrition detection is purely predictive, and researchers' expectations may be of little value in the modeling process, our approach exploits the superior predictive power of machine learning for this task. Moreover, we use permutation inference in conjunction with the estimated ML model to validate our results. If attrition is detected, we recommend a doubly-robust estimation procedure approach to account for the bias. To illustrate the utility of our approach, we demonstrate our method through both Monte Carlo simulations and real-world examples. Our results demonstrate the superior detective abilities of the attrition permutation test over standard methods. Additionally, we show that in cases where attrition bias is detected, doubly-robust estimates recover more precise treatment effect estimates, with similar coverage rates, relative to standard estimation procedures. Finally, we introduce an efficient and easy-to-use R package, MLattrition, that implements the attrition permutation test. We believe this new method represents a dramatic improvement in researcher’s ability to detect systematic attrition and adjust their estimation procedure in a principled manner.

Authors

• Sam Fuller, Harvard University

• Jack Rametta, University of California, Davis