A Statistical Framework to Engage the Problem of Disengaged Survey Respondents

• In Event: Advances in Survey Design

Abstract

There is growing concern across democracies that partisans are increasingly willing to support violence to resolve political disputes. Evidence for these claims come from surveys that are often plagued by bias from disengaged respondents, who cause overestimates of the share of individuals who hold low prevalence attitudes, like support for violence. Researchers commonly use attention checks to address this bias, but discarding those who fail checks also risks bias and changes the population studied. In this paper we introduce partial- and point-identified estimators that use attention checks to obtain credible estimates of population-level attitudes from surveys comprised of both engaged and disengaged respondents. Using transparent assumptions about the quality of the attention checks, we obtain sharp bounds on the proportion of respondents who hold a particular attitude. We also provide point-identified estimators that adjust for differences in measured covariates between engaged and disengaged respondents. To perform this adjustment we provide estimation strategies to flexibly adjust for covariates, including a modification to double machine learning. Using our estimators we find that survey estimates severely overstate support for political violence and disengaged respondents are unlikely to sincerely hold pro-violence views.

Authors

• Matthew Tyler, Rice University

• Justin Grimmer, Stanford University

• Sean Westwood, Dartmouth College