A Meta-Reanalysis of Experiments on Religion and Political Attitudes

• In Event: Experimental Methods in Religion and Politics Research

Abstract

The experimental study of the effects of religious messages and endorsements by religious leaders on political attitudes has a relatively short history – the earliest study with publicly-available
data we are able to find appeared in 2010 (Djupe and Gwiasda 2010). In the intervening years, sociologists, political scientists, and religious scholars have conducted at least 30 more such studies in a variety of contexts, with equally varied treatments and outcomes. By our count, however, exactly zero pairs of studies have investigated the effects of substantively similar treatments on substantively similar outcomes. While novelty is often prized among social scientists, this lack of
commensurability across studies impedes scientific generalization.

The goals of our paper are threefold. First, we aim to document this variety in experimental design, to show how this particular experimental literature is not yet building upon itself.

Second, we obtain the original datasets, select, clean, and standardize the appropriate covariate, treatment, and outcome variables, and then re-estimate causal effects using the same statistical models across all primary studies. This “meta-reanalysis” (Galos and Coppock 2023) allows us to characterize whether, despite the obvious variation in experimental design, we nevertheless obtain generalizable knowledge from each idiosyncratic study’s estimate of the effects of religious messages or religious endorsements. While traditional meta-analyses extract estimates from published papers and then use various transformations to standardize effect sizes, in our approach we only include estimates we can recalculate ourselves. This method has the clear advantage that we can standardize outcome variables directly (here rescaled not by a sample variance estimate but instead to fall in the 0 to 1 range) as well as the less obvious advantage that we can report effect estimates supported by the design but not reported by the original study authors. In particular, some studies measured target attitudes post-treatment but reported results on different outcome variables; some studies measured subjects' religious affiliations but did not report the corresponding conditional effects.

We synthesize the extant experimental record on three theoretically distinct estimands. The first is the effect of exposure to a religious message versus a pure control condition with no message on the ``target attitude'' of the religious message. For example, we might study the effect of a Christian appeal to charity rooted in the example of Jesus helping the poor on subjects' attitudes toward redistribution. This estimand is important to study in the sense that the total effect of religious appeals is what is most politically relevant. Our second estimand is the effect of a religious message compared with a substantively similar secular message, for instance, an appeal to charity rooted in our common humanity. This estimand is theoretically important because it teases out whether the religiosity of the message per se is persuasive, or whether secular messages with similar content achieve the same goal. Our final estimand is the effect of a particular class of group cues, i.e., the effects of policy endorsements by religious leaders on their target attitudes. We synthesize the average estimates for each estimand, and also the conditional average estimates depending on subjects' religious affiliation.


Finally, we also aim to include in the paper a discussion of innovative research designs that we are unable to analyze here, representing unique manipulations of religious exposure (e.g. Bryan, Choi, and Karlan 2020), or under-studied outcome (e.g. Grewal and Cebul 2023; Siegel and Badaan 2020). Our goal in characterizing the variety of experimental designs used to study the effects of religion (broadly conceived) on politics (similarly broadly conceived) is to induce researchers to choose designs that balance commensurability with innovation, for example by including common arm treatments and outcomes that serve as bridges to the extant literature alongside novel manipulations and measurements.

Authors

• Radha Sarkar, Tecnologico de Monterrey

• Alexander Coppock, Yale University