Navigating Confounderland: Uncovering Latent Variables in Legal Texts

• In Event: Methods and Measures

Abstract

Legal analysis and the empirical research of law and courts often circles around the pursuit of establishing causal relations. Often, this involves drawing inferences about causality from legal documents. Scholars, whether using statistical models or experimental designs, have endeavored to identify and assess the impact of document-specific factors on outcomes of interest. For example, they have assessed how the readability of a petitioner’s brief might affect her chance of winning an appeal (see, e.g., Long & Christensen, 2011). Even prominent theories on judicial behavior that have previously shown skepticism towards the role of written law in explaining the actions of legal actors (like studies employing attitudinal models) have leveraged legal texts to identify and gauge relevant variables.

Recent advancements in the realm of causal theory and natural language processing have automated and refined this procedure. Emerging methodologies, often referred to as “causal text methods” (Howe, 2021), now allow the recognition of causal effects in situations where textual data plays a pivotal role. Because extracting causal relationships from text holds considerable significance within legal research, introducing these causal text methods carries profound implications for scholars in this field. Nonetheless, prevailing research has adhered to a traditional and linear approach, wherein researchers are expected to a priori identify the pertinent variables, typically based on existing theories. Subsequently, they employ a methodology to measure the prevalence of these variables, followed by the formulation of hypotheses and a research design to test the observable implications derived from their established theory (King, Keohane, & Verba, 2021). While this deductive methodology may aid in mitigating false discoveries stemming from researchers’ subjectivity, it could also result in analysts overlooking valuable opportunities to uncover intriguing questions, patterns, and measurement techniques directly from the text data, while also missing or overlooking other important factors critical to effective causal design.

Earlier research has effectively leveraged low-dimensional representations of legal texts to measure latent variables like ideology or sentiment, aiming to evaluate their influence on specific outcomes like judicial rulings (see, e.g., Ash, Chen, & Galletta, 2022; Ash, Chen, & Naidu, 2022). However, legal texts, like any other form of written content, are complex and high-dimensional. Unseen attributes within these texts could potentially impact or refine the identified connections in various manners. A research method that values the inclusion of inductive reasoning, not solely deduction, as imperative for discovering and quantifying relevant text features, could significantly enhance our theories and refine our findings. Drawing inspiration from the methodological groundwork laid by Fong and Grimmer (2016, 2023), this paper centers methods to uncovering unseen treatments or confounding elements (features that affect the outcome of interest) within legal texts, and how to utilize them for causal inference effectively.

To achieve this, I focus on the impact of amicus briefs in the U.S. Supreme Court. More precisely, I delve into the question of which features of amicus briefs are related to a brief’s in- fluence on the Court. A well-established body of literature has examined the influence of these briefs in terms of determining winners and losers in legal proceedings, the ideological orientation of court decisions and judges’ votes, as well as the substance and presence of judicial opinions (Collins, 2018). Certain studies have treated the content of amicus briefs as the primary tool through which those submitting them attempt to sway judges (Collins, 2004, 2008). Nonetheless, all these investigations have followed a deductive approach based on measuring the effect of predefined text attributes from the briefs on their influence. In contrast, I employ the approach advocated by Fong and Grimmer to identify “inductively” concealed features within the briefs’ texts that serve as key factors for causal inference–either as potential treatments or confounding variables for control purposes. In summary, the findings reveal that language associated with Intentionalism (pertaining to legal interpretation) and legislative authority enhances the likelihood of influence, whereas briefs discussing criminal justice and employee benefits might also have a positive impact when controlled by other variables. Additionally, it is observed that the inclusion of these attributes as potential confounders in existing models does not alter previous findings.

Author

• Santiago Virgüez, University of Masachusetts Amherst