Leveraging LLMs for Analyzing Policy Stances and Arguments in Political Debate

• In Event: New Approaches to Measurement from Text Data Using Generative AI

Abstract

Analyzing the stances speakers take and the arguments they make is crucial for understanding policy debates in representative bodies. However, existing computational methods face limitations in accurately capturing speakers' positions and the rationale behind them. This paper introduces a novel approach that leverages Large Language Models' (LLMs) instruction-following, extractive summarization, and text coding capabilities to identify and code stances taken and arguments made in a polical debate. I demonstrates the effectiveness and reliability of this approach in an analysis of the Council of the European Union debates and German committee meeting transcripts by benchmarking model outputs against trained coders' annotations as well as expert judgments. The proposed approach enables comprehensive insights into argumentation and stance-taking in political debates, enriching quantitative measures with qualitative evidence. Consequently, this paper contributes by offering a new automated text analysis tool for political scientists, advancing the application of LLMs in political analysis, and bridging the divide between computational and qualitative approaches to analyzing political debates.

Author

• Hauke Licht, University of Innsbruck