Challenges in Using Large Language Models

Session Submission Type: Created Panel

Session Description

The panels offers critical perspectives on the use of Generative Large Language Models (LLMs) in Political Science research. Papers offer studies on the on the impact of different learning approaches (Zero-Shot, Few-Shot, and Fine-Tuning), the impact of uncertainty and robustness quantification on LLM outputs, the differences between open-source and proprietary models, the extent to which biases can be identified in the models' output, how useful they are in processing open-ended responses in political surveys, and the challenges posed by identifying frames.

Sub Unit

• Division 8: Political Methodology

Individual Presentations

• Detecting Policy Frames from Text - Meysam Alizadeh, University of Zurich; Maël Dominique Kubli, University of Zurich; Fabrizio Gilardi, University of Zurich

• Beyond Human Judgment: How to Evaluate Language Model Uncertainty - Arthur Spirling, Princeton University; Christopher Barrie; Alexis Palmer, Dartmouth College

• Negativity Bias in Political Perception Revisited - Richard R. Lau, Rutgers University, New Brunswick; Barea M. Sinno, Rutgers University; George D. Quinn, Rutgers University

• Is GPT-4 Right Wing or Left Wing? - Joan C. Timoneda, Purdue University; Christina Walker, Purdue University

Chair

• Marc Ratkovic

Discussants

• Marc Ratkovic

• Adam Breuer Dartmouth