New Approaches to Measurement from Text Data Using Generative AI

Session Submission Type: Full Paper Panel

Session Description

In just a few short years, Large Language Models (LLMs) have revolutionized the field of natural language processing. Generative LLMs are trained on massive datasets of text with millions of dollars of computing resources, and are capable of performing a wide variety of tasks, including question answering, summarization, translation, and annotation. The most capable of these models (such as OpenAI's GPT4) can write code, generate images, and reason about complex topics.

However, it is not yet clear how political scientists should take advantage of these models' capabilities for their research. Should we use high powered LLMs for classification instead of more lightweight open-source options like BERT? When should we use LLMs instead of established tools like topic modeling for unsupervised exploration of large datasets? Is there any problem with relying on OpenAI's closed-source, proprietary models? If so, what open-source alternatives exist and how can we use them? When should we use prompt engineering or fine-tuning techniques when using LLMs to interpret data?

This panel seeks to provide clarity on these questions, as well as guidance for researchers looking for examples of innovative new applications of generative AI to political science research.

“Large Language Models for Measuring Contested and Multi-Dimensional Concepts”, shows how we can use prompt engineering techniques instead of more laborious model training and classification routines to measure fundamentally contested concepts like populism from political texts.

Both "Making the Implicit Explicit with Generated Natural Language Representations" and "Leveraging LLMs for Analyzing Policy Stances and Arguments in Political Debate" shows how we can generate rich representations of meaning from large text corpora by using language models to generate generalized abstractions of political, then operating directly on the generated abstractions with clustering and labeling approaches.

Finally, "Evaluating RAG Systems for Information Retrieval from Political Corpora" frames measurement as an information retrieval problem, and compares the performance of both open- and closed-source implementations of Retrieval-Augmented Generation (RAG) systems for summarizing information from a large corpus of legislative text data.

Sub Unit

• Division 40: Information Technology, & Politics

Individual Presentations

• Large Language Models for Measuring Contested and Multi-Dimensional Concepts - Yaoyao Dai, University of North Carolina, Charlotte; Benjamin J Radford, University of North Carolina at Charlotte

• Making the Implicit Explicit with Generated Natural Language Representations - Alexander Hoyle, ETH Zurich; Philip Resnik, University of Maryland; Rupak Sarkar, University of Maryland, College Park

• Leveraging LLMs for Analyzing Policy Stances and Arguments in Political Debate - Hauke Licht, University of Innsbruck

• Evaluating RAG Systems for Information Retrieval from Political Corpora - Mitchell Bosley, University of Michigan

Chair

• Walter R. Mebane University of Michigan, Ann Arbor

Discussant

• Arthur Spirling Princeton University