Is GPT-4 Right Wing or Left Wing?

• In Event: Challenges in Using Large Language Models

Abstract

GPT-4 is increasingly popular among political science scholars to generate data, classify text, or complement human coders. However, its biases remain underexplored, as do potential solutions that could reduce bias in downstream analyses. For political scientists, one of the most important and most understudied biases is ideology. Does GPT-4 tend to produce left-leaning or right-leaning output? In this article, we provide a novel approach to identify ideological bias in GPT-4, exploiting linguistic and issue-based differences across countries. Since bias stems from both filtered and unfiltered training data, we hypothesize that GPT-4 is going to reflect the political attitudes of the people who produced the text on which it is trained. As a result, ideological bias in GPT-4 output will vary widely depending on whether issues are contested or uncontested in a given locale.

To show this, we focus on four issues --(1) abortion, (2) gun rights, (3) Catalan independence, and (4) immigration-- in four different countries: the US (English), Spain (Spanish, Catalan), Italy (Italian) and Germany (German). In the English-speaking world, abortion and gun rights are only sensitive, politically important issues within the United States. Similarly, in the Spanish-speaking world, Catalan independence is of salience only in Spain. We also know, for instance, that Italians tend to be less favorable of abortion due to a higher share of observing Catholics and Catholic values, as do Americans. Germans, on the other hand, tend to be more accepting of abortion, as are Catalans. By tapping into (1) languages that are geographically confined and (2) issues that are geographically confined across languages, we can approximate the extent of ideological bias within GPT-4. For each of the topics and languages above, we generated 1,000 responses from GPT-4 to the question ``What is your stance on [blank]''. We then asked research assistants to code each of these outputs as right- or left-leaning. Early results show that GPT-4 tends to produce scores similar to those in the World Values Survey, but that some topics may exhibit more extreme positions. When it does take more extreme positions, it tends to be left-leaning. Our warning to applied researchers is that ideological bias may be of concern in politically sensitive topics. We recommend applying a version of our approach to determine the exact extent of the bias for their application, and then applying a correction in downstream analysis to minimize bias in the results.

Authors

• Joan C. Timoneda, Purdue University

• Christina Walker, Purdue University