A New Measure of Democratic Backsliding Using Large Language Models

• In Event: Democratic Backsliding: New Measures and Approaches

Abstract

Most measures of democratic backsliding use expert coders to approximate the level of democracy in a given country-year. These measures have been incredibly useful to the field in recent years, but there is still space for measures that use data from various countries to estimate democratic erosion. In this article, we propose one such approach. We use an advanced natural language processing model to code attacks on democracy, both in terms of inclusion and contestation, in parliamentary interventions in 42 countries. In particular, we train a RoBERTa model to identify anti-democratic messages in floor speeches concerning (1) minority populations, (2) opposition, (3) judicial independence, (4) the need to rule by decree, (5) media groups, and (6) the electoral process. We constructed a large, high-quality, and balanced training set of 5,000 debate interventions containing both attacks on democracy as well as other unrelated interventions. After training the model, we apply it to 52.3 million debate interventions. With the codings, we perform ideal point estimation to obtain a score for every country-year for its overall level of commitment to democracy in legislative speeches. We believe this score is a good proxy for the level of democratic backsliding.

The intuition behind this new measure lies with the importance of the legislative chamber in democratic systems: if an anti-democracy party manages to capture a large share of seats in parliament, the whole system may collapse, as happened in the 1930s in Germany. Other institutions, such as the presidency in presidential systems, the Supreme Court in countries where justices have a lifetime appointment, or constitutions, can slow the process of backsliding, but if an anti-democracy party has full legislative power for a period of time, it is unlikely these institutions will be able to resist. Preliminary testing shows that our measure performs well compared to other current expert-coded alternatives, such as V-Dem. One of its main advantages, however, is that it provides more variation across time units, lowering serial correlation. This helps account for an aspect of democratic backsliding that is often overlooked, namely, that countries can be on the brink of breakdown but resist and return to relative normalcy after an election removes anti-democratic parties from positions of power. This also matches another intuition from Germany in the early 1930s, which showed how quickly backsliding can turn into breakdown. Measures that are less temporally correlated can thus better reflect changing rapidly conditions in many countries across the world.

Author

• Joan C. Timoneda, Purdue University