Assessing the Potential of Off-the-Shelf AI/ML for Autonomous Protest Coding

• In Event: Poster Session: Comparative Politics

Abstract

Protest event analysis is a crucial method for understanding social mobilization. However, the traditional process of collecting structured protest event data has been time-consuming and costly, primarily due to manual labor. Artificial intelligence and machine learning (AI/ML) offer a clear avenue for reducing human involvement, but their efficient deployment often requires advanced knowledge in AI/ML and vast resources for training. Against this backdrop, I empirically evaluate the efficacy of ‘off-the-shelf’ AI/ML without similar requirements. I specifically assess OpenAI’s GPT-4 (which powers ChatGPT) as well as topic modeling approaches. Above all, I find that GPT-4 and human coders demonstrate remarkably similar performance in both classification and coding. While the results underscore the practicality of employing off-the-shelf AI/ML to collect protest data, they also point out potential pitfalls. Lastly, off-the-shelf AI/ML raise hope for the creation of more datasets in the future, especially for lesser-studied places in the Global South.

Author

• Andy Buschmann, University of Michigan