Using Generative LLMs to Scale Othering and Moralization in Social Media Posts

• In Event: Navigating Political Narratives: New Frontiers in Text Analysis, LLMs and NLP

Abstract

Certain concepts of interest on social media, such as moralization and othering in posts, can be difficult for coders to label consistently. For example, multi-coder labels of tweets for othering and moralization have very low average inter-rater reliability, even after multiple rounds of coding (Chen et al., 2022). In this paper, we specifically tackle moralization and othering using concept-guided chain-of-thought (CGCoT) (Wu et al., 2023), a text scaling method using generative large language models. This approach uses an LLM to generate a concept-specific breakdown for each tweet. We then make pairwise comparisons along moralization and othering dimensions using an LLM. The results of the pairwise comparisons are scaled using the Bradley-Terry model, giving us a moralization CGCoT score and an othering CGCoT score for each tweet. We use a novel dataset of randomly sampled pairwise comparisons between tweets made by multiple coders to validate these scales. The advantage is that we can make a much larger number of pairwise comparisons compared to what coders can do while still retaining the advantages of pairwise comparisons in labeling settings (Carlson and Montgomery, 2017). Preliminary results across both moralization and othering show that as the distance in pairs of tweets' CGCoT scores increases, the coder is more likely to agree with the LLM's pairwise comparison. This approach could also be used to assess how well LLMs align with human preferences.

Authors

• Patrick Y. Wu, American University

• Jonathan Nagler, New York University

• Joshua A. Tucker, New York University

• Solomon Messing, New York University