Search
Browse By Day
Browse By Time
Browse By Person
Browse By Mini-Conference
Browse By Division
Browse By Session or Event Type
Browse Sessions by Fields of Interest
Browse Papers by Fields of Interest
Search Tips
Conference
Location
About APSA
Personal Schedule
Change Preferences / Time Zone
Sign In
X (Twitter)
Certain concepts of interest on social media, such as moralization and othering in posts, can be difficult for coders to label consistently. For example, multi-coder labels of tweets for othering and moralization have very low average inter-rater reliability, even after multiple rounds of coding (Chen et al., 2022). In this paper, we specifically tackle moralization and othering using concept-guided chain-of-thought (CGCoT) (Wu et al., 2023), a text scaling method using generative large language models. This approach uses an LLM to generate a concept-specific breakdown for each tweet. We then make pairwise comparisons along moralization and othering dimensions using an LLM. The results of the pairwise comparisons are scaled using the Bradley-Terry model, giving us a moralization CGCoT score and an othering CGCoT score for each tweet. We use a novel dataset of randomly sampled pairwise comparisons between tweets made by multiple coders to validate these scales. The advantage is that we can make a much larger number of pairwise comparisons compared to what coders can do while still retaining the advantages of pairwise comparisons in labeling settings (Carlson and Montgomery, 2017). Preliminary results across both moralization and othering show that as the distance in pairs of tweets' CGCoT scores increases, the coder is more likely to agree with the LLM's pairwise comparison. This approach could also be used to assess how well LLMs align with human preferences.