Generative User Content for Social Media: LLM Effectiveness and Approaches

• In Event: Poster Session: Political Communication 1 & Politics, Literature, and the Arts
  In Poster Session: Poster Session: Political Communication 1

Abstract

The study of social networks has become an integral part of the study of social media dynamics in contemporary communications science research. While data across a number of platforms is relatively abundant, there is limited scope for researchers to control the real-life behaviors of users, making research centered around A/B testing, experiments, and controlled manipulations logistically challenging and, at times, entirely unfeasible. We therefore present an AI-based alternative to the use of real-life data for these research designs and test its viability as a reliable and straightforward tool for researchers in the study of social networks and online behavior.

The rise of Large Language Models, through the use of realistic AI generated content, offer an emerging alternative avenue for researchers aiming to study behavioral, network, and algorithmic effects on social media, even at a time when open access to readily available contemporary social media data is decreasing (for example, recent limitations on both the Twitter and Reddit APIs). Indeed, recent works have shown that LLM-generated content (e.g., Törnberg et al. 2023 using ChatGPT) is able to simulate real-world social media content, with additional fine-tuning and prompting facilitating the realistic manipulation of user-level opinions, interactive behaviors, and syntactical styles, as well as for message-level adjustments in tone and sentiment. Therefore, LLMs can theoretically be used to study both content generation and subsequent user responses.

Given that generated LLM content shows great promise but is, to date, an emerging and understudied approach to the study of social media communications, a comprehensive review of the promises and challenges of this approach is warranted. With this in mind, this paper systematically tests the realism of LLM-generated social media content across two distinct classes of content generation (generating original content and replying contextually to existing user content). We do this by comparing four different LLM models (GPT-4, LlaMA, Falcon, Flan-T5) across four different platforms (Twitter, Facebook, Reddit, Telegram) in five different languages (English, German, Serbian, Dutch, Italian), across both task classes.

Specifically, we fine-tune and train each LLM with over four million real-world social media messages, collected in the first half of 2023, sent by a pool of over 4,000 general users, politicians, and media accounts, spread across platforms and country/language contexts. We then prompt each model to create a series of social media messages, with variation on topic, tone, and user profile. These prompts are repeated across each language and each platform, to create a comparable pool of generated messages across all contexts of interest.

The veracity of generated content is then manually validated using human coders who compare the content generated by one specific LLM model to comparable content generated by different LLM models. These human coders then also compare generated content to real-world social media content, matched on platform, language, topic, and tone, to create “realism” scores, quantifying the degree to which content generated by a given LLM is seen as comparable to real social media content.

Preliminary results suggest that, at this time, GPT modelling stands out as the most realistic for generating social media content, across languages and platforms, often being indistinguishable from human content. In particular, GPT-4 utilizes correct and contextually appropriate usage of features such as hashtags, emojis, and mentions, setting it apart from other LLM models which either lack these outputs or fail to replicate human usage of these features.

Finally, in addition to providing a quantitative assessment and validation of these generative approaches, in multiple contexts, we also provide a general pipeline and roadmap for the generation and validation of generated social media content using LLMs, which we believe will be a valuable tool for researchers working in this rapidly developing field.

Authors

• Michael Heseltine, University of Amsterdam

• Simon Münker, University Trier

• S.B. Stolwijk, University of Amsterdam