Overcoming the AI “Black Box” in Political Analysis: A Hybrid Approach

• In Event: Quantitative Study of Mis/Disinformation

Abstract

Natural Language Processing (NLP) techniques have proven its utility for Political Science, particularly in domains where computational text analysis plays a pivotal role. The deluge of data produced by the mass adoption of digital technologies has emphasized the relevance of these techniques due to the expanding volume and complexity of digital text sources. The NLP field has responded to this challenge incorporating machine learning techniques that have significantly enhanced the capabilities of computational text analysis. The integragion of neural networks has particularly contributed to improved NLP performance, achieving data processing capacities capable to surpass those of human analysts. However, the inherent "black box" nature of the underlying neural network operations imply substantial limitations on their applicability in Political Science. Firstly, the opaque nature of neural networks obscures causal insights, posing challenges in evaluating the validity of analyses. Secondly, implementations such as generative models, exhibit inherent inconsistency, yielding diverse results with each run and occasionally generating responses that, while plausible, are incorrect. Third, the reliance on external data sets for the training of those models, introduce issues of bias that need to be controlled for in order to ensure the validity of the analysis.

In order to address the validity and reliability challenges introduced by these limitations, scholars in Computer Science scholars have advocated for a "hybrid" approach where the lack of transparency of neural networks can be mitigated by integrating structured knowledge or human intervention. In this paper, we present a hybrid approach for applying artificial intelligence techniques to political analysis, so we can fully harness the potential of AI techniques while addressing their methodological limitations. We take the case of the "Great Replacement," a conspiracy theory that has garnered popularity within Western alt-right movements, particularly in the US and Europe. This theory is regarded as a driving force behind societal polarization and is implicated as a motivator in instances of extreme violence, underscoring the critical need for meticulous analysis of this phenomenon to mitigate the adverse effects of its ideology on democratic societies. The proliferation of disinformation in social media supporting this theory provides a rich source of data to test our approach.

Our research design goes as follows. First, we define the semantic field addressing the 'Great Replacement Theory', exploring linguistic and contextual variances using computational text analysis to identify the most relevant terms, consistently with the political science literature. Second, we apply conventional political discourse analysis to produce an annotated corpus where the polarity of relevant sentences is identified for the topics more closely connected to the 'Great Replacement' theory. Third, we perform the same polarity analysis, this time applying several transformer-based Artificial Intelligence algorithms, and using the semantic set as an ontology consistently with computer science standards. Finally, we use the annotated corpus as a gold standard to evaluate the performance of the tested AI approaches, gaining insights about their potentials and limitations.

Our work contributes with a research design that can be applied to other research topics, particularly in the field of political communication, discourse analysis or disinformation studies. In order to ensure that our approach is reusable, we make our code and data available through the Harvard Dataverse repository, so researchers can independently evaluate our analysis, and use the code for their own research in a way that ensures a strong commitment to ethical research practices, particularly concerning data access.

Authors

• Suso B Baleato, Universidade de Santiago de Compostela

• Erik Marino, University of Évora

• Renata Vieira, University of Évora