Algorithms without Trauma: Against Human Content Moderation

• In Event: Promise and Perils of Content Moderation on Digital Platforms

Abstract

This paper explores the morality of human content moderation. Content moderation is the activity of classifying and filtering harmful content, among other things, to minimize the amount of violent and sexual content that circulates on digital platforms or is included in training data. The problem with human moderation is that it has been documented to inflict great psychological harm on moderators, who are often exposed to a steady flow of disturbing imagery, including child sexual abuse imagery and extreme violence. The paper makes the case that, even if we grant that social media platforms or chatbots are a valuable—or inevitable—force in current societies, and even if moderation plays an important role in protecting users and society more generally from the noxious effects of these digital spaces, it is far from clear that tasking humans to conduct such moderation is permissible given the psychic toll of the practice on moderators. Human moderation is commonly criticized because of the working conditions of moderators, but that is just the surface of the problem. The morality of the practice is dubious at best for reasons that go beyond improper working conditions hitherto unrecognized.

The case against human content moderation and for automated moderation in the paper unfolds as follows. Platforms that rely on moderators have a dual nature as commercial services and as public squares of sorts. As such, they can be considered economic and political agents. The paper first considers trauma from the economic perspective and focuses on the problem of exploitation. It argues that, seen under this light, moderation may be acceptable notwithstanding its harms, since the trauma it inflicts on those who conduct may be a lesser evil compared to the lack of employment opportunities and the ensuing deprivation moderation workers and their dependents would otherwise endure. The paper then explores human moderation from the political perspective and through a different theoretical lens, interpreting it as a public sacrifice—a risk in the service of protecting society. There are many lines of work which we would deem permissible, notwithstanding the personal risk to those who undertake them: firefighters, police persons, humanitarian rescuers, undercover agents, soldiers, and so on. The main justification for tolerating potential harms in these professions is that they serve the common good. The paper casts doubt on the idea that human moderation is analogous to these professions in serving the public interest. And even assuming for the sake of the argument that it is, human moderation is a disproportionate measure to protect the public interest because the harm it inflicts on moderators is of a kind and magnitude that compromises the moral integrity of moderators and must therefore not be expected, let alone demanded, from anyone. The paper concludes by arguing that algorithms should be left to do the job of cleansing large language models and digital platforms. The consequences of the ensuing lack of accuracy, if such exist, must be borne by society.

Author

• Juan Espindola, UNAM