Taming Affect: On the Construction of Objectivity in Data Annotation Practices
Open Access
- 7 May 2025
- journal article
- research article
- Published by Springer Nature in Digital Society
- Vol. 4 (2) , 1-23
Abstract
Affective computing applications promise to decode human emotional expressions and provide objective insights into users’ affective experience, ranging from frustration and boredom to states of clinical relevance such as depression and anxiety. In this paper, I explore the contexts of development of facial expression recognition systems, machine learning systems designed to recognize changes in facial expressive behaviors and produce some form of meaningful knowledge about such expressions. Drawing on 20 interviews with engineers and annotators involved in developing or working with these models, this paper sheds empirical light on the socio-material practices through which practitioners articulate notions of objectivity, truth, and accuracy to make sense of affective data and validate their systems’ predictions. By exploring two common annotation schemes used to label facial expression recognition training datasets, the Facial Action Coding System (FACS) and the Valence, Arousal, Dominance (VAD) model, I foreground the situated judgments and negotiation practices required to make sense of affective data. Here, I juxtapose annotators’ account of data work – the non-linear interpretative practices that are necessary to ‘fit’ the data into FACS and VAD coding grids – to engineers’ perceptions and expectations around annotation. Here, I show how practitioners mobilize ideal notions of ‘subjectivity’ and ‘intuition’ to manage the uncertainty and ambiguity of affective data.Keywords
This publication has 56 references indexed in Scilit:
- “Everybody knows what a pothole is”: representations of work and intelligence in AI practice and governanceAI & SOCIETY, 2025
- Paul Ekman and the search for the isolated face in the 1960sJournal of the History of the Behavioral Sciences, 2024
- Artificial intelligence and the conjectural sciencesBJHS Themes, 2023
- A Silicon Valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditabilityPatterns, 2022
- Algorithmic injustice: a relational ethics approachPatterns, 2021
- Mechanized Significance and Machine Learning: Why It Became Thinkable and Preferable to Teach Machines to Judge the World2020
- Quantifying Quality: Towards a Post-humanist Perspective on Sensemaking2018
- Artificial intelligence as craftwork1993
- The Taming of Chance1990
- Evidence for a three-factor theory of emotionsJournal of Research in Personality, 1977