Eva Krumhuber

In the past decade, spontaneous and dynamic facial expression databases (FEDBs) have expanded rapidly, advancing emotion research beyond static, posed expressions. Yet existing reviews remain fragmented due to limited search strategies and a lack of systematic methods. This article synthesizes 128 spontaneous and dynamic FEDBs, examining their conceptual, technical, and practical dimensions. It summarizes how emotions are represented, measured, and validated across diverse populations; how databases vary in composition, recording methods, and modalities; and how accessibility, usability, and ethics shape their use. By integrating these perspectives, the review consolidates current knowledge, identifies methodological gaps, and outlines directions for developing more diverse and ecologically valid FEDBs that better capture the complexity of human emotion.

The growing availability of facial expression databases (FEDBs) has accelerated the development of empathic AI systems designed to promote emotional awareness and well-being. However, most existing systems are trained solely on posed (acted), static databases featuring exaggerated and stereotypical displays. Such portrayals may not accurately represent the real-world expressions that are often subtle, heterogeneous, and ambiguous, raising concerns about the performance of these AI systems in inferring human emotions. Furthermore, the lack of cross-database evaluation has limited assessments of how well these systems generalize to diverse facial behaviors. To address these gaps, the present study evaluates five spontaneous and dynamic databases that provide more ecologically valid representations of affective responses observed in everyday life. We assessed the performance of a widely adopted affective computing system, AFFDEX (v1.0; iMotions, Copenhagen, Denmark), to examine how basic emotions are inferred from spontaneous facial movements. Results reveal substantial variability in decoding accuracy across emotion categories, database contexts, and demographic factors. Prototypical and complex expressions were decoded more accurately than subtle or heterogeneous ones, while ambiguous expressions that blend multiple affective signals impaired machine predictions. Together, these findings underscore the crucial need to train and validate affective computing systems using diverse FEDBs that encompass a wider spectrum of behaviors to improve robustness and real-world generalizability.

As decoding emotional expressions is essential to navigating the social world, it is imperative that measures of facial expression labeling ability are psychometrically rigorous and easy to administer. Unfortunately, most studies have used images of prototypically posed expressions, which lack the nuance and variation of real-life expressions and are often perceived as fake. To address the need for a more ecologically valid measure of expression labeling, we introduce the Naturalistic Expression Labeling Task (NELT), modeled after an established posed expression labeling task. We investigated whether the NELT shows expected associations with empathy and general cognitive ability, and the extent to which these associations align with those found for the corresponding posed expression task, despite differences in the realism and perceived genuineness of the expressions. Across three studies, we found that the NELT had strong psychometric properties—including high reliability—that make it well suited to examining individual differences in expression labeling ability. While both the NELT and the posed expression task showed similarly sized positive associations with measures of cognitive and affective empathy, the NELT exhibited a stronger positive association with cognitive ability than did the posed expression task. Our findings suggest that naturalistic expressions can provide insights into expression labeling ability that are at least as robust as those derived from posed expressions. The NELT can serve as a valuable tool for researchers seeking to enhance the ecological validity of their studies by incorporating naturalistic stimuli.

It is widely believed that being mimicked causes liking and social bonding, but the underlying neural mechanisms remain largely unknown. Additionally, it is unclear whether similar effects arise when abstract choices or physical actions are mimicked. In this study, we used functional near-infrared spectroscopy (fNIRS) to examine the involvement of temporal and parietal brain regions during in-person interactions in which participants were either mimicked by a confederate in terms of their motor movements (n = 30) or their choices (n = 30). In the Choice group, the participant and confederate chose the same (mimicry) or different (no mimicry) paintings in the absence of any physical mirroring action by the confederate. In the Motor group, the participant and confederate performed the same (mimicry) or different (no mimicry) right-hand movements while viewing distinct pairs of pictures. Choice mimicry led to a strong sense of affiliation across various measures, which was evident through activation of the bilateral Inferior Parietal Lobule (IPL). In contrast, motor mimicry produced only a subtle liking effect and showed activation in the left Posterior Superior Temporal Sulcus (STS) and the left Temporo-Parietal Junction (TPJ). In addition, awareness of the study’s goal increased the right Intraparietal Sulcus (IPS) activity in the Choice group but did not do so in the Motor group. These findings suggest that choice mimicry exerts a stronger influence on affiliation than motor mimicry and highlight differential neural pathways for detecting mimicked choices and mimicked actions.

Most prior research on basic emotions has relied upon posed, static displays that do not accurately reflect the facial behavior seen in everyday life. To address this gap, the present paper aims to highlight existing facial expression databases (FEDBs) that feature spontaneous and dynamic displays of the six basic emotions. To assist readers in their decisions about stimulus selection, we comprehensively review 25 FEDBs in terms of three key dimensions: (a) conceptual features which reflect thematic approaches in database construction and validation, i.e., emotional content and elicitation procedures, encoder demographics, measurement and elicitation techniques; (b) technical features which concern technological aspects in stimulus development, i.e., stimulus numbers and duration, frame rate, and resolution; and (c) practical features which entail information about database access and potential ethical restrictions. Finally, we outline some of the remaining challenges in stimulus generation and make recommendations for future research.

The pursuit of affective computing aims to endow machines with the ability to read and respond to human emotions. Unfortunately, several high-profile publicly available machine classifiers are based on psychological theory that over-emphasises the prototypical, morphological features of six or seven basic emotional expressions. This approach is poorly suited to the complex and nuanced repertoire of naturalistic facial expressions; it also omits other physical information that humans use in emotion perception such as tears, eye-gaze, and facial colouring. Prior studies have focused on lab-generated expression stimuli, which are mostly posed, and evaluated machine and human classification against database labels. Here, we use a large, novel set of naturalistic expression stimuli (2,453 still images from 824 YouTube clips) to directly compare machine (Affdex) and human emotion classification. Overall, we found that human-machine agreement was strikingly low. At best, humans and Affdex agreed on two-thirds of happy stimuli, but this was more than double the rate for the next agreed-upon emotion (surprise = 33%). Agreement was under twelve percent for the remaining five emotion categories, with Affdex detecting only prototypical expressions. The findings underscore the importance of designing machine systems that can adapt to the subtleties of human emotional expressions for greater human-machine consistency.

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

Recent evidence shows that AI-generated faces are now indistinguishable from human faces. However, algorithms are trained disproportionately on White faces, and thus White AI faces may appear especially realistic. In Experiment 1 ( N = 124 adults), alongside our reanalysis of previously published data, we showed that White AI faces are judged as human more often than actual human faces—a phenomenon we term AI hyperrealism. Paradoxically, people who made the most errors in this task were the most confident (a Dunning-Kruger effect). In Experiment 2 ( N = 610 adults), we used face-space theory and participant qualitative reports to identify key facial attributes that distinguish AI from human faces but were misinterpreted by participants, leading to AI hyperrealism. However, the attributes permitted high accuracy using machine learning. These findings illustrate how psychological theory can inform understanding of AI outputs and provide direction for debiasing AI algorithms, thereby promoting the ethical use of AI.

Background: To achieve the goal of enhancing facial beauty it is crucial for aesthetic physicians and plastic surgeons to have a deep understanding of aesthetic ideals. Although numerous aesthetic criteria have been proposed over the years, there is a lack of empirical analysis supporting many of these standards. Objectives: This aim of this review was to undertake the first exploration of the empirical evidence concerning the aesthetic ideals of the face in the existing literature. Methods: A comprehensive search in MEDLINE, Embase, Scopus and CENTRAL databases was conducted for primary clinical studies reporting on the classification of the facial aesthetic units as per the Gonzales-Ulloa facial aesthetic unit classification from January 1962 to November 2022. Results: A total of 36 articles were included in the final review: 12 case series, 14 cohort studies, and 10 comparative studies. These described the aesthetic ideals of the following areas: forehead (6 studies; mean level of evidence, 3.33); nose (9 studies; mean level of evidence, 3.6); orbit (6 studies; mean level of evidence, 3); cheek (4 studies; mean level of evidence, 4.07); lips (6 studies; mean level of evidence, 3.33); chin (4 studies; mean level of evidence, 3.75); ear (1 study; level of evidence, 4). Conclusions: The units that were most extensively studied were the nose, forehead, and lip, and these studies also appeared in journals with higher impact factors than other subunits. Conversely, the chin and ear subunits had the fewest studies conducted on them and had lower impact factors. To provide a useful resource for readers, it would be prudent to identify and discuss influential papers for each subunit. Level of Evidence: 3: