Abstract
Inferring others’ emotions is crucial for survival. Much work focused on facial expressions; others argue that rich body and contextual cues also shape emotion inferences from faces, often based on experimentally combined face images and contextual cues. However, people may use a different strategy to infer emotions in real-world contexts where rich cues naturally co-occur. Here, we had US representative participants (N = 475) view everyday images (N = 1,125) and infer the targets’ facial expressions of 8 emotions. We modeled how rich cues in the images quantified by computational algorithms and US representative participants (N = 1,093) predicted inferences of facial expressions of emotions. We found significant predictions by not only facial cues (e.g., action units, facial structure) but also body cues (e.g., posture, high-level actions) and contexts (e.g., environmental valence, gist). However, these predictions were based on different mechanisms: Facial and body actions carried unique information that was unexplainable by other co-occurring cues, whereas other cues primarily carried information that was shared. Given the differential mechanisms, we investigated whether integrating merely cues that carried unique information was sufficient for predicting human emotion inferences. We built artificial neural networks that allowed for process-level manipulations: by lesioning artificial neurons, we found that neurons that were crucial for human-like inferences integrated rich cues asymmetrically beyond facial and body actions. Together, these findings suggest that naturalistic inferences of facial expressions of emotions may be driven by facial and body actions that are contextualized by other co-occurring cues to support a coherent interpretation.