Learning realistic lip motions for humanoid face robots
Top Cited Papers
- 14 January 2026
- journal article
- Published by American Association for the Advancement of Science (AAAS) in Science Robotics
- Vol. 11 (110) , eadx3017
Abstract
Lip motion represents outsized importance in human communication, capturing nearly half of our visual attention during conversation. Yet anthropomorphic robots often fail to achieve lip-audio synchronization, resulting in clumsy and lifeless lip behaviors. Two fundamental barriers underlay this challenge. First, robotic lips typically lack the mechanical complexity required to reproduce nuanced human mouth movements; second, existing synchronization methods depend on manually predefined movements and rules, restricting adaptability and realism. Here, we present a humanoid robot face designed to overcome these limitations, featuring soft silicone lips actuated by a 10–degree-of-freedom mechanism. To achieve lip synchronization without predefined movements, we used a self-supervised learning pipeline based on a variational autoencoder (VAE) combined with a facial action transformer, enabling the robot to autonomously infer more realistic lip trajectories directly from speech audio. Our experimental results suggest that this method outperforms simple heuristics like amplitude-based baselines in achieving more visually coherent lip-audio synchronization. Furthermore, the learned synchronization successfully generalizes across multiple linguistic contexts, enabling robot speech articulation in 10 languages unseen during training.Keywords
Funding Information
- National Science Foundation (2112085)
This publication has 39 references indexed in Scilit:
- Egocentric visual self-modeling for autonomous robot dynamics prediction and adaptationnpj Robotics, 2025
- Teaching robots to build simulations of themselvesNature Machine Intelligence, 2025
- LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization2021
- A Novel Speech to Mouth Articulation System for Realistic Humanoid RobotsJournal of Intelligent & Robotic Systems, 2021
- Facially expressive humanoid robotic faceHardwareX, 2020
- Capture, Learning, and Synthesis of 3D Speaking Styles2019
- Learning multiple layers of representationTrends in Cognitive Sciences, 2007
- Hearing Lips and Seeing Voices: How Cortical Areas Supporting Speech Production Mediate Audiovisual Speech PerceptionCerebral Cortex, 2007
- Audio-visual integration in multimodal communicationProceedings of the IEEE, 1998
- Hearing lips and seeing voicesNature, 1976