Let's Face It
Authors:
Open Access
- 19 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Abstract
To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures - represented by highly expressive FLAME parameters - in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the different modalities in the synthesized output. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code are available at: https://jonepatr.github.io/lets_face_it/.Keywords
All Related Versions
Funding Information
- Knut och Alice Wallenbergs Stiftelse (WASP)
- Stiftelsen för Strategisk Forskning (RIT15-0107)
This publication has 19 references indexed in Scilit:
- Novel Realizations of Speech-Driven Head Movements with Generative Adversarial Networks2018
- Limiting the reconstruction capability of generative neural network using negative learning2017
- Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory2017
- Predicting Head Pose in Dyadic Conversation2017
- Predicting Head Pose from Speech with a Conditional Variational Autoencoder2017
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Representing Affective Facial Expressions for Robots and Embodied Conversational Agents by Facial LandmarksInternational Journal of Social Robotics, 2013
- An Examination of Whether People Prefer Agents Whose Gestures Mimic Their Own2013
- Face recognition based on fitting a 3D morphable modelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003