Generation of speech and facial animation with controllable articulatory effort for amusing conversational characters
Open Access
- 19 September 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Abstract
Engaging embodied conversational agents need to generate expressive behavior in order to be believable in socializing interactions. We present a system that can generate spontaneous speech with supporting lip movements. The neural conversational TTS voice is trained on a multi-style speech corpus that has been prosodically tagged (pitch and speaking rate) and transcribed (including tokens for breathing, fillers and laughter). We introduce a speech animation algorithm where articulatory effort can be adjusted. The facial animation is driven by time-stamped phonemes and prominence estimates from the synthesised speech waveform to modulate the lip-and jaw movements accordingly. In objective evaluations we show that the system is able to generate speech and facial animation that vary in articulation effort. In subjective evaluations we compare our conversational TTS system's capability to deliver jokes with a commercial TTS. Both system succeeded equally good.Keywords
Funding Information
- Swedish Research Council (VR-2020-0239)
- Swedish Research Counci (VR-2019-05003)
- Riksbankens Jubileumsfond (P20-0298)
This publication has 40 references indexed in Scilit:
- MACH2013
- Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides2010
- Embodied conversational agents in computer assisted language learningSpeech Communication, 2009
- ‘It's just like you talk to a friend’ relational agents for older adultsInteracting with Computers, 2005
- Providing Computer Game Characters with Conversational AbilitiesLecture Notes in Computer Science, 2005
- A Conversational Agent as Museum Guide – Design and Evaluation of a Real-World Application2005
- FearNot! – An Experiment in Emergent Narrative2005
- Employing ai methods to control the behavior of animated interface agentsApplied Artificial Intelligence, 1999
- Animated agents for procedural training in virtual reality: Perception, cognition, and motor controlApplied Artificial Intelligence, 1999
- Numerical Model of CoarticulationThe Journal of the Acoustical Society of America, 1967