DialogueNeRF: towards realistic avatar face-to-face conversation video generation
Open Access
- 7 August 2024
- journal article
- research article
- Published by Springer Nature in Visual Intelligence
- Vol. 2 (1) , 1-15
Abstract
Conversation is an essential component of virtual avatar activities in the metaverse. With the development of natural language processing, significant breakthroughs have been made in text and voice conversation generation. However, face-to-face conversations account for the vast majority of daily conversations, while most existing methods focused on single-person talking head generation. In this work, we take a step further and consider generating realistic face-to-face conversation videos. Conversation generation is more challenging than single-person talking head generation, because it requires not only the generation of photo-realistic individual talking heads, but also the listener’s response to the speaker. In this paper, we propose a novel unified framework based on the neural radiance field (NeRF) to address these challenges. Specifically, we model both the speaker and the listener with a NeRF framework under different conditions to control individual expressions. The speaker is driven by the audio signal, while the response of the listener depends on both visual and acoustic information. In this way, face-to-face conversation videos are generated between human avatars, with all the interlocutors modeled within the same network. Moreover, to facilitate future research on this task, we also collected a new human conversation dataset containing 34 video clips. Quantitative and qualitative experiments evaluate our method in different aspects, e.g., image quality, pose sequence trend, and natural rendering of the scene in the generated videos. Experimental results demonstrate that the avatars in the resulting videos are able to carry on a realistic conversation, and maintain individual styles.Keywords
All Related Versions
This publication has 55 references indexed in Scilit:
- Hierarchical painter: Chinese landscape painting restoration with fine-grained stylesVisual Intelligence, 2023
- PBR-GAN: Imitating Physically-Based Rendering With Generative Adversarial NetworksIEEE Transactions on Circuits and Systems for Video Technology, 2023
- Deep Face Video Inpainting via UV MappingIEEE Transactions on Image Processing, 2023
- AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis2021
- Realistic Speech-Driven Facial Animation with GANsInternational Journal of Computer Vision, 2019
- Audio-driven facial animation by joint end-to-end learning of pose and emotionACM Transactions on Graphics, 2017
- Synthesizing ObamaACM Transactions on Graphics, 2017
- Speaker Diarization: A Review of Recent ResearchIEEE Transactions on Audio, Speech, and Language Processing, 2012
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Using Language1996