Haoxiang Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Haoxiang Shi has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Jun Yu
University of Science and Technology of China, Hefei, China
1 shared publication - Lei Wang
University of Science and Technology of China, Hefei, China
1 shared publication - Renbin Su
University of Science and Technology of China, Hefei, China
1 shared publication - Shenshen Du
University of Science and Technology of China, Hefei, China
1 shared publication - Yiwei Zhang
University of Science and Technology of China, Hefei, China
1 shared publication - Zhongpeng Cai
University of Science and Technology of China, Hefei, China
1 shared publication
Latest Publications
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-143
Responsive Listening Head Synthesis with 3DMM and Dual-Stream Prediction Network
In a conversation, it is crucial for the listener to provide appropriate reactions to the speaker, as the dialogue becomes challenging to sustain without the listener's involvement. Consequently, responsive listening head synthesis has become an important task. However, the existing methods fail to adequately utilize the audio and video of the speaker to generate listening heads, resulting in unnatural or even distorted generated videos. In this paper, we propose a framework to effectively encode audio and video features to address this problem. The framework includes speaker audio encoder, speaker video encoder, a Dual-Stream Prediction Network and a rendering network. The audio encoder utilizes a transformer encoder to encode the audio features, enabling better focus on the contextual features of long audio inputs. The speaker video encoder uses a 3D morphable model (3DMM) to extract the speaker's video features. Then we fuse the speaker's audio-video features with the listener's identity information extracted by 3DMM to preserve the listener's style. Additionally, a Dual-Stream Prediction Network is introduced to further enhance the prediction capability of the network. Finally, a rendering network is used to generate the listening heads based on the result of the prediction network. Comprehensive experiments demonstrate that our approach is capable of generating responsive listening heads with higher visual quality, better naturalness and higher reconstruction fidelity.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-143
Open AccessResponsive Listening Head Synthesis with 3DMM and Dual-Stream Prediction Network
In a conversation, it is crucial for the listener to provide appropriate reactions to the speaker, as the dialogue becomes challenging to sustain without the listener's involvement. Consequently, responsive listening head synthesis has become an important task. However, the existing methods fail to adequately utilize the audio and video of the speaker to generate listening heads, resulting in unnatural or even distorted generated videos. In this paper, we propose a framework to effectively encode audio and video features to address this problem. The framework includes speaker audio encoder, speaker video encoder, a Dual-Stream Prediction Network and a rendering network. The audio encoder utilizes a transformer encoder to encode the audio features, enabling better focus on the contextual features of long audio inputs. The speaker video encoder uses a 3D morphable model (3DMM) to extract the speaker's video features. Then we fuse the speaker's audio-video features with the listener's identity information extracted by 3DMM to preserve the listener's style. Additionally, a Dual-Stream Prediction Network is introduced to further enhance the prediction capability of the network. Finally, a rendering network is used to generate the listening heads based on the result of the prediction network. Comprehensive experiments demonstrate that our approach is capable of generating responsive listening heads with higher visual quality, better naturalness and higher reconstruction fidelity.