HSHaoxiang Shi

Haoxiang Shi

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Haoxiang Shi has a total of 6 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Responsive Listening Head Synthesis with 3DMM and Dual-Stream Prediction Network

    by Jun Yu, Shenshen Du, Haoxiang Shi, Yiwei Zhang, Renbin Su, Zhongpeng Cai, Lei Wang

    In a conversation, it is crucial for the listener to provide appropriate reactions to the speaker, as the dialogue becomes challenging to sustain without the listener's involvement. Consequently, responsive listening head synthesis has become an important task. However, the existing methods fail to adequately utilize the audio and video of the speaker to generate listening heads, resulting in unnatural or even distorted generated videos. In this paper, we propose a framework to effectively encode audio and video features to address this problem. The framework includes speaker audio encoder, speaker video encoder, a Dual-Stream Prediction Network and a rendering network. The audio encoder utilizes a transformer encoder to encode the audio features, enabling better focus on the contextual features of long audio inputs. The speaker video encoder uses a 3D morphable model (3DMM) to extract the speaker's video features. Then we fuse the speaker's audio-video features with the listener's identity information extracted by 3DMM to preserve the listener's style. Additionally, a Dual-Stream Prediction Network is introduced to further enhance the prediction capability of the network. Finally, a rendering network is used to generate the listening heads based on the result of the prediction network. Comprehensive experiments demonstrate that our approach is capable of generating responsive listening heads with higher visual quality, better naturalness and higher reconstruction fidelity.