Shenshen Du
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Shenshen Du has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Jun Yu
University of Science and Technology of China, Hefei, China
3 shared publications - Lei Wang
University of Science and Technology of China, Hefei, China
3 shared publications - Zhongpeng Cai
University of Science and Technology of China, Hefei, China
2 shared publications - Fang Gao
School of Electrical Engineering, Guangxi University, Nanning, China
1 shared publication - Haoxiang Shi
University of Science and Technology of China, Hefei, China
1 shared publication - Jing Xiao
Ping An Technology, Shenzhen, China
1 shared publication
Latest Publications
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-142
Image- and Instance-Level Data Augmentation for Occluded Instance Segmentation
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-142
Open AccessImage- and Instance-Level Data Augmentation for Occluded Instance Segmentation
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-143
Responsive Listening Head Synthesis with 3DMM and Dual-Stream Prediction Network
In a conversation, it is crucial for the listener to provide appropriate reactions to the speaker, as the dialogue becomes challenging to sustain without the listener's involvement. Consequently, responsive listening head synthesis has become an important task. However, the existing methods fail to adequately utilize the audio and video of the speaker to generate listening heads, resulting in unnatural or even distorted generated videos. In this paper, we propose a framework to effectively encode audio and video features to address this problem. The framework includes speaker audio encoder, speaker video encoder, a Dual-Stream Prediction Network and a rendering network. The audio encoder utilizes a transformer encoder to encode the audio features, enabling better focus on the contextual features of long audio inputs. The speaker video encoder uses a 3D morphable model (3DMM) to extract the speaker's video features. Then we fuse the speaker's audio-video features with the listener's identity information extracted by 3DMM to preserve the listener's style. Additionally, a Dual-Stream Prediction Network is introduced to further enhance the prediction capability of the network. Finally, a rendering network is used to generate the listening heads based on the result of the prediction network. Comprehensive experiments demonstrate that our approach is capable of generating responsive listening heads with higher visual quality, better naturalness and higher reconstruction fidelity.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 137-143
Open AccessResponsive Listening Head Synthesis with 3DMM and Dual-Stream Prediction Network
In a conversation, it is crucial for the listener to provide appropriate reactions to the speaker, as the dialogue becomes challenging to sustain without the listener's involvement. Consequently, responsive listening head synthesis has become an important task. However, the existing methods fail to adequately utilize the audio and video of the speaker to generate listening heads, resulting in unnatural or even distorted generated videos. In this paper, we propose a framework to effectively encode audio and video features to address this problem. The framework includes speaker audio encoder, speaker video encoder, a Dual-Stream Prediction Network and a rendering network. The audio encoder utilizes a transformer encoder to encode the audio features, enabling better focus on the contextual features of long audio inputs. The speaker video encoder uses a 3D morphable model (3DMM) to extract the speaker's video features. Then we fuse the speaker's audio-video features with the listener's identity information extracted by 3DMM to preserve the listener's style. Additionally, a Dual-Stream Prediction Network is introduced to further enhance the prediction capability of the network. Finally, a rendering network is used to generate the listening heads based on the result of the prediction network. Comprehensive experiments demonstrate that our approach is capable of generating responsive listening heads with higher visual quality, better naturalness and higher reconstruction fidelity.
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 9541-9545
Efficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow Feature
Human facial expressions can convey a great deal of information in daily life. Spotting macro-expression (MaE) and micro-expression (ME) intervals from long video sequences is a difficult challenge. In this paper, we propose an efficient framework for the expression spotting task. This framework consists of three main modules: Face Cropping and Alignment Module (FCAM), optical flow Feature Extraction Module (FEM), and expression Proposal Generation Module (PGM). The noise of optical flow features is reduced by face cropping and alignment, and the Main Directional Mean Optical Flow Feature of the regions of interest is extracted as the feature for expression spotting. Finally, the expression intervals are spotted by our designed expression proposal generation module. Our approach achieves very good results on the SAMM Long Videos and CAS(ME)^2. To demonstrate the transferability of our method, we tested it on the MEGC2023 unseen dataset and finally achieved the third place, proving the effectiveness of our method.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 9541-9545
Open AccessEfficient Micro-Expression Spotting Based on Main Directional Mean Optical Flow Feature
Human facial expressions can convey a great deal of information in daily life. Spotting macro-expression (MaE) and micro-expression (ME) intervals from long video sequences is a difficult challenge. In this paper, we propose an efficient framework for the expression spotting task. This framework consists of three main modules: Face Cropping and Alignment Module (FCAM), optical flow Feature Extraction Module (FEM), and expression Proposal Generation Module (PGM). The noise of optical flow features is reduced by face cropping and alignment, and the Main Directional Mean Optical Flow Feature of the regions of interest is extracted as the feature for expression spotting. Finally, the expression intervals are spotted by our designed expression proposal generation module. Our approach achieves very good results on the SAMM Long Videos and CAS(ME)^2. To demonstrate the transferability of our method, we tested it on the MEGC2023 unseen dataset and finally achieved the third place, proving the effectiveness of our method.