Enhancing Temporal Coherence in Image-to-Video Facial Expression Synthesis: A Dual-Loss Framework for Smoother Generation
Open Access
- 22 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 170876-170894
Abstract
Facial expression synthesis for video sequences presents significant challenges in terms of maintaining temporal coherence while preserving identity and expression accuracy. Most current approaches primarily focus on individual frame quality. This results in videos with unnatural transitions, flickering artifacts, and inconsistent expression dynamics. This paper presents a novel method for improving temporal coherence in facial expression image-to-video synthesis. The proposed approach uses a dual-focus framework that explicitly models both pixel-level transitions and geometric consistency between consecutive frames. Our method includes two specialized loss functions: a frame difference loss and a landmark consistency loss. We also implement a two-stage training strategy and a frame-blending post-processing technique. Extensive testing on the MUG Facial Expression Database demonstrates that the enhanced-coherence approach outperforms baseline methods and competes with other approaches from the literature. The proposed method has the lowest Average Content Distance score of all the approaches tested. This indicates smoother frame transitions while maintaining high visual quality. Subjective evaluation with 140 participants confirms that the synthesized expressions are more realistic and expressive. Our approach outperformed real videos in recognition rates for certain emotion expressions, particularly disgust and sadness. These findings demonstrate the effectiveness of explicitly modeling temporal coherence in generating realistic facial expression videos. This has promising applications in therapy, entertainment, and human-computer interaction.Keywords
Funding Information
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), Brazil (001)
- Dean’s Office for Research of the University of São Paulo (18.5.245.86.7)
- Brazilian National Council of Scientific and Technological Development (307710/2022-0)
- São Paulo Research Foundation (FAPESP): National Institute of Science and Technology—Medicine Assisted by Scientific Computing (14/50889-7, 2020/01992-0)
This publication has 67 references indexed in Scilit:
- Generative Adversarial Nets2023
- Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forwardApplied Intelligence, 2022
- Facial expression synthesis based on similar facesMultimedia Tools and Applications, 2021
- Unsupervised video-to-video translation with preservation of frame modification tendencyThe Visual Computer, 2020
- Deep learning for face image synthesis and semantic manipulations: a review and future perspectivesArtificial Intelligence Review, 2020
- GANimation: One-Shot Anatomically Consistent Facial AnimationInternational Journal of Computer Vision, 2019
- Recycle-GAN: Unsupervised Video Retargeting2018
- Assessing the Utility of a Virtual Environment for Enhancing Facial Affect Recognition in Adolescents with AutismJournal of Autism and Developmental Disorders, 2014
- Recognition of Emotions in Autism: A Formal Meta-AnalysisJournal of Autism and Developmental Disorders, 2012
- Building highly realistic facial modeling and animation: a surveyThe Visual Computer, 2007