SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis
Preprint
Abstract
Although lip-to-speech synthesis (L2S) has achieved significant progress in recent years, current state-of-the-art methods typically rely on intermediate representations such as mel-spectrograms or discrete self-supervised learning (SSL) tokens. The potential of latent diffusion models (LDMs) in this task remains largely unexplored. In this paper, we introduce SLD-L2S, a novel L2S framework built upon a hierarchical subspace latent diffusion model. Our method aims to directly map visual lip movements to the continuous latent space of a pre-trained neural audio codec, thereby avoiding the information loss inherent in traditional intermediate representations. The core of our method is a hierarchical architecture that processes visual representations through multiple parallel subspaces, initiated by a subspace decomposition module. To efficiently enhance interactions within and between these subspaces, we design the diffusion convolution block (DiCB) as our network backbone. Furthermore, we employ a reparameterized flow matching technique to directly generate the target latent vectors. This enables a principled inclusion of speech language model (SLM) and semantic losses during training, moving beyond conventional flow matching objectives and improving synthesized speech quality. Our experiments show that SLD-L2S achieves state-of-the-art generation quality on multiple benchmark datasets, surpassing existing methods in both objective and subjective evaluations.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- NaturalL2S: End-to-end high-quality multispeaker lip-to-speech synthesis with differential digital signal processingNeural Networks, 2025
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching2025
- Conv2Former: A Simple Transformer-Style ConvNet for Visual RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- VoxCeleb2: Deep Speaker Recognition2018
- TCD-TIMIT: An Audio-Visual Corpus of Continuous SpeechIEEE Transactions on Multimedia, 2015
- An audio-visual corpus for speech perception and automatic speech recognitionThe Journal of the Acoustical Society of America, 2006