AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
Authors:
Preprint
Abstract
Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Ontology-aware Learning and Evaluation for Audio Tagging2023
- Simple Pooling Front-Ends for Efficient Audio Classification2023
- Full-Band General Audio Synthesis with Score-Based Diffusion2023
- Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation2023
- Separate What You Describe: Language-Queried Audio Source Separation2022
- Neural Vocoder is All You Need for Speech Super-resolution2022
- Image Super-Resolution Via Iterative RefinementIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and AggregationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Towards Robust Speech Super-ResolutionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020