Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation
- 3 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 141-153
Abstract
As a combination of visual and audio signals, video is inherently multi-modal. However, existing video generation methods are primarily intended for the synthesis of visual frames, whereas audio signals in realistic videos are disregarded. In this work, we concentrate on a rarely investigated problem of text-guided sounding video generation and propose the Sounding Video Generator (SVG), a unified framework for generating realistic videos along with audio signals. Specifically, we present the SVG-VQGAN to transform visual frames and audio mel-spectrograms into discrete tokens. SVG-VQGAN applies a novel hybrid contrastive learning method to model inter-modal and intra-modal consistency and improve the quantized representations. A cross-modal attention module is employed to extract associated features of visual frames and audio signals for contrastive learning. Then, a Transformer-based decoder is used to model associations between texts, visual frames, and audio signals at token level for auto-regressive sounding video generation. AudioSet-Cap, a human annotated text-video-audio paired dataset, is produced for training SVG. Experimental results demonstrate the superiority of our method when compared with existing text-to-video generation methods as well as audio generation methods on Kinetics and VAS datasets.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2020AAA0106400)
- National Natural Science Foundation of China (U21B2043, 62102419, 62102416)
- CAAI-Huawei MindSpore Open Fund
This publication has 32 references indexed in Scilit:
- Deep Collaborative Embedding for Social Image UnderstandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Temporal Generative Adversarial Nets with Singular Value Clipping2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Audio Set: An ontology and human-labeled dataset for audio events2017
- Weakly Supervised Deep Matrix Factorization for Social Image UnderstandingIEEE Transactions on Image Processing, 2016
- Visually Indicated Sounds2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Multiobjective evolutionary algorithms to identify highly autocorrelated areas: the case of spatial distribution in financially compromised farmsAnnals of Operations Research, 2011
- ImageNet: A large-scale hierarchical image database2009
- Long Short-Term MemoryNeural Computation, 1997