I Hear Your True Colors: Image Guided Audio Generation
Authors:
Top Cited Papers
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
We propose IM2WAV, an image guided open-domain audio generation system. Given an input image or a sequence of images, IM2WAV generates a semantically relevant sound. IM2WAV is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP (Contrastive Language–Image Pre-training) [1] model embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that IM2WAV significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as IM-AGEHEAR. IMAGEHEAR can be used as a benchmark for evaluating future image-to-audio models. Samples and code can be found under the following link.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- Wav2CLIP: Learning Robust Audio Representations from Clip2022
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- Generating Visually Aligned Sound From VideosIEEE Transactions on Image Processing, 2020
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Visual to Sound: Generating Natural Sound for Videos in the Wild2018
- Audio Set: An ontology and human-labeled dataset for audio events2017
- CNN architectures for large-scale audio classification2017
- The influence of bilingualism on third language acquisition: Focus on multilingualismLanguage Teaching, 2011