Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 5025-5034
Abstract
Recently, the AI community has made significant strides in developing powerful foundation models, driven by large-scale multimodal datasets. However, for audio representation learning, existing datasets suffer from limitations in the following aspects: insufficient volume, simplistic content, and arduous collection procedures. To establish an audio dataset with high-quality captions, we propose an innovative, automatic approach leveraging multimodal inputs, such as video frames, audio streams. Specifically, we construct a large-scale, high-quality, audio-language dataset, named as Auto-ACD, comprising over 1.5M audio-text pairs. We exploit a series of pre-trained models or APIs, to determine audio-visual synchronisation, generate image captions, object detection, or audio tags for specific videos. Subsequently, we employ LLM to paraphrase a congruent caption for each audio, guided by the extracted multi-modality clues. To demonstrate the effectiveness of the proposed dataset, we train widely used models on our dataset and show performance improvement on various downstream tasks, for example, audio-language retrieval, audio captioning, zero-shot classification. In addition, we establish a novel benchmark with environmental information and provide a benchmark for audio-text tasks.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (No.2022ZD0161400)
This publication has 41 references indexed in Scilit:
- Audio-Visual Grouping Network for Sound Localization from Mixtures2023
- Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning2023
- AutoAD: Movie Description in Context2023
- Audio–Visual Segmentation2022
- Mix and Localize: Localizing Sound Sources in Mixtures2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Localizing Visual Sounds the Hard Way2021
- Audio-Visual Scene Analysis with Self-Supervised Multisensory Features2018
- An Overview of Audio Event Detection Methods from Feature Extraction to ClassificationApplied Artificial Intelligence, 2017
- ImageNet: A large-scale hierarchical image database2009