Audio Retrieval With Natural Language Queries: A Benchmark Study
Top Cited Papers
- 8 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 2675-2685
Abstract
The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a given written description and vice versa. Text-audio retrieval enables users to search large databases through an intuitive interface: they simply issue free-form natural language descriptions of the sound they would like to hear. To study the tasks of text-audio and audio-text retrieval, which have received limited attention in the existing literature, we introduce three challenging new benchmarks. We first construct text-audio and audio-text retrieval benchmarks from the AudioCaps and Clotho audio captioning datasets. Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho. We employ these three benchmarks to establish baselines for cross-modal text-audio and audio-text retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into audio retrieval with free-form text queries. Code, audio features for all datasets used, and the SoundDescs dataset are publicly available at https://github.com/akoepke/audio-retrieval-benchmark.Keywords
All Related Versions
Funding Information
- ERC (853489 - DEXIM)
- DFG 2064/1 (390727645)
- BMBF (FKZ: 01IS18039A)
- EPSRC DTA studentship
- Royal Academy of Engineering (RF\201819\18\163)
- Engineering and Physical Sciences Research Council (EP/T028572/1 Visual AI)
This publication has 57 references indexed in Scilit:
- Sequence to Sequence -- Video to Text2015
- ESC2015
- Chime-home: A dataset for sound source recognition in a domestic environment2015
- Detection and Classification of Acoustic Scenes and EventsIEEE Transactions on Multimedia, 2015
- Freesound technical demo2013
- Audio-Visual-Based Query by Example Video RetrievalMathematical Problems in Engineering, 2013
- Large-scale content-based audio retrieval from text queries2008
- The bag-of-frames approach to audio pattern recognition: A sufficient model for urban soundscapes but not for polyphonic musicThe Journal of the Acoustical Society of America, 2007
- Mixtures of probability experts for audio retrieval and indexing2003
- Content-based classification, search, and retrieval of audioIEEE MultiMedia, 1996