Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings
Authors:
Top Cited Papers
- 1 May 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3852-3856
Abstract
A considerable challenge in applying deep learning to audio classification is the scarcity of labeled data. An increasingly popular solution is to learn deep audio embeddings from large audio collections and use them to train shallow classifiers using small labeled datasets. Look, Listen, and Learn (L 3 -Net) is an embedding trained through self-supervised learning of audio-visual correspondence in videos as opposed to other embeddings requiring labeled data. This framework has the potential to produce powerful out-of-the-box embeddings for downstream audio classification tasks, but has a number of unexplained design choices that may impact the embeddings' behavior. In this paper we investigate how L 3 -Net design choices impact the performance of downstream audio classifiers trained with these embeddings. We show that audio-informed choices of input representation are important, and that using sufficient data for training the embedding is key. Surprisingly, we find that matching the content for training the embedding to the downstream task is not beneficial. Finally, we show that our best variant of the L 3 -Net embedding outperforms both the VGGish and SoundNet embeddings, while having fewer parameters and being trained on less data. Our implementation of the L 3 -Net embedding model as well as pre-trained models are made freely available online.Keywords
This publication has 18 references indexed in Scilit:
- I-SED2017
- Large-scale audio event discovery in one million YouTube videos2017
- CNN architectures for large-scale audio classification2017
- Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound ClassificationIEEE Signal Processing Letters, 2017
- Deep Metric Learning Using Triplet Network2015
- Environmental sound classification with convolutional neural networks2015
- Detection and Classification of Acoustic Scenes and EventsIEEE Transactions on Multimedia, 2015
- A Dataset and Taxonomy for Urban Sound Research2014
- Individual Comparisons by Ranking MethodsBiometrics Bulletin, 1945
- A Scale for the Measurement of the Psychological Magnitude PitchThe Journal of the Acoustical Society of America, 1937