Deep Multimodal Clustering for Unsupervised Audiovisual Learning
Authors:
Top Cited Papers
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9240-9249
Abstract
The seen birds twitter, the running cars accompany with noise, etc. These naturally audiovisual correspondences provide the possibilities to explore and understand the outside world. However, the mixed multiple objects and sounds make it intractable to perform efficient matching in the unconstrained environment. To settle this problem, we propose to adequately excavate audio and visual components and perform elaborate correspondence learning among them. Concretely, a novel unsupervised audiovisual learning model is proposed, named as Deep Multimodal Clustering (DMC),that synchronously performs sets of clustering with multimodal vectors of convolutional maps in different shared spaces for capturing multiple audiovisual correspondences. And such integrated multimodal clustering network can be effectively trained with max-margin loss in the end-to-end fashion. Amounts of experiments in feature evaluation and audiovisual tasks are performed. The results demonstrate that DMC can learn effective unimodal representation, with which the classifier can even outperform human performance. Further, DMC shows noticeable performance in sound localization, multisource detection, and audiovisual understanding.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Learning to See by Moving2015
- Unsupervised Visual Representation Learning by Context Prediction2015
- Unsupervised Learning of Visual Representations Using Videos2015
- ESC2015
- Environmental sound classification with convolutional neural networks2015
- Multisensory perceptual learning and sensory substitutionNeuroscience & Biobehavioral Reviews, 2014
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Multisensory Integration: Space, Time and SuperadditivityCurrent Biology, 2005
- Activation of Auditory Cortex During Silent LipreadingScience, 1997