Pixels that Sound
Authors:
- 27 July 2005
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1, 88-95
Abstract
People and animals fuse auditory and visual information to obtain robust perception. A particular benefit of such cross-modal analysis is the ability to localize visual events associated with sound sources. We aim to achieve this using computer-vision aided by a single microphone. Past efforts encountered problems stemming from the huge gap between the dimensions involved and the available data. This has led to solutions suffering from low spatio-temporal resolutions. We present a rigorous analysis of the fundamental problems associated with this task. Then, we present a stable and robust algorithm which overcomes pastdeficiencies. It grasps dynamic audio-visual events with high spatial resolution, and derives a unique solution. The algorithm effectively detects pixels that are associated with the sound, while filtering out other dynamic pixels. It is based on canonical correlation analysis (CCA), where we remove inherent ill-posedness by exploiting the typical spatial sparsity of audio-visual events. The algorithm is simple and ef- ficient thanks to its reliance on linear programming and is free of user-defined parameters. To quantitatively assess the performance, we devise a localization criterion. The algorithm capabilities were demonstrated in experiments, where it overcame substantial visual distractions and audio noise.Keywords
This publication has 17 references indexed in Scilit:
- Speaker Association With Signal-Level Audiovisual FusionIEEE Transactions on Multimedia, 2004
- Conducting Audio Files via Computer VisionLecture Notes in Computer Science, 2004
- Sparse representations in unions of basesIEEE Transactions on Information Theory, 2003
- Appearance models based on kernel canonical correlation analysisPattern Recognition, 2003
- A graphical model for audiovisual object trackingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2003
- Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ 1 minimizationProceedings of the National Academy of Sciences of the United States of America, 2003
- A Computational Model of Early Auditory-Visual Integration2003
- Gated Visual Input to the Central Auditory SystemScience, 2002
- Can recent innovations in harmonic analysis `explain' key findings in natural image statistics?Network: Computation in Neural Systems, 2001
- Enhancement of selective listening by illusory mislocation of speech sounds due to lip-readingNature, 1996