Blind Audiovisual Source Separation Based on Sparse Redundant Representations
Authors:
- 18 May 2010
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 12 (5) , 358-371
Abstract
In this paper, we propose a novel method which is able to detect and separate audiovisual sources present in a scene. Our method exploits the correlation between the video signal captured with a camera and a synchronously recorded one-microphone audio track. In a first stage, audio and video modalities are decomposed into relevant basic structures using redundant representations. Next, synchrony between relevant events in audio and video modalities is quantified. Based on this co-occurrence measure, audiovisual sources are counted and located in the image using a robust clustering algorithm that groups video structures exhibiting strong correlations with the audio. Next periods where each source is active alone are determined and used to build spectral Gaussian mixture models (GMMs) characterizing the sources acoustic behavior. Finally, these models are used to separate the audio signal in periods during which several sources are mixed. The proposed approach has been extensively tested on synthetic and natural sequences composed of speakers and music instruments. Results show that the proposed method is able to successfully detect, localize, separate, and reconstruct present audiovisual sources.Keywords
This publication has 22 references indexed in Scilit:
- Audio Visual Speaker Verification Based on Hybrid Fusion of Cross Modal Features2007
- Oracle estimators for the benchmarking of source separation algorithmsSignal Processing, 2007
- Harmony in Motion2007
- Analysis of multimodal sequences using geometric video representationsSignal Processing, 2006
- Developing an audio-visual speech source separation algorithmSpeech Communication, 2004
- Blind Separation of Speech Mixtures via Time-Frequency MaskingIEEE Transactions on Signal Processing, 2004
- Recent advances in the automatic recognition of audiovisual speechProceedings of the IEEE, 2003
- Moving-Talker, Speaker-Independent Feature Study, and Baseline Results Using the CUAVE Multimodal Speech CorpusEURASIP Journal on Advances in Signal Processing, 2002
- Matching pursuits with time-frequency dictionariesIEEE Transactions on Signal Processing, 1993
- Visual Contribution to Speech Intelligibility in NoiseThe Journal of the Acoustical Society of America, 1954