Audiovisual Voice Activity Detection Based on Microphone Arrays and Color Information
Authors:
- 1 January 2013
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 7 (1) , 147-156
Abstract
Audiovisual voice activity detection is a necessary stage in several problems, such as advanced teleconferencing, speech recognition, and human-computer interaction. Lip motion and audio analysis provide a large amount of information that can be integrated to produce more robust audiovisual voice activity detection (VAD) schemes, as we discuss in this paper. Lip motion is very useful for detecting the active speaker, and in this paper we introduce a new approach for lips and visual VAD. First, the algorithm performs skin segmentation to reduce the search area for lip extraction, and the most likely lip and non-lip regions are detected using a Bayesian approach within the delimited area. Lip motion is then detected using Hidden Markov Models (HMMs) that estimate the likely occurrence of active speech within a temporal window. Audio information is captured by an array of microphones, and the sound-based VAD is related to finding spatio-temporally coherent sound sources through another set of HMMs. To increase the robustness of the proposed system, a late fusion approach is employed to combine the result of each modality (audio and video). Our experimental results indicate that the proposed audiovisual approach presents better results when compared to existing VAD algorithms.Keywords
This publication has 29 references indexed in Scilit:
- Voice activity detection and speaker localization using audiovisual cuesPattern Recognition Letters, 2012
- Multimodal fusion for multimedia analysis: a surveyMultimedia Systems, 2010
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- Audio–Visual Active Speaker Tracking in Cluttered Indoors Environments $^{\ast}$IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2008
- Functional TreesMachine Learning, 2004
- A soft voice activity detector based on a laplacian-gaussian modelIEEE Transactions on Speech and Audio Processing, 2003
- Face detection and location based on skin chrominance and lip chrominance transformation from color imagesPattern Recognition, 2001
- Voice activity detection in nonstationary noiseIEEE Transactions on Speech and Audio Processing, 2000
- Improved noise suppression filter using self-adaptive estimator of probability of speech absenceSignal Processing, 1999
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989