Speech Enhancement and Recognition in Meetings With an Audio–Visual Sensor Array
Authors:
- Daniel Gatica-PerezDGDaniel Gatica-Perez
- Iain McCowanIMIain McCowan
- 15 October 2007
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech, and Language Processing
- Vol. 15 (8) , 2257-2269
Abstract
This paper addresses the problem of distant speech acquisition in multiparty meetings, using multiple microphones and cameras. Microphone array beamforming techniques present a potential alternative to close-talking microphones by providing speech enhancement through spatial filtering. Beamforming techniques, however, rely on knowledge of the speaker location. In this paper, we present an integrated approach, in which an audio-visual multiperson tracker is used to track active speakers with high accuracy. Speech enhancement is then achieved using microphone array beamforming followed by a novel postfiltering stage. Finally, speech recognition is performed to evaluate the quality of the enhanced speech signal. The approach is evaluated on data recorded in a real meeting room for stationary speaker, moving speaker, and overlapping speech scenarios. The results show that the speech enhancement and recognition performance achieved using our approach are significantly better than a single table-top microphone and are comparable to a lapel microphone for some of the scenarios. The results also indicate that the audio-visual-based system performs significantly better than audio-only system, both in terms of enhancement and recognition. This reveals that the accurate speaker tracking provided by the audio-visual sensor array proved beneficial to improve the recognition performance in a microphone array-based speech recognition system.Keywords
This publication has 45 references indexed in Scilit:
- Real-Time Speaker Tracking Using Particle Filter Sensor FusionProceedings of the IEEE, 2004
- Detection and Separation of Speech Event Using Audio and Video Information Fusion and Its Application to Robust Speech InterfaceEURASIP Journal on Advances in Signal Processing, 2004
- Microphone array post-filter based on noise field coherenceIEEE Transactions on Speech and Audio Processing, 2003
- Audiovisual localization of multiple speakers in a video teleconferencing settingInternational Journal of Imaging Systems and Technology, 2003
- Two decades of array signal processing research: the parametric approachIEEE Signal Processing Magazine, 1996
- Human and machine recognition of faces: a surveyProceedings of the IEEE, 1995
- Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chainsIEEE Transactions on Speech and Audio Processing, 1994
- Robust adaptive beamformingIEEE Transactions on Acoustics, Speech, and Signal Processing, 1987
- Practical supergainIEEE Transactions on Acoustics, Speech, and Signal Processing, 1986
- Measurement of Correlation Coefficients in Reverberant Sound FieldsThe Journal of the Acoustical Society of America, 1955