Audio-visual speaker tracking with importance particle filters
Authors:
- D. Gatica-PerezDGD. Gatica-Perez
- G. LathoudGLG. Lathoud
- I. McCowanIMI. McCowan
- J.-M. OdobezJOJ.-M. Odobez
- D. MooreDMD. Moore
- 3 June 2004
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 2, 25
Abstract
We present a probabilistic method for audio-visual (AV) speaker tracking, using an uncalibrated wide-angle camera and a micro- phone array. The algorithm fuses 2-D object shape and audio information via importance particle filters (I-PFs), allowing for the asymmetrical integration of AV information in a way that ef- ficiently exploits the complementary features of each modality. Audio localization information is used to generate an importance sampling (IS) function, which guides the random search process of a particle filter towards regions of the configuration space likely to contain the true configuration (a speaker). The measurement pro- cess integrates contour-based and audio observations, which re- sults in reliable head tracking in realistic scenarios. We show that imperfect single modalities can be combined into an algorithm that automatically initializes and tracks a speaker, switches between multiple speakers, tolerates visual clutter, and recovers from total AV object occlusion, in the context of a multimodal meeting room. This work was carried out in the framework of the Swiss National Center of Competence in Research (NCCR) on Interactive Multimodal In- formation Management (IM)2, and the European project M4 through the Swiss Federal Office for Education and Science. Sampling from dynamics (prediction) and measuring (updat- ing) are SMC stages in which data fusion can be introduced. Cur- rent formulations for AV speaker tracking fuse audio and video only at the measurement level (9), (10), thus leading to symmet- rical models in which each modality accounts for the same rele- vance, solely depending on the dynamical model to generate candi- date configurations. Additionally, AV sensors (cameras and micro- phones) tend to be independently calibrated for state modeling and measuring in 2-D or 3-D. Such formulations tend to overlook two important features of AV data. First, audio is a strong cue to model discontinuities that clearly violate usual assumptions in dynamics (including speaker turns), speaker occlusion, and (re)initialization. Its use for sampling would therefore bring benefits to modeling re- alistic situations. Second, even though audio might be inaccurate, and visual calibration can be erroneous due to camera distortion, their joint occurrence tends to be consistent, and can be learned inKeywords
This publication has 6 references indexed in Scilit:
- Sequential Monte Carlo fusion of sound and vision for speaker tracking2002
- Multimodal 3-D tracking and event detection via the particle filter2002
- Multimodal speaker detection using error feedback dynamic Bayesian networks2002
- Sequential Monte Carlo Methods in Practice2001
- Active Contours1998
- Bayesian Data Analysis1995