Audio-visual speaker tracking with importance particle filters

Authors:
  • DGD. Gatica-Perez
    D. Gatica-Perez
  • GLG. Lathoud
    G. Lathoud
  • IMI. McCowan
    I. McCowan
  • JOJ.-M. Odobez
    J.-M. Odobez
  • DMD. Moore
    D. Moore

Abstract
We present a probabilistic method for audio-visual (AV) speaker tracking, using an uncalibrated wide-angle camera and a micro- phone array. The algorithm fuses 2-D object shape and audio information via importance particle filters (I-PFs), allowing for the asymmetrical integration of AV information in a way that ef- ficiently exploits the complementary features of each modality. Audio localization information is used to generate an importance sampling (IS) function, which guides the random search process of a particle filter towards regions of the configuration space likely to contain the true configuration (a speaker). The measurement pro- cess integrates contour-based and audio observations, which re- sults in reliable head tracking in realistic scenarios. We show that imperfect single modalities can be combined into an algorithm that automatically initializes and tracks a speaker, switches between multiple speakers, tolerates visual clutter, and recovers from total AV object occlusion, in the context of a multimodal meeting room. This work was carried out in the framework of the Swiss National Center of Competence in Research (NCCR) on Interactive Multimodal In- formation Management (IM)2, and the European project M4 through the Swiss Federal Office for Education and Science. Sampling from dynamics (prediction) and measuring (updat- ing) are SMC stages in which data fusion can be introduced. Cur- rent formulations for AV speaker tracking fuse audio and video only at the measurement level (9), (10), thus leading to symmet- rical models in which each modality accounts for the same rele- vance, solely depending on the dynamical model to generate candi- date configurations. Additionally, AV sensors (cameras and micro- phones) tend to be independently calibrated for state modeling and measuring in 2-D or 3-D. Such formulations tend to overlook two important features of AV data. First, audio is a strong cue to model discontinuities that clearly violate usual assumptions in dynamics (including speaker turns), speaker occlusion, and (re)initialization. Its use for sampling would therefore bring benefits to modeling re- alistic situations. Second, even though audio might be inaccurate, and visual calibration can be erroneous due to camera distortion, their joint occurrence tends to be consistent, and can be learned in