Multimodal speaker identification using an adaptive classifier cascade based on modality reliability
- 19 September 2005
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 7 (5) , 840-852
Abstract
We present a multimodal open-set speaker identification system that integrates information coming from audio, face and lip motion modalities. For fusion of multiple modalities, we propose a new adaptive cascade rule that favors reliable modality combinations through a cascade of classifiers. The order of the classifiers in the cascade is adaptively determined based on the reliability of each modality combination. A novel reliability measure, that genuinely fits to the open-set speaker identification problem, is also proposed to assess accept or reject decisions of a classifier. A formal framework is developed based on probability of correct decision for analytical comparison of the proposed adaptive rule with other classifier combination rules. The proposed adaptive rule is more robust in the presence of unreliable modalities, and outperforms the hard-level max rule and soft-level weighted summation rule, provided that the employed reliability measure is effective in assessment of classifier decisions. Experimental results that support this assertion are provided.Keywords
This publication has 26 references indexed in Scilit:
- Audio-visual speaker recognition using time-varying stream reliability prediction2004
- BIOMET: A Multimodal Person Authentication Database Including Face, Voice, Fingerprint, Hand and Signature Modalities2003
- Information fusion and decision cascading for audio-visual speaker recognition based on time-varying stream reliability prediction2003
- Combining independent and unbiased classifiers using weighted average2002
- Adaptive Fusion of Speech and Lip Information for Robust Speaker IdentificationDigital Signal Processing, 2001
- Large-vocabulary audio-visual speech recognition: a summary of the Johns Hopkins Summer 2000 Workshop2001
- BiolD: a multimodal biometric identification systemComputer, 2000
- Multimodal decision-level fusion for person authenticationIEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 1999
- Acoustic-labial speaker verificationPattern Recognition Letters, 1997
- Person identification using multiple cuesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1995