A real-time prototype for small-vocabulary audio-visual ASR
Authors:
- 1 January 2003
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 2, 469
Abstract
We present a prototype for the automatic recognition of audio-visual speech, developed to augment the IBM ViaVoice/spl trade/ speech recognition system. Frontal face, full frame video is captured through a USB 2.0 interface by means of an inexpensive PC camera, and processed to obtain appearance-based visual features. Subsequently, these are combined with audio features, synchronously extracted from the acoustic signal, using a simple discriminant feature fusion technique. On the average, the required computations utilize approximately 67% of a Pentium/spl trade/ 4, 1.8 GHz processor, leaving the remaining resources available to hidden Markov model based speech recognition. Real-time performance is there- fore achieved for small-vocabulary tasks, such as connected-digit recognition. In the paper, we discuss the prototype architecture based on the ViaVoice engine, the basic algorithms employed, and their necessary modifications to ensure real-time performance and causality of the visual front end processing. We benchmark the resulting system performance on stored videos against prior research experiments, and we report a close match between the two.Keywords
This publication has 4 references indexed in Scilit:
- Hierarchical discriminant features for audio-visual LVCSR2001
- Audio-visual speech modeling for continuous speech recognitionIEEE Transactions on Multimedia, 2000
- Visionary Speech: Looking Ahead to Practical Speechreading Systems1996
- Lip synchronization using speech-assisted video processingIEEE Signal Processing Letters, 1995