Matthew Kit Khinn Teng

Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.

Micro-expressions are very brief, involuntary facial expressions that reveal hidden emotions, lasting less than a second, while macro-expressions are more prolonged facial expressions that align with a person’s conscious emotions, typically lasting several seconds. Micro-expressions are difficult to detect in lengthy videos because they have tiny amplitudes, short durations, and frequently coexist alongside macro-expressions. Nevertheless, micro- and macro-expression analysis has sparked interest in researchers. Existing methods use optical flow features to capture the temporal differences. However, these optical flow features are limited to two successive images only. To address this limitation, this paper proposes LGNMNet-RF, which integrates a Lite General Network with MagFace CNN and a Random Forest classifier to predict micro-expression intervals. Our approach leverages Motion History Images (MHI) to capture temporal patterns across multiple frames, offering a more comprehensive representation of facial dynamics than optical flow-based methods, which are restricted to two successive frames. The novelty of our approach lies in the combination of MHI with MagFace CNN, which improves the discriminative power of facial micro-expression detection, and the use of a Random Forest classifier to enhance interval prediction accuracy. The evaluation results show that this method outperforms baseline techniques, achieving micro-expression F1-scores of 0.3019 on CAS(ME)² and 0.3604 on SAMM-LV. The results of our experiment indicate that MHI offers a viable alternative to optical flow-based methods for micro-expression detection.