Event-based Lip Reading with Triplane Fusion Network
Open Access
- 23 February 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Abstract
Lip reading via event cameras could capture subtle and similar lip motions with large dynamic range and microsecond latency, offering high temporal resolution over traditional frame-based cameras. However, existing approaches often overlook the exploitation of distinctive lip motion patterns, instead opting to adapt existing video recognition architectures. In this paper, we propose an event-specified framework named the Triplane Fusion Network (TF-Net) to harness lip movements through analysis from three distinct but complementary views. Specifically, while adhering to the standard XYT view, we further incorporate two additional perspectives: XT and YT, aiming to capitalize on the unique streaming nature of events over time. Given the three views, TF-Net contains multiple expert blocks for every view and mutual information exchange blocks that facilitate the multi-directional exchange of motion information among the different views. We observe that these three views complement each other, further enhancing the learning of the event-specified distribution. Extensive experiments validate the effectiveness of the proposed method, surpassing +1.6% and +2.3% accuracy over other competitive methods on the real-world dataset DVS-Lip and the synthetic dataset Modality, respectively.Keywords
This publication has 33 references indexed in Scilit:
- TD3Net: A temporal densely connected multi-dilated convolutional network for lipreadingJournal of Visual Communication and Image Representation, 2025
- LipFormer: Learning to Lipread Unseen Speakers Based on Visual-Landmark TransformersIEEE Transactions on Circuits and Systems for Video Technology, 2023
- Importance-Aware Information Bottleneck Learning Paradigm for Lip ReadingIEEE Transactions on Multimedia, 2022
- EDFLOW: Event Driven Optical Flow Camera With Keypoint Detection and Adaptive Block MatchingIEEE Transactions on Circuits and Systems for Video Technology, 2022
- Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip ReadingIEEE Transactions on Multimedia, 2021
- Deep Audio-Visual Speech RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- An audio-visual corpus for multimodal automatic speech recognitionJournal of Intelligent Information Systems, 2017
- Asynchronous Binaural Spatial Audition Sensor With 2$\,\times\,$64$\,\times\,$4 Channel OutputIEEE Transactions on Biomedical Circuits and Systems, 2013
- A 128$\,\times$128 1.5% Contrast Sensitivity 0.9% FPN 3 µs Latency 4 mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance PreamplifiersIEEE Journal of Solid-State Circuits, 2013
- Accurate and Quasi-Automatic Lip TrackingIEEE Transactions on Circuits and Systems for Video Technology, 2004