EISNet: A Multi-Modal Fusion Network for Semantic Segmentation With Events and Images
- 21 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 8639-8650
Abstract
Bio-inspired event cameras record a scene as sparse and asynchronous “events” by detecting per-pixel brightness changes. Such cameras show great potential in challenging scene understanding tasks, benefiting from the imaging advantages of high dynamic range and high temporal resolution. Considering the complementarity between event and standard cameras, we propose a multi-modal fusion network (EISNet) to improve the semantic segmentation performance. The key challenges of this topic lie in ( i ) how to encode event data to represent accurate scene information and ( ii ) how to fuse multi-modal complementary features by considering the characteristics of two modalities. To solve the first challenge, we propose an Activity-Aware Event Integration Module (AEIM) to convert event data into frame-based representations with high-confidence details via scene activity modeling. To tackle the second challenge, we introduce the Modality Recalibration and Fusion Module (MRFM) to recalibrate modal-specific representations and then aggregate multi-modal features at multiple stages. MRFM learns to generate modal-oriented masks to guide the merging of complementary features, achieving adaptive fusion. Based on these two core designs, our proposed EISNet adopts an encoder-decoder transformer architecture for accurate semantic segmentation using events and images. Experimental results show that our model outperforms state-of-the-art methods by a large margin on event-based semantic segmentation datasets. The code is publicly available at https://github.com/bochenxie/EISNet .Keywords
Funding Information
- Research Grants Council of Hong Kong (CityU11213420, CityU11206122)
- National Natural Science Foundation of China (62173286, 62203024, 61976191)
- R&D Program of Beijing Municipal Education Commission (KM202310005027)
This publication has 45 references indexed in Scilit:
- Event Tubelet Compressor: Generating Compact Representations for Event-Based Action Recognition2022
- Event-Based Fusion for Motion Deblurring with Cross-modal Attention2022
- UCTNet: Uncertainty-Aware Cross-Modal Transformer Network for Indoor RGB-D Semantic Segmentation2022
- ESS: Learning Event-Based Semantic Segmentation from Still Images2022
- Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation2020
- Unsupervised Event-Based Optical Flow Using Motion Compensation2019
- Unified Perceptual Parsing for Scene Understanding2018
- Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars2018
- Deep Residual Learning for Image Recognition2016
- Indoor Segmentation and Support Inference from RGBD Images2012