MambaSeg: Harnessing Mamba for Accurate and Efficient Image-Event Semantic Segmentation
Preprint
Abstract
Semantic segmentation is a fundamental task in computer vision with wide-ranging applications, including autonomous driving and robotics. While RGB-based methods have achieved strong performance with CNNs and Transformers, their effectiveness degrades under fast motion, low-light, or high dynamic range conditions due to limitations of frame cameras. Event cameras offer complementary advantages such as high temporal resolution and low latency, yet lack color and texture, making them insufficient on their own. To address this, recent research has explored multimodal fusion of RGB and event data; however, many existing approaches are computationally expensive and focus primarily on spatial fusion, neglecting the temporal dynamics inherent in event streams. In this work, we propose MambaSeg, a novel dual-branch semantic segmentation framework that employs parallel Mamba encoders to efficiently model RGB images and event streams. To reduce cross-modal ambiguity, we introduce the Dual-Dimensional Interaction Module (DDIM), comprising a Cross-Spatial Interaction Module (CSIM) and a Cross-Temporal Interaction Module (CTIM), which jointly perform fine-grained fusion along both spatial and temporal dimensions. This design improves cross-modal alignment, reduces ambiguity, and leverages the complementary properties of each modality. Extensive experiments on the DDD17 and DSEC datasets demonstrate that MambaSeg achieves state-of-the-art segmentation performance while significantly reducing computational cost, showcasing its promise for efficient, scalable, and robust multimodal perception.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Accurate and Efficient Event-Based Semantic Segmentation Using Adaptive Spiking Encoder–Decoder NetworkIEEE Transactions on Neural Networks and Learning Systems, 2024
- Low-latency automotive vision with event camerasNature, 2024
- EISNet: A Multi-Modal Fusion Network for Semantic Segmentation With Events and ImagesIEEE Transactions on Multimedia, 2024
- Cross-Modal Learning for Event-Based Semantic Segmentation via Attention Soft AlignmentIEEE Robotics and Automation Letters, 2024
- CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With TransformersIEEE Transactions on Intelligent Transportation Systems, 2023
- Event-Based Semantic Segmentation With Posterior AttentionIEEE Transactions on Image Processing, 2023
- Exploring Event-Driven Dynamic Context for Accident Scene SegmentationIEEE Transactions on Intelligent Transportation Systems, 2021
- High Speed and High Dynamic Range Video with an Event CameraIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Survey on semantic segmentation using deep learning techniquesNeurocomputing, 2019
- SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017