Learning Frame-Event Fusion for Motion Deblurring
- 11 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6836-6849
Abstract
Motion deblurring is a highly ill-posed problem due to the significant loss of motion information in the blurring process. Complementary informative features from auxiliary sensors such as event cameras can be explored for guiding motion deblurring. The event camera can capture rich motion information asynchronously with microsecond accuracy. In this paper, a novel frame-event fusion framework is proposed for event-driven motion deblurring (FEF-Deblur), which can sufficiently explore long-range cross-modal information interactions. Firstly, different modalities are usually complementary and also redundant. Cross-modal fusion is modeled as complementary-unique features separation-and-aggregation, avoiding the modality redundancy. Unique features and complementary features are first inferred with parallel intra-modal self-attention and inter-modal cross-attention respectively. After that, a correlation-based constraint is designed to act between unique and complementary features to facilitate their differentiation, which assists in cross-modal redundancy suppression. Additionally, spatio-temporal dependencies among neighboring inputs are crucial for motion deblurring. A recurrent cross attention is introduced to preserve inter-input attention information, in which the current spatial features and aggregated temporal features are attending to each other by establishing the long-range interaction between them. Extensive experiments on both synthetic and real-world motion deblurring datasets demonstrate our method outperforms state-of-the-art event-based and image/video-based methods. The code will be made publicly available.Keywords
Funding Information
- National Natural Science Foundation of China (62402366)
- Fundamental Research Funds for the Central Universities (QTZX23038)
- China Postdoctoral Science Foundation (2024M752535)
- Natural Science Basic Research Program of Shaanxi (2024JC-YBQN-0627)
This publication has 82 references indexed in Scilit:
- Simple Baselines for Image Restoration2022
- Stripformer: Strip Transformer for Fast Image Deblurring2022
- Efficient Video Deblurring Guided by Motion Magnitude2022
- Event-Based Fusion for Motion Deblurring with Cross-modal Attention2022
- Event Enhanced High-Quality Image Recovery2020
- Multi-Temporal Recurrent Neural Networks for Progressive Non-uniform Single Image Deblurring with Incremental Temporal Training2020
- Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring2020
- Learning Event-Driven Video Deblurring and Interpolation2020
- Continuous-Time Intensity Estimation Using Event Cameras2019
- Image Super-Resolution Using Very Deep Residual Channel Attention Networks2018