CrossZoom: Simultaneous Motion Deblurring and Event Super-Resolving
- 20 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 8209-8227
Abstract
Even though the collaboration between traditional and neuromorphic event cameras brings prosperity to frame-event based vision applications, the performance is still confined by the resolution gap crossing two modalities in both spatial and temporal domains. This paper is devoted to bridging the gap by increasing the temporal resolution for images, i.e., motion deblurring, and the spatial resolution for events, i.e., event super-resolving, respectively. To this end, we introduce C ross Z oom, a novel unified neural Net work (CZ-Net) to jointly recover sharp latent sequences within the exposure period of a blurry input and the corresponding High-Resolution (HR) events. Specifically, we present a multi-scale blur-event fusion architecture that leverages the scale-variant properties and effectively fuses cross-modal information to achieve cross-enhancement. Attention-based adaptive enhancement and cross-interaction prediction modules are devised to alleviate the distortions inherent in Low-Resolution (LR) events and enhance the final results through the prior blur-event complementary information. Furthermore, we propose a new dataset containing HR sharp-blurry images and the corresponding HR-LR event streams to facilitate future research. Extensive qualitative and quantitative experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of the proposed method.Keywords
Funding Information
- National Natural Science Foundation of China (62271354, 62523111, 61871297)
This publication has 55 references indexed in Scilit:
- Boosting Event Stream Super-Resolution with a Recurrent Neural Network2022
- Event-Based Fusion for Motion Deblurring with Cross-modal Attention2022
- Real-Time Intermediate Flow Estimation for Video Frame Interpolation2022
- Computer Vision2022
- Event Enhanced High-Quality Image Recovery2020
- Super-resolution of spatiotemporal event-stream imageNeurocomputing, 2019
- Unsupervised Event-Based Optical Flow Using Motion Compensation2019
- Inverse Kernels for Fast Spatial Deconvolution2014
- Blind Deblurring Using Internal Patch Recurrence2014
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004