Learning to Super-Resolve Blurry Images With Events
- 30 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 45 (8) , 10027-10043
Abstract
S uper- R esolution from a single motion B lurred image (SRB) is a severely ill-posed problem due to the joint degradation of motion blurs and low spatial resolution. In this article, we employ events to alleviate the burden of SRB and propose an E vent-enhanced SRB (E-SRB) algorithm, which can generate a sequence of sharp and clear images with H igh R esolution (HR) from a single blurry image with L ow R esolution (LR). To achieve this end, we formulate an event-enhanced degeneration model to consider the low spatial resolution, motion blurs, and event noises simultaneously. We then build an e vent-enhanced S parse L earning Net work ( eSL-Net++ ) upon a dual sparse learning scheme where both events and intensity frames are modeled with sparse representations. Furthermore, we propose an event shuffle-and-merge scheme to extend the single-frame SRB to the sequence-frame SRB without any additional training process. Experimental results on synthetic and real-world datasets show that the proposed eSL-Net++ outperforms state-of-the-art methods by a large margin. Datasets, codes, and more results are available at https://github.com/ShinyWang33/eSL-Net-Plusplus .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62271354, 61871297, 61922065, U22B201, 41820104006, 61871299)
- Natural Science Foundation of Hubei province, China (2021CFB467)
This publication has 69 references indexed in Scilit:
- LRTV: MR Image Super-Resolution With Low-Rank and Total Variation RegularizationsIEEE Transactions on Medical Imaging, 2015
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor2014
- Simultaneous Mosaicing and Tracking with an Event Camera2014
- Event-Based Visual FlowIEEE Transactions on Neural Networks and Learning Systems, 2013
- A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision SensorIEEE Journal of Solid-State Circuits, 2008
- $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse RepresentationIEEE Transactions on Signal Processing, 2006
- An iterative thresholding algorithm for linear inverse problems with a sparsity constraintCommunications on Pure and Applied Mathematics, 2004
- Multiscale structural similarity for image quality assessment2004
- Motion deblurring and super-resolution from an image sequence1996