Rethinking Image Deblurring via CNN-Transformer Multiscale Hybrid Architecture
- 19 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 72 (00189456) , 1-15
Abstract
Image deblurring is a representative low-level vision task that aims to estimate latent sharp images from blurred images. Recently, convolutional neural network (CNN)-based methods have dominated image deblurring. However, traditional CNN-based deblurring methods suffer from two essential issues: first, existing multiscale deblurring methods process blurred images at different scales through sub-networks with the same composition, which limits the model performance. Second, the convolutional layers fail to adapt to the input content and cannot effectively capture long-range dependencies. To alleviate the above issues, we rethink the multiscale architecture that follows a coarse-to-fine strategy and propose a novel hybrid architecture that combines CNN and transformer (CTMS). CTMS has three distinct features. First, the finer-scale sub-networks in CTMS are designed as architectures with larger receptive fields to obtain the pixel values around the blur, which can be used to efficiently handle large-area blur. Then, we propose a feature modulation network to alleviate the disadvantages of CNN sub-networks that lack input content adaptation. Finally, we design an efficient transformer block, which significantly reduces the computational burden and requires no pre-training. Our proposed deblurring model is extensively evaluated on several benchmark datasets, and achieves superior performance compared to state-of-the-art deblurring methods. Especially, the peak signal to noise ratio (PSNR) and structural similarity (SSIM) values are 32.73 dB and 0.959, respectively, on the popular dataset GoPro. In addition, we conduct joint evaluation experiments on the proposed method deblurring performance, object detection, and image segmentation to demonstrate the effectiveness of CTMS for subsequent high-level computer vision tasks.Keywords
Funding Information
- National Natural Science Foundation of China (61966037, 62066047)
This publication has 63 references indexed in Scilit:
- Deblurring Low-Light Images with Light StreaksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deblurring Images via Dark Channel PriorIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Pyramid Scene Parsing Network2017
- Deep Residual Learning for Image Recognition2016
- Learning a convolutional neural network for non-uniform motion blur removal2015
- Blind Deblurring Using Internal Patch Recurrence2014
- Unnatural L0 Sparse Representation for Natural Image Deblurring2013
- Image information and visual qualityIEEE Transactions on Image Processing, 2006
- Multiscale structural similarity for image quality assessment2004
- An image enhancement technique combining sharpening and noise reductionIEEE Transactions on Instrumentation and Measurement, 2002