Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks
Open Access
Abstract
Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.Keywords
This publication has 45 references indexed in Scilit:
- Accelerating the Super-Resolution Convolutional Neural Network2016
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Impact of detector-element active-area shape and fill factor on super-resolutionFrontiers in Physics, 2015
- Robust super-resolution by fusion of interpolated frames for color and grayscale imagesFrontiers in Physics, 2015
- Fast super-resolution using an adaptive Wiener filter with robustness to local motionOptics Express, 2012
- Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imagingOptics Express, 2011
- Contour Detection and Hierarchical Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Super-resolution image reconstruction: a technical overviewIEEE Signal Processing Magazine, 2003