OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13283-13292
Abstract
Omnidirectional images (ODIs) have obtained lots of research interest for immersive experiences. Although ODIs require extremely high resolution to capture details of the entire scene, the resolutions of most ODIs are insufficient. Previous methods attempt to solve this issue by image super-resolution (SR) on equirectangular projection (ERP) images. However, they omit geometric properties of ERP in the degradation process, and their models can hardly generalize to real ERP images. In this paper, we propose Fisheye downsampling, which mimics the real-world imaging process and synthesizes more realistic low-resolution samples. Then we design a distortion-aware Transformer (OSRT) to modulate ERP distortions continuously and self-adaptively. Without a cumbersome process, OSRT outperforms previous methods by about 0.2dB on PSNR. Moreover, we propose a convenient data augmentation strategy, which synthesizes pseudo ERP images from plain images. This simple strategy can alleviate the over-fitting problem of large networks and significantly boost the performance of ODISR. Extensive experiments have demonstrated the state-of-the-art performance of our OSRT.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022ZD0160505)
- National Natural Science Foundation of China (62276251)
- Youth Innovation Promotion Association of Chinese Academy of Sciences (2020356)
This publication has 36 references indexed in Scilit:
- Deformable Convolutional Networks2017
- NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results2017
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Weighted-to-Spherically-Uniform Quality Evaluation for Omnidirectional VideoIEEE Signal Processing Letters, 2017
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Joint Registration and Super-Resolution With Omnidirectional ImagesIEEE Transactions on Image Processing, 2011
- Scope of validity of PSNR in image/video quality assessmentElectronics Letters, 2008
- Super-resolution omnidirectional camera images using spatio-temporal analysisElectronics and Communications in Japan (Part III: Fundamental Electronic Science), 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Super-resolution from an omnidirectional image sequence2002