MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers
Authors:
- Ruoxi ZhuRZRuoxi Zhu
- Zhengzhong TuZTZhengzhong Tu
- Jiaming LiuJLJiaming Liu
- Alan C. BovikABAlan C. Bovik
- Yibo FanYFYibo Fan
Preprint
Abstract
Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: https://github.com/taco-group/MWFormerKeywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Deep Image Deblurring: A SurveyInternational Journal of Computer Vision, 2022
- Benchmarking Single-Image Dehazing and BeyondIEEE Transactions on Image Processing, 2018
- FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image DenoisingIEEE Transactions on Image Processing, 2018
- DesnowNet: Context-Aware Deep Network for Snow RemovalIEEE Transactions on Image Processing, 2018
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image DenoisingIEEE Transactions on Image Processing, 2017
- LIME: Low-Light Image Enhancement via Illumination Map EstimationIEEE Transactions on Image Processing, 2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Making a “Completely Blind” Image Quality AnalyzerIEEE Signal Processing Letters, 2012
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004