MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers
- 25 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 6790-6805
Abstract
Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: https://github.com/taco-group/MWFormer.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2023YFB4502802)
- National Natural Science Foundation of China (62031009)
- Fudan-ZTE Joint Laboratory
- Alibaba Innovative Research (AIR) Program
- Alibaba Research Fellow (ARF) Program
This publication has 55 references indexed in Scilit:
- Not Just Streaks: Towards Ground Truth for Single Image Deraining2022
- Deep Image Deblurring: A SurveyInternational Journal of Computer Vision, 2022
- JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal2020
- GIA-Net: Global Information Aware Network for Low-Light Imaging2020
- Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning2019
- Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining2018
- DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks2018
- Attentive Generative Adversarial Network for Raindrop Removal from A Single Image2018
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015