TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2343-2353
Abstract
Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the all adverse weather removal problem. To this end, we propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder that can restore an image degraded by any weather condition. Specifically, we utilize a novel transformer encoder using intra-patch transformer blocks to enhance attention inside the patches to effectively remove smaller weather degradations. We also introduce a transformer decoder with learnable weather type embeddings to adjust to the weather degradation at hand. Trans Weather achieves significant improvements across multiple test datasets over both All-in-One network as well as methods fine-tuned for specific tasks. TransWeather is also validated on real world test images and found to be more effective than previous methods. Implementation code can be found in the supplementary document. Code is available at https//github.com/jeya-maria-jose/TransWeather.Keywords
All Related Versions
Funding Information
- ARO (W911NF-21-1-0135)
This publication has 53 references indexed in Scilit:
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Single Image Dehazing via Multi-scale Convolutional Neural Networks2016
- DehazeNet: An End-to-End System for Single Image Haze RemovalIEEE Transactions on Image Processing, 2016
- Non-local Image Dehazing2016
- Dehazing Using Color-LinesACM Transactions on Graphics, 2014
- Automatic Single-Image-Based Rain Streaks Removal via Image DecompositionIEEE Transactions on Image Processing, 2011
- Adaptive shape processing in primary visual cortexProceedings of the National Academy of Sciences, 2011
- Single Image Haze Removal Using Dark Channel PriorIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- Fields of Experts: A Framework for Learning Image Priors2005
- Perceptual learning and top-down influences in primary visual cortexNature Neuroscience, 2004