Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 25606-25616
Abstract
Real-world vision tasks frequently suffer from the appearance of unexpected adverse weather conditions, including rain, haze, snow, and raindrops. In the last decade, convolutional neural networks and vision transformers have yielded outstanding results in single-weather video removal. However, due to the absence of appropriate adaptation, most of them fail to generalize to other weather conditions. Although ViWS-Net is proposed to remove ad-verse weather conditions in videos with a single set of pre-trained weights, it is seriously blinded by seen weather at train-time and degenerates when coming to unseen weather during test-time. In this work, we introduce test-time adaptation into adverse weather removal in videos, and propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process. Specifically, we devise a diffusion-based network with a novel temporal noise model to efficiently explore frame-correlated information in degraded video clips at training stage. During inference stage, we introduce a proxy task named Diffusion Tubelet Self-Calibration to learn the primer distribution of test video stream and optimize the model by approx-imating the temporal noise model for online adaptation. Experimental results, on benchmark datasets, demonstrate that our Test-Time Adaptation method with Diffusion-based network(Diff- TTA) outperforms state-of-the-art methods in terms of restoring videos degraded by seen weather conditions. Its generalizable capability is validated with unseen weather conditions in synthesized and real-world videos.Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Snow Mask Guided Adaptive Residual Network for Image Snow RemovalComputer Vision and Image Understanding, 2023
- Simple Baselines for Image Restoration2022
- Rethinking Video Rain Streak Removal: A New Synthesis Model and a Deraining Network with Video Rain Prior2022
- Fully Test-Time Adaptation for Image Segmentation2021
- JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal2020
- Frame-Consistent Recurrent Video Deraining With Dual-Level Flow2019
- Video Rain Streak Removal by Multiscale Convolutional Sparse Coding2018
- Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework2018
- Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos2018
- Box and Jenkins: Time Series Analysis, Forecasting and Control2013