Controlling Vision-Language Models for Multi-Task Image Restoration
Preprint
Abstract
Vision-language models such as CLIP have shown great impact on diverse downstream tasks for zero-shot or label-free predictions. However, when it comes to low-level vision such as image restoration their performance deteriorates dramatically due to corrupted inputs. In this paper, we present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself will also output a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we construct a mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances state-of-the-art performance on both \emph{degradation-specific} and \emph{unified} image restoration tasks, showing a promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available at https://github.com/Algolzw/daclip-uir.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior2024
- Exploiting Diffusion Prior for Real-World Image Super-ResolutionInternational Journal of Computer Vision, 2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Vision Transformers for Single Image DehazingIEEE Transactions on Image Processing, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- EnlightenGAN: Deep Light Enhancement Without Paired SupervisionIEEE Transactions on Image Processing, 2021
- Joint Rain Detection and Removal from a Single Image with Contextualized Deep NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- DesnowNet: Context-Aware Deep Network for Snow RemovalIEEE Transactions on Image Processing, 2018
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image DenoisingIEEE Transactions on Image Processing, 2017
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004