COLA: Context-Aware Language-Driven Test-Time Adaptation
- 19 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 6002-6015
Abstract
Test-time adaptation (TTA) has gained increasing popularity due to its efficacy in addressing “distribution shift” issue while simultaneously protecting data privacy. However, most prior methods assume that a paired source domain model and target domain sharing the same label space coexist, heavily limiting their applicability. In this paper, we investigate a more general source model capable of adaptation to multiple target domains without needing shared labels. This is achieved by using a pre-trained vision-language model (VLM), e.g., CLIP, that can recognize images through matching with class descriptions. While the zero-shot performance of VLMs is impressive, they struggle to effectively capture the distinctive attributes of a target domain. To that end, we propose a novel method – Context-aware Language-driven TTA (COLA). The proposed method incorporates a lightweight context-aware module that consists of three key components: a task-aware adapter, a context-aware unit, and a residual connection unit for exploring task-specific knowledge, domain-specific knowledge from the VLM and prior knowledge of the VLM, respectively. It is worth noting that the context-aware module can be seamlessly integrated into a frozen VLM, ensuring both minimal effort and parameter efficiency. Additionally, we introduce a Class-Balanced Pseudo-labeling (CBPL) strategy to mitigate the adverse effects caused by class imbalance. We demonstrate the effectiveness of our method not only in TTA scenarios but also in class generalisation tasks. The source code is available at https://github.com/NUDT-Bai-Group/COLA-TTA.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62206305)
This publication has 39 references indexed in Scilit:
- Few-shot adaptation of multi-modal foundation models: a surveyArtificial Intelligence Review, 2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Cross-Modal Knowledge Transfer Without Task-Relevant Source Data2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Recognition in Terra Incognita2018
- Cats and dogs2012
- Undoing the Damage of Dataset Bias2012
- Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories2005