Learning Without Forgetting for Vision-Language Models
Top Cited Papers
- 11 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (6) , 4489-4504
Abstract
Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world, which requires a learning system to adapt to new tasks without forgetting former ones. While traditional CIL methods focus on visual information to grasp core features, recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations with the aid of textual information. However, when continually trained with new classes, VLMs often suffer from catastrophic forgetting of former knowledge. Applying VLMs to CIL poses two major challenges: 1) how to adapt the model without forgetting and 2) how to make full use of the multi-modal information. To this end, we propose PROjectiOn Fusion (Proof) that enables VLMs to learn without forgetting. To handle the first challenge, we propose training task-specific projections based on the frozen image/text encoders. When facing new tasks, new projections are expanded, and former projections are fixed, alleviating the forgetting of old concepts. For the second challenge, we propose the fusion module to better utilize the cross-modality information. By jointly adjusting visual and textual features, the model can capture better task-specific semantic information that facilitates recognition. Extensive experiments on nine benchmark datasets with various continual learning scenarios and various VLMs validate that Proof achieves state-of-the-art performance.Keywords
All Related Versions
Funding Information
- National Science and Technology Major Project (2022ZD0114805)
- National Natural Science Foundation of China (62476123, 62376118, 62006112, 62250069)
- Fundamental Research Funds for the Central Universities (2024300373, 14380021)
- CCF-Tencent Rhino-Bird Open Research Fund (RAGR20240101)
- Collaborative Innovation Center of Novel Software Technology and Industrialization
- China Scholarship Council
- Science Project of Nanjing University
- Ministry of Education
- Singapore
- MOE AcRF Tier 2 (MOET2EP20221-0012)
- NTU NAP
- Industry Alignment Fund
- Collaboration Projects (IAF-ICP) Funding Initiative (RIE2020)
This publication has 76 references indexed in Scilit:
- Visual Prompt Tuning2022
- DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning2022
- SLIP: Self-supervision Meets Language-Image Pre-training2022
- FOSTER: Feature Boosting and Compression for Class-Incremental Learning2022
- Topology-Preserving Class-Incremental Learning2020
- PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning2020
- Memory-Efficient Incremental Learning Through Feature Adaptation2020
- Memory Aware Synapses: Learning What (not) to Forget2018
- Lifelong Learning via Progressive Distillation and Retrospection2018
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence2018