HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning
- 18 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (8) , 6687-6702
Abstract
The deployment of pre-trained models (PTMs) has greatly advanced the field of continual learning (CL), enabling positive knowledge transfer and resilience to catastrophic forgetting. To sustain these advantages for sequentially arriving tasks, a promising direction involves keeping the pre-trained backbone frozen while employing parameter-efficient tuning (PET) techniques to instruct representation learning. Despite the popularity of Prompt-based PET for CL, its empirical design often leads to sub-optimal performance in our evaluation of different PTMs and target tasks. To this end, we propose a unified framework for CL with PTMs and PET that provides both theoretical and empirical advancements. We first perform an in-depth theoretical analysis of the CL objective in a pre-training context, decomposing it into hierarchical components namely within-task prediction, task-identity inference and task-adaptive prediction. We then present Hierarchical Decomposition PET (HiDe-PET), an innovative approach that explicitly optimizes the decomposed objective through incorporating task-specific and task-shared knowledge via mainstream PET techniques along with efficient recovery of pre-trained representations. Leveraging this framework, we delve into the distinct impacts of implementation strategy, PET technique and PET architecture, as well as adaptive knowledge accumulation amidst pronounced distribution changes. Finally, across various CL scenarios, our approach demonstrates remarkably superior performance over a broad spectrum of recent strong baselines.Keywords
All Related Versions
Funding Information
- NSFC Projects (62406160, 62350080, 62106123, 62106120, 92370124, 92248303)
- Beijing Natural Science Foundation (L247011)
- Tsinghua Institute for Guo Qiang
- High Performance Computing Center, Tsinghua University
- Postdoctoral Fellowship Program of CPSF (GZB20230350)
- Shuimu Tsinghua Scholar
- XPLORER PRIZE
This publication has 31 references indexed in Scilit:
- Generalized Out-of-Distribution Detection: A SurveyInternational Journal of Computer Vision, 2024
- Visual Prompt Tuning2022
- DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning2022
- CoSCL: Cooperation of Small Continual Learners is Stronger Than a Big One2022
- Biological underpinnings for lifelong learning machinesNature Machine Intelligence, 2022
- Continual lifelong learning with neural networks: A reviewNeural Networks, 2019
- Memory Aware Synapses: Learning What (not) to Forget2018
- What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory UpdatedTrends in Cognitive Sciences, 2016
- Dynamics of Retrieval Strategies for Remote MemoriesCell, 2011
- The organization of recent and remote memoriesNature Reviews Neuroscience, 2005