Shunyu Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Shunyu Liu has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Jie Zhou
East China Normal University, Shanghai, China
1 shared publication - Liang He
East China Normal University, Shanghai, China
1 shared publication - Qin Chen
East China Normal University, Shanghai, China
1 shared publication - Tianyu Huai
School of Computer Science and Technology, East China Normal University, Shanghai, China
1 shared publication - Xuanwen Ding
School of Computer Science and Technology, East China Normal University
1 shared publication - Yuan Xie
The Hong Kong University of Science and Technology, Hong Kong, Hong Kong
1 shared publication
Latest Publications
- Top Cited PapersOpen Access
- 9 January 2025
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 57 (5) , 1-38
Recent Advances of Foundation Language Models-based Continual Learning: A Survey
Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing and computer vision. Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich common sense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. Despite these capabilities, LMs still struggle with catastrophic forgetting, hindering their ability to learn continuously like humans. To address this, continual learning (CL) methodologies have been introduced, allowing LMs to adapt to new tasks while retaining learned knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking. In this article, we delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models, large language models, and vision-language models. We divide these studies into offline and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.
- 9 January 2025
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 57 (5) , 1-38
Top Cited PapersOpen AccessRecent Advances of Foundation Language Models-based Continual Learning: A Survey
Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing and computer vision. Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich common sense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. Despite these capabilities, LMs still struggle with catastrophic forgetting, hindering their ability to learn continuously like humans. To address this, continual learning (CL) methodologies have been introduced, allowing LMs to adapt to new tasks while retaining learned knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking. In this article, we delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models, large language models, and vision-language models. We divide these studies into offline and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.