Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation
Preprint
Abstract
Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called \textit{catastrophic forgetting}, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples attract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model update, so we name this problem \textit{imbalanced training}. To alleviate this problem, we propose Complementary Online Knowledge Distillation (COKD), which uses dynamically updated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experimental results on multiple machine translation tasks show that our method successfully alleviates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Vocabulary Learning via Optimal Transport for Neural Machine Translation2021
- Importance-based Neuron Allocation for Multilingual Neural Machine Translation2021
- Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation2021
- Online Knowledge Distillation via Collaborative Learning2020
- Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation2020
- Online Distilling from Checkpoints for Neural Machine Translation2019
- Deep Mutual Learning2018
- Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination2018
- Deep Residual Learning for Image Recognition2016
- Sequence-Level Knowledge Distillation2016