PIVOT: Prompting for Video Continual Learning
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 24214-24223
Abstract
Modern machine learning pipelines are limited due to data availability, storage quotas, privacy regulations, and expensive annotation processes. These constraints make it difficult or impossible to train and update large-scale models on such dynamic annotated sets. Continual learning directly approaches this problem, with the ultimate goal of devising methods where a deep neural network effectively learns relevant patterns for new (unseen) classes, without significantly altering its performance on previously learned ones. In this paper, we address the problem of continual learning for video data. We introduce PIVOT, a novel method that leverages extensive knowledge in pre-trained models from the image domain, thereby reducing the number of trainable parameters and the associated forgetting. Unlike previous methods, ours is the first approach that effectively uses prompting mechanisms for continual learning without any in-domain pre-training. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.Keywords
All Related Versions
Funding Information
- King Abdullah University of Science and Technology (KAUST)
- FONDECYT (1221425)
This publication has 22 references indexed in Scilit:
- Temporal Segment Networks for Action Recognition in VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Learning without ForgettingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Mask R-CNN2017
- iCaRL: Incremental Classifier and Representation Learning2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Catastrophic forgetting in connectionist networksTrends in Cognitive Sciences, 1999