Visual Prompt Multi-Modal Tracking
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9516-9526
Abstract
Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multi-modal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2018AAA0102001)
- National Natural Science Foundation of China (62293542,U1903215)
- Fundamental Research Funds for the Central Universities (DUT22ZD210)
This publication has 31 references indexed in Scilit:
- Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive LearningSensors, 2020
- RGB-T object tracking: Benchmark and baselinePattern Recognition, 2019
- Dense Feature Aggregation and Pruning for RGBT Tracking2019
- Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting schemeInfrared Physics & Technology, 2019
- Context-Aware Three-Dimensional Mean-Shift With Occlusion Handling for Robust Object Tracking in RGB-D VideosIEEE Transactions on Multimedia, 2018
- Weighted Sparse Representation Regularized Graph Learning for RGB-T Object Tracking2017
- Combined frame- and event-based detection and tracking2016
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Thermo-visual feature fusion for object tracking using multiple spatiogram trackersMachine Vision and Applications, 2007