Mamba Adapter: Efficient Multi-Modal Fusion for Vision-Language Tracking
- 3 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (9) , 9300-9311
Abstract
Utilizing the high-level semantic information of language to compensate for the limitations of vision information is a highly regarded approach in single-object tracking. However, most existing vision-language (VL) trackers employ full-parameter fine-tuning, which can easily lead to catastrophic forgetting. Therefore, they fail to fully exploit the prior knowledge of pre-trained models from upstream tasks, resulting in unsatisfactory tracking performance. To alleviate the above problem, we propose a simple yet effective Vision-Language Tracking pipeline based on Mamba Adapter, named MAVLT, which adopts the idea of parameter-efficient fine-tuning (PEFT) to realize the interaction between vision-language modalities. This novel approach offers the following advantages: (1)The knowledge of the upstream pre-trained model is efficiently inherited by freezing its parameters. This ensures that the VL tracking framework only learns the modules for vision and language interaction, with a focus on the fusion between modalities. (2)The modal interaction between language and vision encoders is flexibly bridged in each encoder layer via proposed mamba adapter, enabling efficient interaction of visual and language information at multiple levels. Extensive experiments on five popular vision-language tracking benchmarks validate the effectiveness of the proposed MAVLT. Particularly, the MAVLT achieves 73.4% AUC score on the LaSOT benchmarks with only 0.18%(0.32M) of the total parameters updates. Code and models are available at https://github.com/GXNU-ZhongLab/MAVLT.Keywords
Funding Information
- National Natural Science Foundation of China (U23A20383, 62472109, 62466051)
- Project of Guangxi Science and Technology (2024GXNSFGA010001, 2025GXNSFAA069676, 2025GXNSFAA069417)
- Research Project of Guangxi Normal University (2024DF001)
- Innovation Project of Guangxi Graduate Education (YCBZ2024083)
- Guangxi Colleges and Universities Key Laboratory of Intelligent Software (2024B01)
This publication has 58 references indexed in Scilit:
- Visual Prompt Tuning2022
- Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework2022
- Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking2022
- LaSOT: A High-quality Large-scale Single Object Tracking BenchmarkInternational Journal of Computer Vision, 2020
- Real-time visual detection and tracking system for traffic monitoringEngineering Applications of Artificial Intelligence, 2019
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017
- Fully-Convolutional Siamese Networks for Object Tracking2016
- Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- Visual object tracking using adaptive correlation filters2010