Enhanced RGBT Tracking Network With Semantic Generation and Historical Context
- 13 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-17
Abstract
Multimodal tracking is a crucial visual task that focuses on accurately locating specific targets in video frames. The primary challenge lies in effectively utilizing visual features to identify relevant positions. Existing methods often rely on advanced visual encoders and decoders to extract features from both visible and other modalities. However, due to the limited availability of multimodal data, relying solely on visual information is insufficient. Inspired by Vision-Language models, we propose the RGBT Tracking Network with Semantic Generation and Historical Context (SHT). This approach addresses the lack of linguistic information in visual tracking and explores the semantic relationships between the target and its search area. Our approach utilizes large models to generate image descriptions, enhancing the target’s appearance information. Furthermore, it introduces the Detail Text Visual Focus (DTVF) module to improve the consistency between visual and textual data. Additionally, we present a historical prompt generation method that combines historical foreground masks with visual features to provide precise cues for tracking purposes. Experimental results show that incorporating image descriptions and historical information significantly enhances multimodal tracking performance.Keywords
Funding Information
- National Natural Science Foundation of China (62066047, 61966037)
- Practice Innovation Fund of Yunnan University (ZC-23234092)
This publication has 62 references indexed in Scilit:
- Contextual Object Detection with Multimodal Large Language ModelsInternational Journal of Computer Vision, 2024
- TATrack: Target-aware transformer for object trackingEngineering Applications of Artificial Intelligence, 2023
- RGBT tracking based on prior least absolute shrinkage and selection operator and quality aware fusion of deep and handcrafted featuresKnowledge-Based Systems, 2023
- Siamese infrared and visible light fusion network for RGB-T trackingInternational Journal of Machine Learning and Cybernetics, 2023
- Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework2022
- Learning reliable modal weight with transformer for robust RGBT trackingKnowledge-Based Systems, 2022
- Learning Adaptive Attribute-Driven Representation for Real-Time RGB-T TrackingInternational Journal of Computer Vision, 2021
- Challenge-Aware RGBT Tracking2020
- Linguistic Structure Guided Context Modeling for Referring Image Segmentation2020
- Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries2018