Abstract
Multimodal tracking is a crucial visual task that focuses on accurately locating specific targets in video frames. The primary challenge lies in effectively utilizing visual features to identify relevant positions. Existing methods often rely on advanced visual encoders and decoders to extract features from both visible and other modalities. However, due to the limited availability of multimodal data, relying solely on visual information is insufficient. Inspired by Vision-Language models, we propose the RGBT Tracking Network with Semantic Generation and Historical Context (SHT). This approach addresses the lack of linguistic information in visual tracking and explores the semantic relationships between the target and its search area. Our approach utilizes large models to generate image descriptions, enhancing the target’s appearance information. Furthermore, it introduces the Detail Text Visual Focus (DTVF) module to improve the consistency between visual and textual data. Additionally, we present a historical prompt generation method that combines historical foreground masks with visual features to provide precise cues for tracking purposes. Experimental results show that incorporating image descriptions and historical information significantly enhances multimodal tracking performance.
Funding Information
  • National Natural Science Foundation of China (62066047, 61966037)
  • Practice Innovation Fund of Yunnan University (ZC-23234092)

This publication has 62 references indexed in Scilit: