HyPSAM: Hybrid Prompt-Driven Segment Anything Model for RGB-Thermal Salient Object Detection
- 24 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (3) , 2697-2712
Abstract
RGB-thermal salient object detection (RGB-T SOD) aims to identify prominent objects by integrating complementary information from RGB and thermal modalities. However, learning the precise boundaries and complete objects remains challenging due to the intrinsic insufficient feature fusion and the extrinsic limitations of data scarcity. In this paper, we propose a novel hybrid prompt-driven segment anything model (HyPSAM), which leverages the zero-shot generalization capabilities of the segment anything model (SAM) for RGB-T SOD. Specifically, we first propose a dynamic fusion network (DFNet) that generates high-quality initial saliency maps as visual prompts. DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction, overcoming the limitations of fixed-parameter kernels and enhancing multi-modal feature representation. Moreover, we propose a plug-and-play refinement network (P2RNet) which serves as a general optimization strategy to guide SAM in refining saliency maps by using hybrid prompts. The text prompt ensures reliable modality input, while the mask and box prompts enable precise salient object localization. Extensive experiments on three public datasets demonstrate that our method achieves state-of-the-art performance. Notably, HyPSAM has remarkable versatility, seamlessly integrating with different RGB-T SOD methods to achieve significant performance gains, thereby highlighting the potential of prompt engineering in this field. The code and results of our method are available at: https://github.com/milotic233/HyPSAM.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62072232)
- National Natural Science Foundation of China (62576098)
- Key Research and Development Project of Jiangsu Province (BE2022138)
- Fundamental Research Funds for the Central Universities (021714380026)
- program B for Outstanding Ph.D. Candidate of Nanjing University (202201B055)
- Collaborative Innovation Center of Novel Software Technology and Industrialization
This publication has 80 references indexed in Scilit:
- Cross-Modal Fusion and Progressive Decoding Network for RGB-D Salient Object DetectionInternational Journal of Computer Vision, 2024
- UMINet: a unified multi-modality interaction network for RGB-D and RGB-T salient object detectionThe Visual Computer, 2023
- Thermal images-aware guided early fusion network for cross-illumination RGB-T salient object detectionEngineering Applications of Artificial Intelligence, 2022
- Modal complementary fusion network for RGB-T salient object detectionApplied Intelligence, 2022
- Unidirectional RGB-T salient object detection with intertwined driving of encoding and fusionEngineering Applications of Artificial Intelligence, 2022
- Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection2020
- Dynamic Filtering with Large Sampling Field for ConvNets2018
- RGB-T Saliency Detection Benchmark: Dataset, Baselines, Analysis and a Novel Approach2018
- RGBD Salient Object Detection: A Benchmark and Algorithms2014
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005