CAE-Net: Cross-Modal Attention Enhancement Network for RGB-T Salient Object Detection
Open Access
- 14 February 2023
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 12 (4) , 953
Abstract
RGB salient object detection (SOD) performs poorly in low-contrast and complex background scenes. Fortunately, the thermal infrared image can capture the heat distribution of scenes as complementary information to the RGB image, so the RGB-T SOD has recently attracted more and more attention. Many researchers have committed to accelerating the development of RGB-T SOD, but some problems still remain to be solved. For example, the defective sample and interfering information contained in the RGB or thermal image hinder the model from learning proper saliency features, meanwhile the low-level features with noisy information result in incomplete salient objects or false positive detection. To solve these problems, we design a cross-modal attention enhancement network (CAE-Net). First, we concretely design a cross-modal fusion (CMF) module to fuse cross-modal features, where the cross-attention unit (CAU) is employed to enhance the two modal features, and channel attention is used to dynamically weigh and fuse the two modal features. Then, we design the joint-modality decoder (JMD) to fuse cross-level features, where the low-level features are purified by higher level features, and multi-scale features are sufficiently integrated. Besides, we add two single-modality decoder (SMD) branches to preserve more modality-specific information. Finally, we employ a multi-stream fusion (MSF) module to fuse three decoders’ features. Comprehensive experiments are conducted on three RGB-T datasets, and the results show that our CAE-Net is comparable to the other methods.Keywords
Funding Information
- National Key Research and Development Program of China (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- Fundamental Research Funds for the Provincial Universities of Zhejiang (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- National Natural Science Foundation of China (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- “Pioneer” and “Leading Goose” R&D Program of Zhejiang Province (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- Zhejiang Province Nature Science Foundation of China (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- Hangzhou Dianzi University (HDU) and the China Electronics Corporation DATA (CECDATA) Joint Research Center of Big Data Technologies (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- Higher Education Discipline Innovation Project (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
- Fundamental Research Funds for the Provincial Universities of Zhejiang (2020YFB1406604, GK229909299001-009, 62271180, 62171002, 61901145, U21B2024, 61931008, 62071415, 61972123, 62001146, 2022C01068, LR17F030006, LY19F030022, LZ22F020003, KYH063120009, D17019, GK219909299001-407)
This publication has 58 references indexed in Scilit:
- Deeply Supervised Salient Object Detection with Short Connections2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Fully convolutional networks for semantic segmentation2015
- Depth saliency based on anisotropic center-surround difference2014
- RGBD Salient Object Detection: A Benchmark and Algorithms2014
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998