Teaching Segment-Anything-Model Domain-Specific Knowledge for Road Crack Segmentation From On-Board Cameras
- 16 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (12) , 20588-20601
Abstract
Road crack segmentation from on-board cameras is a highly desirable yet challenging task for road condition inspection and maintenance. However, existing methods trained on small-scale datasets present limited performance from such challenging perspectives due to the lack of sufficient prior knowledge and effective generalizability. To address this limitation, this paper incorporates a vision foundation model named Segment-Anything-Model (SAM) and fully leverages its rich prior knowledge and strong generalizability to achieve crack segmentation. Also, we construct a customized crack segmentation dataset shot from on-board cameras. Considering the direct use of SAM might not correctly segment cracks, some lightweight and learnable crack adaptation layers are developed and integrated into SAM’s image encoder, which take the patch embeddings of the input image and its high-frequency components within road regions as joint inputs. During training, the parameters of the crack adaptation layers are fine-tuned to acquire domain-specific knowledge, while the parameters of the image encoder remain frozen, preserving SAM’s rich prior knowledge. Additionally, a sparse prompt generation method is proposed based on the high-frequency components within road regions, which guides the SAM model to better focus on high-frequency regions that may contain cracks. Experimental results demonstrate that the proposed framework achieves state-of-the-art performance, with an improved average precision of 8.14% compared to Mask2Former. Furthermore, the parameter-efficient transfer learning framework significantly reduces the number of parameters requiring fine-tuning, thereby improving efficiency and reducing training costs. The dataset proposed in this paper is available at https://github.com/TRMetaGroup/CrackSeg.Keywords
Funding Information
- National Key Research and Development Program of China (2023YFE0106800)
- Science Fund for Distinguished Young Scholars of Jiangsu Province (BK20231531)
This publication has 47 references indexed in Scilit:
- Asymmetric dual-decoder-U-Net for pavement crack semantic segmentationAutomation in Construction, 2023
- Road defect detection from on-board cameras with scarce and cross-domain dataAutomation in Construction, 2022
- Image-processing-based automatic crack detection and classification for refractory evaluationCeramics International, 2022
- Automatic concrete crack segmentation model based on transformerAutomation in Construction, 2022
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Densely connected deep neural network considering connectivity of pixels for automatic crack detectionAutomation in Construction, 2020
- Computer vision-based concrete crack detection using U-net fully convolutional networksAutomation in Construction, 2019
- Unified Perceptual Parsing for Scene Understanding2018
- Innovative method for recognizing subgrade defects based on a convolutional neural networkConstruction and Building Materials, 2018
- Pothole detection in asphalt pavement imagesAdvanced Engineering Informatics, 2011