From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models
Preprint
Abstract
Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Can generative AI detect and fix real-world cryptographic misuses?Journal of Systems and Software, 2025
- On Inter-Dataset Code Duplication and Data Leakage in Large Language ModelsIEEE Transactions on Software Engineering, 2024
- Research on Software Defect Detection Based on Random Forest Algorithm2024
- The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluationBMC Genomics, 2020