Auditing the Shadows: A Review of Methods to Detect Shared Training Data in Large Language Models
Author:
Open Access
- 24 December 2025
- journal article
- review article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 58 (7) , 1-34
Abstract
Large language models (LLMs) are often trained on undisclosed data. This practice has intensified debates about transparency, copyright compliance, and reproducibility. From this lens, this article systematically reviews methodologies to detect shared training data across LLMs. More specifically, our review spans five methodological families: (1) lexical/semantic overlap metrics, which compare output similarity but struggle with knowledge convergence; (2) memorization analysis, which identifies verbatim regurgitation of rare training examples yet risks extracting copyrighted content; (3) temporal alignment by leveraging models’ knowledge cutoffs to infer shared data timelines; (4) adversarial susceptibility correlation to measure shared failure modes under attack; and (5) synthetic fingerprinting by embedding detectable artifacts in training data. Our review reveals several critical gaps: existing methods are highly siloed, with little cross-community dialogue, evaluation inconsistency epidemics, and ethical risks are understudied and potentially violate data privacy laws. We propose the taxonomy of auditing techniques, an approach to conduct the same, and conduct a case study to showcase its merit. Finally, this review highlights an under-explored knowledge gap and sets a roadmap for future research directions.Keywords
This publication has 105 references indexed in Scilit:
- ChatGPT in Academia: University Students’ Attitudes Towards the use of ChatGPT and PlagiarismJournal of Academic Ethics, 2025
- Enhancing vulnerability detection efficiency: An exploration of light-weight LLMs with hybrid code featuresJournal of Information Security and Applications, 2024
- Survey on AI-Generated Plagiarism Detection: The Impact of Large Language Models on Academic IntegrityJournal of Academic Ethics, 2024
- ST-LLM: Large Language Models Are Effective Temporal Learners2024
- Privacy issues in Large Language Models: A surveyComputers and Electrical Engineering, 2024
- Easy-read and large language models: on the ethical dimensions of LLM-based text simplificationEthics and Information Technology, 2024
- Adversarial Attacks on Large Language Models2024
- A survey on large language model (LLM) security and privacy: The Good, The Bad, and The UglyHigh-Confidence Computing, 2024
- Pre‐trained language models: What do they know?WIREs Data Mining and Knowledge Discovery, 2023
- Learning similarity with cosine similarity ensembleInformation Sciences, 2015