Large Language Model–Generated Synthetic Data for Parole and Probation Detection in Clinical Notes: Development and Evaluation Study
Open Access
- 19 October 2026
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e104586
Abstract
Background: Identifying individuals on parole or probation can help inform care pathways and ensure continuity of care in health care systems. However, this sensitive justice-related information is often absent from structured electronic health record fields and is instead documented in narrative clinical notes. Developing automated parole and probation detection (PPD) systems typically requires expert-annotated clinical data, but manual annotation is costly, time-consuming, and constrained by data sparsity and privacy concerns. Objective: This study aimed to develop and evaluate a synthetic-only training paradigm for PPD from clinical text that eliminates the need for expert-annotated training data while maintaining strong performance and cross-domain transfer to real-world data. Methods: We developed Synth-PPD, a synthetic dataset comprising 7579 labeled clinical notes generated using large language models (LLMs), redundancy filtering, and automated quality evaluation. Dataset quality was assessed through human review of 1517 synthetic notes by 2 annotators. The primary outcome was the macro–F1-score for PPD. We evaluated Synth-PPD across unsupervised and supervised learning settings using LLMs, traditional machine learning models, and transformer encoders. Performance was assessed on 293 expert-annotated real-world clinical notes from the US Department of Veterans Affairs. Results: Synth-PPD achieved 90.05% human-LLM alignment. In the unsupervised setting, Llama 3.3-70B-Instruct with chain-of-thought prompting and few-shot examples from Synth-PPD achieved a macro–F1-score of 87.22%, outperforming both the corresponding zero-shot model (80.10%) and the best rule-based baseline (77.55%; P<.001 for all comparisons). In the supervised setting, Llama 3.1-8B fine-tuned on Synth-PPD achieved a macro–F1-score of 81.43%, exceeding the best rule-based baseline (77.55%; P<.001) and achieving a numerically higher score than the substantially larger zero-shot Llama 3.3-70B-Instruct model (80.10%; P=.08). In contrast, traditional machine learning models and transformer encoders exhibited greater performance degradation under domain shift and derived limited benefit from increasing synthetic training data. Additional experiments showed that increasing synthetic training data improved cross-domain performance for LLM-based models but yielded comparatively smaller gains for traditional approaches. Conclusions: Synthetic data generated by LLMs can serve as a scalable, cost-effective, and privacy-preserving alternative to expert-annotated real-world clinical data for PPD from clinical text. The findings demonstrate successful cross-domain transfer to real-world US Department of Veterans Affairs clinical notes.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- A survey on LLM-as-a-judgeThe Innovation, 2026
- Association between PTSD and health-related social needs in US Veterans: an NLP analysis using Veterans Health Administration DataJournal of Affective Disorders, 2025
- An empirical study of LLMs via in-context learning for stance classificationInformation Processing & Management, 2025
- Large language models to identify social determinants of health in electronic health recordsnpj Digital Medicine, 2024
- Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation studyThe Lancet Digital Health, 2023
- Large language models propagate race-based medicinenpj Digital Medicine, 2023
- Associations Between Natural Language Processing–Enriched Social Determinants of Health and Suicide Death Among US VeteransJAMA Network Open, 2023
- Use of Veterans Health Administration Mental Health and Substance Use Disorder Treatment After Exiting Prison: The Health Care for Reentry Veterans ProgramAdministration and Policy in Mental Health and Mental Health Services Research, 2015
- Risk of incarceration and clinical characteristics of incarcerated veterans by race/ethnicitySocial Psychiatry and Psychiatric Epidemiology, 2013
- Toward a criminal justice epidemiology: Behavioral and physical health of probationers and parolees in the United StatesJournal of Criminal Justice, 2012