Machine learning in psychiatric health records: A gold standard approach to trauma annotation
Authors:
Open Access
- 1 August 2025
- journal article
- research article
- Published by Springer Nature in Translational Psychiatry
- Vol. 15 (1) , 1-8
Abstract
Psychiatric electronic health records present unique challenges for machine learning due to their unstructured, complex, and variable nature. This study aimed to create a gold standard dataset by identifying a cohort of patients with psychotic disorders and posttraumatic stress disorder, (PTSD), developing clinically-informed guidelines for annotating traumatic events in their health records and to create a gold standard publicly available dataset, and demonstrating the dataset’s suitability for training machine learning models to detect indicators of symptoms, substance use, and trauma in new records. We compiled a representative corpus of 200 narrative heavy health records (470,489 tokens) from a centralized database and developed a detailed annotation scheme with a team of clinical experts and computational linguistics. Clinicians annotated the corpus for trauma-related events and relevant clinical information with high inter-annotator agreement (0.715 for entity/span tags and 0.874 for attributes). Additionally, machine learning models were developed to demonstrate practical viability of the gold standard corpus for machine learning applications, achieving a micro F1 score of 0.76 and 0.82 for spans and attributes respectively, indicative of their predictive reliability. This study established the first gold-standard dataset for the complex task of labelling traumatic features in psychiatric health records. High inter-annotator agreement and model performance illustrate its utility in advancing the application of machine learning in psychiatric healthcare in order to better understand disease heterogeneity and treatment implications.Keywords
Funding Information
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (R21MH125076, R21MH125076, R21MH125076, R21MH125076)
This publication has 35 references indexed in Scilit:
- A Robustly Optimized BERT Pre-training Approach with Post-training2021
- Current evidence of childhood traumatic experiences in psychosis – focus on gender differencesPsychiatry Research, 2019
- Association of Trauma Type, Age of Exposure, and Frequency in Childhood and Adolescence With Psychotic Experiences in Early AdulthoodJAMA Psychiatry, 2019
- Childhood trauma and psychosis: Beyond the associationChild Abuse & Neglect, 2017
- Toward a unified theory of childhood trauma and psychosis: A comprehensive review of epidemiological, clinical, neuropsychological and biological findingsNeuroscience & Biobehavioral Reviews, 2017
- Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpusJournal of Biomedical Informatics, 2015
- The effect of lifetime adversities on resistance to antipsychotic treatment in schizophrenia patientsSchizophrenia Research, 2015
- Prevalence of self-reported childhood abuse in psychosis: A meta-analysis of retrospective studiesPsychiatry Research, 2013
- Trauma and Posttraumatic Stress Disorder (PTSD) in Patients with Schizophrenia or Schizoaffective DisorderCommunity Mental Health Journal, 2009
- Size of burden of schizophrenia and psychotic disordersEuropean Neuropsychopharmacology, 2005