Automated redaction of names in adverse event reports using transformer-based neural networks
Open Access
- 23 December 2024
- journal article
- research article
- Published by Springer Nature in BMC Medical Informatics and Decision Making
- Vol. 24 (1) , 1-11
Abstract
Background: Automated recognition and redaction of personal identifiers in free text can enable organisations to share data while protecting privacy. This is important in the context of pharmacovigilance since relevant detailed information on the clinical course of events, differential diagnosis, and patient-reported reflections may often only be conveyed in narrative form. The aim of this study is to develop and evaluate a method for automated redaction of person names in English narrative text on adverse event reports. The target domain for this study was case narratives from the United Kingdom’s Yellow Card scheme, which collects and monitors information on suspected side effects to medicines and vaccines. Methods: We finetuned BERT – a transformer-based neural network – for recognising names in case narratives. Training data consisted of newly annotated records from the Yellow Card data and of the i2b2 2014 deidentification challenge. Because the Yellow Card data contained few names, we used predictive models to select narratives for training. Performance was evaluated on a separate set of annotated narratives from the Yellow Card scheme. In-depth review determined whether (parts of) person names missed by the de-identification method could enable re-identification of the individual, and whether de-identification reduced the clinical utility of narratives by collaterally masking relevant information. Results: Recall on held-out Yellow Card data was 87% (155/179) at a precision of 55% (155/282) and a false-positive rate of 0.05% (127/ 263,451). Considering tokens longer than three characters separately, recall was 94% (102/108) and precision 58% (102/175). For 13 of the 5,042 narratives in Yellow Card test data (71 with person names), the method failed to flag at least one name token. According to in-depth review, the leaked information could enable direct identification for one narrative and indirect identification for two narratives. Clinically relevant information was removed in less than 1% of the 5,042 processed narratives; 97% of the narratives were completely untouched. Conclusions: Automated redaction of names in free-text narratives of adverse event reports can achieve sufficient recall including shorter tokens like patient initials. In-depth review shows that the rare leaks that occur tend not to compromise patient confidentiality. Precision and false positive rates are acceptable with almost all clinically relevant information retained.Keywords
This publication has 28 references indexed in Scilit:
- De-identifying Australian hospital discharge summaries: An end-to-end framework using ensemble of deep learning modelsJournal of Biomedical Informatics, 2022
- 21st ISoP Annual Meeting “A New Era of Pharmacovigilance: Challenges and Opportunities” 20–23 September 2022 Verona, ItalyDrug Safety, 2022
- The OpenDeID corpus for patient de-identificationScientific Reports, 2021
- Improving domain adaptation in de-identification of electronic health records through self-trainingJournal of the American Medical Informatics Association, 2021
- Building a best-in-class automated de-identification tool for electronic health records through ensemble learningPatterns, 2021
- Protected Health Information filter (Philter): accurately and securely de-identifying free-text clinical notesnpj Digital Medicine, 2020
- Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpusJournal of Biomedical Informatics, 2015
- Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1Journal of Biomedical Informatics, 2015
- Text de-identification for privacy protection: A study of its impact on clinical text information contentJournal of Biomedical Informatics, 2014
- New approaches to drug safety: a pharmacovigilance tool kitNature Reviews Drug Discovery, 2009