An in-depth evaluation of federated learning on biomedical natural language processing for information extraction
Open Access
- 15 May 2024
- journal article
- research article
- Published by Springer Nature in npj Digital Medicine
- Vol. 7 (1) , 1-9
Abstract
Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: (1) FL models consistently outperformed models trained on individual clients’ data and sometimes performed comparably with models trained with polled data; (2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. (3) FL models significantly outperformed pre-trained LLMs with few-shot prompting.Keywords
This publication has 32 references indexed in Scilit:
- Fine-tuning large neural language models for biomedical natural language processingPatterns, 2023
- A large language model for electronic health recordsnpj Digital Medicine, 2022
- Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitalsJournal of the American Medical Informatics Association, 2022
- CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health recordsJournal of the American Medical Informatics Association, 2022
- Federated Learning for Open Banking2020
- How to Fine-Tune BERT for Text Classification?2019
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- NCBI disease corpus: A resource for disease name recognition and concept normalizationJournal of Biomedical Informatics, 2014
- The EU-ADR corpus: Annotated drugs, diseases, targets, and their relationshipsJournal of Biomedical Informatics, 2012