Adapting and evaluating a deep learning language model for clinical why-question answering
Authors:
Open Access
- 4 February 2020
- journal article
- research article
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 3 (1) , 16-20
Abstract
Objectives: To adapt and evaluate a deep learning language model for answering why-questions based on patient-specific clinical text. Materials and Methods: Bidirectional encoder representations from transformers (BERT) models were trained with varying data sources to perform SQuAD 2.0 style why-question answering (why-QA) on clinical notes. The evaluation focused on: (1) comparing the merits from different training data and (2) error analysis. Results: The best model achieved an accuracy of 0.707 (or 0.760 by partial match). Training toward customization for the clinical language helped increase 6% in accuracy. Discussion: The error analysis suggested that the model did not really perform deep reasoning and that clinical why-QA might warrant more sophisticated solutions. Conclusion: The BERT model achieved moderate accuracy in clinical why-QA and should benefit from the rapidly evolving technology. Despite the identified limitations, it could serve as a competent proxy for question-driven clinical information extraction.Keywords
All Related Versions
Funding Information
- i2b2 National Center for Biomedical Computing (U54LM008748)
This publication has 5 references indexed in Scilit:
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- MEANS: A medical question-answering system combining NLP techniques and semantic Web technologiesInformation Processing & Management, 2015
- AskHERMES: An online question answering system for complex clinical questionsJournal of Biomedical Informatics, 2011
- Agreement, the F-Measure, and Reliability in Information RetrievalJournal of the American Medical Informatics Association, 2005
- Analysis of questions asked by family doctors regarding patient careBMJ, 1999