Identifying Functional Status Impairment in People Living With Dementia Through Natural Language Processing of Clinical Documents: Cross-Sectional Study (Preprint)
Preprint
- 30 March 2023
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Assessment of activities of daily living (ADLs) and instrumental ADLs (iADLs) is key to determining the severity of dementia and care needs among older adults. However, such information is often only documented in free-text clinical notes within the electronic health record and can be challenging to find. OBJECTIVE This study aims to develop and validate machine learning models to determine the status of ADL and iADL impairments based on clinical notes. METHODS This cross-sectional study leveraged electronic health record clinical notes from Mass General Brigham’s Research Patient Data Repository linked with Medicare fee-for-service claims data from 2007 to 2017 to identify individuals aged 65 years or older with at least 1 diagnosis of dementia. Notes for encounters both 180 days before and after the first date of dementia diagnosis were randomly sampled. Models were trained and validated using note sentences filtered by expert-curated keywords (filtered cohort) and further evaluated using unfiltered sentences (unfiltered cohort). The model’s performance was compared using area under the receiver operating characteristic curve and area under the precision-recall curve (AUPRC). RESULTS The study included 10,000 key-term–filtered sentences representing 441 people (n=283, 64.2% women; mean age 82.7, SD 7.9 years) and 1000 unfiltered sentences representing 80 people (n=56, 70% women; mean age 82.8, SD 7.5 years). Area under the receiver operating characteristic curve was high for the best-performing ADL and iADL models on both cohorts (>0.97). For ADL impairment identification, the random forest model achieved the best AUPRC (0.89, 95% CI 0.86-0.91) on the filtered cohort; the support vector machine model achieved the highest AUPRC (0.82, 95% CI 0.75-0.89) for the unfiltered cohort. For iADL impairment, the Bio+Clinical bidirectional encoder representations from transformers (BERT) model had the highest AUPRC (filtered: 0.76, 95% CI 0.68-0.82; unfiltered: 0.58, 95% CI 0.001-1.0). Compared with a keyword-search approach on the unfiltered cohort, machine learning reduced false-positive rates from 4.5% to 0.2% for ADL and 1.8% to 0.1% for iADL. CONCLUSIONS In this study, we demonstrated the ability of machine learning models to accurately identify ADL and iADL impairment based on free-text clinical notes, which could be useful in determining the severity of dementia.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 26, e47739.
This publication has 22 references indexed in Scilit:
- Comparing clinician descriptions of frailty and geriatric syndromes using electronic health records: a retrospective cohort studyBMC Geriatrics, 2017
- Prediction Score for Anticoagulation Control Quality Among Older AdultsJournal of the American Heart Association, 2017
- Gendered Expectations Distort Male–Female Differences in Instrumental Activities of Daily Living in Later AdulthoodThe Journals of Gerontology, Series B: Psychological Sciences and Social Sciences, 2017
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Considerations for the analysis of longitudinal electronic health records linked to claims data to study the effectiveness and safety of drugsClinical Pharmacology & Therapeutics, 2016
- Mild cognitive impairment and deficits in instrumental activities of daily living: a systematic reviewAlzheimer's Research & Therapy, 2015
- TF-IDF2012
- Falls and Risk Factors for Falls in Community-Dwelling Adults With Dementia (NutriAlz Trial)Alzheimer Disease & Associated Disorders, 2012
- Activities of daily living performance in dementiaActa Neurologica Scandinavica, 2007
- Random Forest: A Classification and Regression Tool for Compound Classification and QSAR ModelingJournal of Chemical Information and Computer Sciences, 2003