Clinical Validation and Machine Learning Optimization of MyCog: A Self-Administered Cognitive Screener for Primary Care Settings
Preprint
Abstract
Importance Early detection of cognitive impairment in older adults is crucial for identifying reversible causes, managing symptoms, and enabling future planning; yet primary care settings face significant barriers to implementing effective cognitive screening. Objective To clinically validate an innovated self-administered, tablet-based cognitive screening tool (“MyCog”, adapted from the NIH Toolbox Cognitive Battery) and to develop an optimized scoring algorithm for detecting cognitive impairment in older adults during primary care visits. Design Cross-sectional clinical validation study conducted between summer 2021 and fall 2024, comparing cognitive performance between impaired and non-impaired participants using multiple machine learning approaches to identify optimal predictors of impairment. Setting Academic medical centers and community-based general population recruitment. Participants 248 adults aged 65 and older, including 70 participants with clinically diagnosed cognitive impairment (MCI or dementia), 81 participants confirmed to be cognitively intact from clinical settings, and 97 participants from the general population. Participants were predominantly White (88%) with varied educational backgrounds and a mean age of 76.3 years (SD = 6.9). Main Outcomes and Measures Sensitivity, specificity, and area under the receiver operating characteristic curve (ROC AUC) were the primary metrics used for evaluating model fit in classifying cognitive impairment of known groups from the MyCog Picture Sequence Memory and Dimensional Change Card Sort tasks. Results Across nine machine learning model classes, Picture Sequence Memory exact match score and Dimensional Change Card Sort rate-correct score were identified as the most robust predictors of cognitive impairment. The final optimized model achieved a sensitivity of 82.5% (bootstrapped 95% CI [0.733, 0.913]) and a specificity of 76.8% (bootstrapped 95% CI [0.709, 0.828]) with an ROC AUC of 0.892, comparable to traditional, long form cognitive screening measures while requiring less than 7 minutes to self-administer. Conclusions and Relevance MyCog demonstrates strong diagnostic accuracy for detecting cognitive impairment in older adults with a parsimonious model focusing on episodic memory and executive function. As a brief, self-administered screening tool that can be integrated into electronic health records, MyCog addresses key barriers to cognitive screening in primary care settings, which may help increase screening and early detection rates.Keywords
This publication has 55 references indexed in Scilit:
- Optimizing Health for Complex Adults in Primary Care: Current Challenges and a Way ForwardJournal of General Internal Medicine, 2014
- Measuring Episodic Memory Across the Lifespan: NIH Toolbox Picture Sequence Memory TestJournal of the International Neuropsychological Society, 2014
- Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the medianJournal of Experimental Social Psychology, 2013
- Communication skills training in dementia care: a systematic review of effectiveness, training content, and didactic methods in different care settingsInternational Psychogeriatrics, 2013
- Episodic memory on the path to Alzheimer's diseaseCurrent Opinion in Neurobiology, 2011
- Time to confront the global dementia crisisThe Lancet Neurology, 2008
- Dementia Screening in Primary CareJAMA, 2007
- Executive function deficits in early Alzheimer's disease and their relations with episodic memoryArchives of Clinical Neuropsychology, 2006
- Neuropsychology of cognitive ageing, minimal cognitive impairment, Alzheimer's disease, and vascular cognitive impairmentEuropean Journal of Pharmacology, 2004
- The ABC of Alzheimer’s Disease: Cognitive Changes and Their Management in Alzheimer’s Disease and Related DementiasInternational Psychogeriatrics, 2002