Qualitative Evaluation of an Artificial Intelligence-based Clinical Decision Support System to Guide Rhythm Management of Atrial Fibrillation (Preprint)
Authors:
- John StacyJSJohn Stacy
- Rachel KimRKRachel Kim
- Christopher D BarrettCBChristopher D Barrett
- Balaviknesh SekarBSBalaviknesh Sekar
- Steve SimonSSSteve Simon
- Farnoush Banaei-KashaniFBFarnoush Banaei-Kashani
- Michael A. RosenbergMRMichael A. Rosenberg
Preprint
- 14 January 2022
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Despite the numerous studies evaluating various rhythm-control strategies for patients with atrial fibrillation, determination of the optimal strategy in a single patient is often based on trial-and-error, with no one-size-fits-all approach based on international guideline recommendations. The decision, therefore, remains an individualized one that lends itself well to the aid of a clinical decision support system, specifically one guided by artificial intelligence (AI). OBJECTIVE In this study, we performed qualitative evaluation of a novel, AI-based, clinical decision support system (CDSS) for rhythm management of AF called QRhythm, which uses a combination of supervised and reinforcement learning to recommend either a rate-control or one of three types of rhythm-control strategies—external cardioversion, antiarrhythmic medication, or ablation—based on individual patient characteristics. METHODS QRhythm utilizes a two-stage machine learning model to identify the optimal rhythm-management strategy in a given patient based on a set of clinical factors, in which the model first uses supervised learning to mimic an expert clinician, and then applies reinforcement learning to identify the best strategy to obtain the best clinical outcome—a composite of symptomatic recurrence, hospitalization, and stroke. 33 clinical providers, including cardiologists and internal medicine physicians, performed an assessment of QRhythm, followed by a survey to assess relative comfort with automated CDSS in rhythm-management, and examine areas for future development. RESULTS 33 providers were surveyed with training levels ranging from resident to fellow to attending. Of the characteristics of the app surveyed, safety was most important to providers with an average importance rating of 4.7 out of 5 (SD 0.72). This priority was followed by a desire for the advice provided to make clinical sense with an importance of 4.5 out of 5 (SD 0.9), transparency in the population used to create the algorithm, 4.3 out of 5 (SD 0.65), and reasoning behind the decisions made, 4.3 out of 5 (SD0.88), and the ability to challenge the decisions made by the model, 3.85 out of 5 (SD 0.83). Providers that used the app ranked their highest concern with ongoing clinical use of the model as accuracy of recommendations followed by ineffectiveness of the application and patient data security. Trust in the app was varied. 16.7% of providers responded that they somewhat disagreed with the statement “I trust the recommendations provided by the QRhythm app,” 33.3% responded with neutrality to the statement, and 50% somewhat agreed with the statement. CONCLUSIONS Safety of machine learning applications was the number one priority of the providers surveyed, and trust of such models remains varied. Widespread clinical acceptance of machine learning in health care is dependent on how much providers trust the algorithms. Building this trust involves ensuring transparency and interpretability of the model.Keywords
This publication has 43 references indexed in Scilit:
- Evaluation of risk stratification schemes for ischaemic stroke and bleeding in 182 678 patients with atrial fibrillation: the Swedish Atrial Fibrillation cohort studyEuropean Heart Journal, 2012
- Incidence and Prevalence of Atrial Fibrillation and Associated Mortality Among Medicare Beneficiaries: 1993–2007Circulation: Cardiovascular Quality and Outcomes, 2012
- Use of UpToDate and outcomes in US hospitalsJournal of Hospital Medicine, 2011
- The Cox maze IV procedure: Predictors of late recurrenceThe Journal of Thoracic and Cardiovascular Surgery, 2011
- The Effect of Computerized Physician Order Entry with Clinical Decision Support on the Rates of Adverse Drug Events: A Systematic ReviewJournal of General Internal Medicine, 2008
- Time Trends of Ischemic Stroke Incidence and Mortality in Patients Diagnosed With First Atrial Fibrillation in 1980 to 2000Stroke, 2005
- Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient OutcomesJAMA, 2005
- A Comparison of Rate Control and Rhythm Control in Patients with Atrial FibrillationNew England Journal of Medicine, 2002
- Amiodarone to Prevent Recurrence of Atrial FibrillationNew England Journal of Medicine, 2000
- A theoretical framework for decision trees in uncertain domains: Application to medical data sets1997