Anita M. Preininger

BACKGROUND Patients more actively engaged in their care have better clinical outcomes. OBJECTIVE To examine the performance of a widely adopted cloud-based patient engagement system for management of diabetes, hypertension, and preventative screenings at two large healthcare systems. METHODS We included patients with diabetes, hypertension, and those due for breast cancer screenings and/or bi-annual wellness visits and at Loudon Medical Group (LMG) in Virginia, and Basset Medical Group (BMG) in New York between 2019-2021. We examined outreach outcomes including successful contact rate (SCR), patient response rate (PRR), and adherent response rate (ARR). Associations between demographic characteristics and outcome measures were explored with Chi-square or Fisher’s exact tests. RESULTS Attempted contacts with patients with diabetes (N=2567) at LMG resulted in SCR of 87.1%, PRR of 47.2%, and ARR of 38.4%. At BMG, attempted contacts with patients with diabetes (N=3621), hypertension (N=10881), patients for breast cancer screening (N=2441), and wellness visits (N=9070) resulted in SCR values of 91.0%, 93.2%, 90.1%, and 92.1%; PRR of 57.0%, 59.8%, 50.8%, and 52.0%; and ARR of 45.2%, 39.8%, 6.8%, and 11.5% respectively. Outreach outcomes measures were significantly higher in patients with hypertension, and those carrying commercial or Medicare insurance vs. those who were self-insured or on Medicaid plans (SCR P<.0001, PRR P<.0001, and ARR P=.0013). Among diabetic patients, only SCR (P=.0008) was significantly associated with commercial or Medicare insurance. In general, rate of contact (SCR) had more favorable results as compared to the rate of follow-up visits (PRR, ARR), with higher PRR significantly associated with female sex (P<.0001), and a higher SCR seen among patients contacted by text (P<.0001). Age-stratified analysis showed that 60-69 years old diabetic patients had the highest SCR and ARR. Hypertensive patients between the ages of 70-79 had the highest SCR, PRR, and ARR. CONCLUSIONS This cloud-based patient outreach system demonstrated high rates of contact in patients with hypertension, diabetes and/or eligible for preventative screenings. Overall response rates were over 50%, with moderate adherence rates for diabetes and hypertension, but low adherence for preventative screenings. Older patients had higher adherence rates. Additional strategies are needed to translate successful patient contact into response and adherence. CLINICALTRIAL NA

Background: Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. Objective: This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials. Methods: This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test. Results: In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. Conclusions: The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

BACKGROUND Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. OBJECTIVE This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials. METHODS This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test. RESULTS In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. CONCLUSIONS The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

The rapidly evolving science about the Coronavirus Disease 2019 (COVID-19) pandemic created unprecedented health information needs and dramatic changes in policies globally. We describe a platform, Watson Assistant (WA), which has been used to develop conversational agents to deliver COVID-19 related information. We characterized the diverse use cases and implementations during the early pandemic and measured adoption through a number of users, messages sent, and conversational turns (ie, pairs of interactions between users and agents). Thirty-seven institutions in 9 countries deployed COVID-19 conversational agents with WA between March 30 and August 10, 2020, including 24 governmental agencies, 7 employers, 5 provider organizations, and 1 health plan. Over 6.8 million messages were delivered through the platform. The mean number of conversational turns per session ranged between 1.9 and 3.5. Our experience demonstrates that conversational technologies can be rapidly deployed for pandemic response and are adopted globally by a wide range of users.

IBM^(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice. This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH’s institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage. Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest. This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

PURPOSE: To examine the impact of a clinical decision support system (CDSS) on breast cancer treatment decisions and adherence to National Comprehensive Cancer Center (NCCN) guidelines. PATIENTS AND METHODS: A cross-sectional observational study was conducted involving 1,977 patients at high risk for recurrent or metastatic breast cancer from the Chinese Society of Clinical Oncology. Ten oncologists provided blinded treatment recommendations for an average of 198 patients before and after viewing therapeutic options offered by the CDSS. Univariable and bivariable analyses of treatment changes were performed, and multivariable logistic regressions were estimated to examine the effects of physician experience (years), patient age, and receptor subtype/TNM stage. RESULTS: Treatment decisions changed in 105 (5%) of 1,977 patients and were concentrated in those with hormone receptor (HR)–positive disease or stage IV disease in the first-line therapy setting (73% and 58%, respectively). Logistic regressions showed that decision changes were more likely in those with HR-positive cancer (odds ratio [OR], 1.58; P < .05) and less likely in those with stage IIA (OR, 0.29; P < .05) or IIIA cancer (OR, 0.08; P < .01). Reasons cited for changes included consideration of the CDSS therapeutic options (63% of patients), patient factors highlighted by the tool (23%), and the decision logic of the tool (13%). Patient age and oncologist experience were not associated with decision changes. Adherence to NCCN treatment guidelines increased slightly after using the CDSS (0.5%; P = .003). CONCLUSION: Use of an artificial intelligence–based CDSS had a significant impact on treatment decisions and NCCN guideline adherence in HR-positive breast cancers. Although cases of stage IV disease in the first-line therapy setting were also more likely to be changed, the effect was not statistically significant ( P = .22). Additional research on decision impact, patient-physician communication, learning, and clinical outcomes is needed to establish the overall value of the technology.

Objective: The objective of this technical study was to evaluate the performance of an artificial intelligence (AI)-based system for clinical trials matching for a cohort of lung cancer patients in an Australian cancer hospital. Methods: A lung cancer cohort was derived from clinical data from patients attending an Australian cancer hospital. Ten phases I–III clinical trials registered on clinicaltrials.gov and open to lung cancer patients at this institution were utilized for assessments. The trial matching system performance was compared to a gold standard established by clinician consensus for trial eligibility. Results: The study included 102 lung cancer patients. The trial matching system evaluated 7252 patient attributes (per patient median 74, range 53–100) against 11 467 individual trial eligibility criteria (per trial median 597, range 243–4132). Median time for the system to run a query and return results was 15.5 s (range 7.2–37.8). In establishing the gold standard, clinician interrater agreement was high (Cohen’s kappa 0.70–1.00). On a per-patient basis, the performance of the trial matching system for eligibility was as follows: accuracy, 91.6%; recall (sensitivity), 83.3%; precision (positive predictive value), 76.5%; negative predictive value, 95.7%; and specificity, 93.8%. Discussion and Conclusion: The AI-based clinical trial matching system allows efficient and reliable screening of cancer patients for clinical trials with 95.7% accuracy for exclusion and 91.6% accuracy for overall eligibility assessment; however, clinician input and oversight are still required. The automated system demonstrates promise as a clinical decision support tool to prescreen a large patient cohort to identify subjects suitable for further assessment.

This article describes the system architecture, training, initial use, and performance of Watson Assistant (WA), an artificial intelligence-based conversational agent, accessible within Micromedex^®. The number and frequency of intents (target of a user’s query) triggered in WA during its initial use were examined; intents triggered over 9 months were compared to the frequency of topics accessed via keyword search of Micromedex. Accuracy of WA intents assigned to 400 queries was compared to assignments by 2 independent subject matter experts (SMEs), with inter-rater reliability measured by Cohen’s kappa. In over 126 000 conversations with WA, intents most frequently triggered involved dosing (N = 30 239, 23.9%) and administration (N = 14 520, 11.5%). SMEs with substantial inter-rater agreement (kappa = 0.71) agreed with intent mapping in 247 of 400 queries (62%), including 16 queries related to content that WA and SMEs agreed was unavailable in WA. SMEs found 57 (14%) of 400 queries incorrectly mapped by WA; 112 (28%) queries unanswerable by WA included queries that were either ambiguous, contained unrecognized typographical errors, or addressed topics unavailable to WA. Of the queries answerable by WA (288), SMEs determined 231 (80%) were correctly linked to an intent. A conversational agent successfully linked most queries to intents in Micromedex. Ongoing system training seeks to widen the scope of WA and improve matching capabilities. WA enabled Micromedex users to obtain answers to many medication-related questions using natural language, with the conversational agent facilitating mapping to a broader distribution of topics than standard keyword searches.

Introduction: Artificial intelligence (AI) technologies continue to attract interest from a broad range of disciplines in recent years, including health. The increase in computer hardware and software applications in medicine, as well as digitization of health-related data together fuel progress in the development and use of AI in medicine. This progress provides new opportunities and challenges, as well as directions for the future of AI in health. Objective: The goals of this survey are to review the current state of AI in health, along with opportunities, challenges, and practical implications. This review highlights recent developments over the past five years and directions for the future. Methods: Publications over the past five years reporting the use of AI in health in clinical and biomedical informatics journals, as well as computer science conferences, were selected according to Google Scholar citations. Publications were then categorized into five different classes, according to the type of data analyzed. Results: The major data types identified were multi-omics, clinical, behavioral, environmental and pharmaceutical research and development (R&D) data. The current state of AI related to each data type is described, followed by associated challenges and practical implications that have emerged over the last several years. Opportunities and future directions based on these advances are discussed. Conclusion: Technologies have enabled the development of AI-assisted approaches to healthcare. However, there remain challenges. Work is currently underway to address multi-modal data integration, balancing quantitative algorithm performance and qualitative model interpretability, protection of model security, federated learning, and model bias.