Onil Bhattacharyya

Background: The COVID-19 pandemic accelerated the adoption of virtual care, including patient-to-provider asynchronous secure messaging (ASM). However, while ASM offers access and convenience, the surge in message volume can increase physician workload and contribute to burnout. Furthermore, it has raised additional concerns about message appropriateness, medicolegal liability, and sustainability of this communication virtual care modality. Objective: This study aims to explore the experience with ASM in Ontario in specialty ambulatory care to inform current practice support strategies. We sought to characterize messaging usage quantitatively while also exploring experience with messaging, contextual barriers, and facilitators qualitatively. Methods: We used a convergent mixed methods design, analyzing aggregated electronic health record (EHR) usage and qualitative interview data. The study was conducted at a large ambulatory care hospital in Ontario, Canada, across four high-messaging specialty clinics: mental health, rheumatology, dermatology, and surgery. Quantitative data (October 2019 to Octtober 2022) included message volumes, response patterns, and timing. Semistructured interviews with physicians, nurses, administrative staff, and patients (n=19), conducted in 2022 explored ASM workflows, barriers, and facilitators. Quantitative data were analyzed descriptively, stratified by clinic. Qualitative data were analyzed using thematic analysis, informed by an interpretative phenomenological orientation. Findings from the two strands were then compared to identify areas of convergence and divergence. Results: Message volumes surged post pandemic, particularly in mental health, ranging from about 6000 in surgery, rheumatology, and dermatology to close to 13,000 messages in mental health across the study period. Message rate per patient per month was 0.6 (SD 0.2; mental health), 0.5 (SD 0.2; rheumatology), 0.2 (SD 0.2; dermatology), and 0.2 (SD 0.1; surgery). Physicians averaged 12.6 (SD 8.0) messages/month (rheumatology) and 7.6 (SD 4.4) messages/month (mental health) in clinics without nursing support. Nurses handled 27.4 (SD 11.6) messages/month (surgery) and 19.2 (SD 14.1) messages/month (dermatology). Three percent (1263/38,370) to 10% (3619/36,022) of provider messages were sent outside regular hours. The majority of conversations (2156/6859, 46% to 6913/13,615, 62%) had only a single reply. Physicians reported challenging experiences due to increased workload, lack of compensation, and poor integration into local clinical workflows. High patient-to-physician ratios and limited nursing support for message triage were associated with a poor messaging experience. Patients and administrative staff valued messaging for its convenience, accessibility, and efficiency. A key finding was insufficient engagement across all user groups in decisions about messaging implementation. Conclusions: The study highlights a disconnect between the high perceived value of ASM for patients and administrative staff and the challenging experiences of physicians. Successful implementation requires thoughtful integration into care models, clear guidelines for appropriate use, and proper triage and “channel management” to guide patients to appropriate modalities. Future research should explore triaging algorithms as part of a digital front door, specialty-specific variations, and the crucial role of nursing staff in message channel management.

Background: AI-based clinical decision support systems (AI-CDSSs) can help primary care clinicians provide higher quality care amid rising case complexity and resource constraints. However, evidence regarding their effectiveness and safety in primary care practice is lacking. Objective: We synthesized evidence on the effectiveness and safety of AI-CDSSs compared with usual care or non–AI-based systems in primary care settings. Methods: We conducted a systematic review following the Cochrane Handbook for Systematic Reviews of Interventions and reported the results according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. We included experimental and quasi-experimental studies evaluating AI-CDSSs used by health care professionals in primary care settings, compared with usual care or non–AI-based decision support systems, and that reported clinician-, patient-, or system-level outcomes, or safety. An information specialist designed the search strategies and systematically reviewed MEDLINE, Embase, CINAHL, Web of Science, and CENTRAL from inception to June 27, 2025. Reviewers independently performed the selection and extraction processes. Data extraction was informed by the DECIDE-AI and CONSORT-AI (Consolidated Standards of Reporting Trials–AI) reporting guidelines and the APPRAISE-AI tool. We independently assessed the risk of bias using Risk of Bias-2 or Risk of Bias in Non-Randomized Studies of Interventions. Owing to substantial heterogeneity, we conducted a synthesis without meta-analysis and explored heterogeneity using an adapted Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability–informed implementation-context classification. The review protocol was registered in PROSPERO (CRD420251119225). Results: We identified 4074 records through database searches and 12 additional records through citation searching and trial registration checking. We included 14 records on 11 distinct studies, including 4 experimental and 7 quasi-experimental studies. The risk of bias was high in all experimental studies and serious to critical in quasi-experimental studies. AI-CDSSs showed more consistent positive effects on workflow-related outcomes in structured and protocolized decision contexts, particularly for screening, referral, and alert optimization tasks. Patient-level effects were limited and inconsistent, particularly in more complex contexts involving heterogeneous conditions or preference-sensitive decisions. System-level outcomes were heterogeneous, with isolated improvements in service delivery and utilization outcomes. Safety evaluation was rare and poorly specified. Only 2 studies prespecified safety outcomes, and the reported safety information mainly involved technical malfunctions or self-reported side effects. No study reported clinician interaction measures such as agreement with, override of, or reliance on AI outputs. Interpretation of the findings is limited by substantial heterogeneity, short follow-up periods, predominantly high or critical risk of bias, and evidence derived exclusively from high-income countries. Conclusions: AI-CDSSs in primary care may provide more consistent benefits for workflow-related and process-level outcomes in structured and protocolized decision contexts, while evidence on safety remains insufficient to support confident routine implementation. This systematic review reveals an important gap between the rapid development of AI-CDSSs and the lack of robust, comparable evidence to support their reliable use in primary care.

Objectives: To investigate the performance of artificial intelligence (AI) scribes and their impact on clinical documentation time. Materials and Methods: Artificial intelligence scribes were assessed using clinical simulations with 9 primary care physicians, each conducting 4 simulated encounters with standardized patients with and without the use of an AI scribe. Clinical simulations were audio and video recorded, and videos were coded for documentation behaviors using a designed coding scheme. Results: Clinical documentation accounts for 36.3% of the simulated clinical encounter without the use of the AI scribe, whereas primary care physicians spend 11.2% of the simulated encounter on clinical documentation when using an AI scribe. The use of AI scribes accounts for a 69.1% reduction in documentation time. Discussion: Clinical simulations offered a controlled and realistic environment to assess the impact of AI scribes on primary care physicians’ workflows, demonstrating their potential to reduce documentation time. Conclusion: Artificial intelligence scribes demonstrate potential to alleviate the burden of clinical documentation, addressing a critical factor contributing to physician burnout. Further research is needed to understand the impact of AI scribes in real-life clinical settings across complex patient scenarios.

BACKGROUND Artificial intelligence-based clinical decision support systems (AI-CDSS) have the potential to help primary care physicians provide higher quality care with rising case complexity and resource constraints. However, evidence on their effectiveness and safety in real-world primary care practice is lacking. OBJECTIVE To synthesize evidence on the real-world effectiveness and safety of AI-CDSS compared with usual care or non–AI-based systems in primary care settings. METHODS We conducted a systematic review following the Cochrane Handbook for Systematic Reviews of Interventions and reported the results according to PRISMA 2020. An information specialist designed search strategies and systematically reviewed Medline, Embase, CINAHL, Web of Science, and CENTRAL. Reviewers independently performed the selection and extraction processes. Data extraction was informed by the DECIDE-AI and CONSORT-AI reporting guidelines, and the APPRAISE-AI tool. We independently assessed risk of bias using RoB-2 or ROBINS-I. Due to heterogeneity, we conducted a narrative synthesis across clinician-, patient-, and system-level outcomes as well as safety information. RESULTS We identified 4085 records and selected 14 records on 11 distinct studies. We identified 11 AI-CDSS across four experimental (high risk of bias) and seven quasi-experimental studies (serious to critical risk of bias). All studies were conducted in high-income countries. Follow-up ranged from one to 12 months. Machine-learning and deep-learning approaches were the most common (n=8/11). The interventions fulfilled different decision-support functions, most frequently treatment or order facilitation (n=8/11). Clinician-level outcomes were most frequently reported (n=8/11), followed by system-level (n=6/11) and patient-level outcomes (n=5/11). Clinician-level findings were mixed, with several studies reporting reductions in administrative or alert burden. System-level outcomes were heterogeneous and sparsely reported, with isolated studies suggesting improvements in cost-effectiveness, service utilization, patient engagement, program completion, and care delivery efficiency. Patient-level effects were limited or inconsistent, with improvements mainly observed in well-defined conditions with established diagnostic and treatment pathways. Safety evaluation was rare (n=2/11) and limited to technical malfunctions or self-reported side effects. CONCLUSIONS AI-CDSS implemented in primary care demonstrate limited and inconsistent effectiveness at clinician-, patient-, and system-level. Safety evaluation relied mainly on self-reported side effects and technical malfunctions, without structured monitoring for AI-related risks. These findings highlight a gap between the pace of innovation and the readiness of healthcare systems for reliable clinical use. Closing this gap will require workflow-integrated system design, timely and standardized evaluation, and active safety monitoring to support trustworthy implementation in primary care.

We leveraged linked, health-administrative data to compare the real-world effectiveness of Internet-delivered iCBT (online modules with minimal, asynchronous therapist support) to therapist-guided CBT (fully-synchronous, scheduled, one-on-one sessions) for depression and anxiety disorders at an outpatient clinic within a large psychiatric hospital in Ontario, Canada (Jan 2020-Aug 2021). Using Difference-in-Differences analyses, we assessed pre-post change in depression (PHQ-9) and anxiety (GAD-7) symptoms and examined mental health (MH)-related service utilization (emergency department visits, hospitalizations, outpatient visits) using Logistic and Negative Binomial regressions. Among N = 167 iCBT and N = 300 CBT participants, treatment modality was significantly associated with pre-post symptom change (p < 0.01), with greater reductions in CBT (-4pt PHQ-9, -3.6pt GAD-7), compared to iCBT (-2.3pt PHQ-9, + 0.4pt GAD-7). Sensitivity analyses restricting to treatment completers somewhat attenuated this finding. Both groups experienced reductions in MH-related service use, though iCBT clients were more likely to continue visiting acute (OR_adj = 2.92, 95% CI 1.29-6.64) and outpatient services (RR_adj = 1.32, 95% CI 1.03-1.69) 1-year post-treatment, adjusted for age, comorbidities, and baseline symptoms. In routine care, minimally-guided iCBT was associated with more modest symptom reductions than therapist-guided CBT. While iCBT remains a promising approach to meeting increasing MH service demands, efforts to ensure treatment adherence are likely needed to maximize its utility.

BACKGROUND The COVID-19 pandemic significantly increased adoption of virtual care, including patient-to-provider secure messaging. However, this surge has heightened physician workload and burnout and has raised concerns about message appropriateness and liability among physicians. OBJECTIVE This study characterizes secure messaging use in Canadian hospital-based specialty care and explores the experiences of healthcare providers, administrative staff, and patients. METHODS We employed a convergent mixed-methods design, analyzing aggregated electronic health record (EHR) usage data and qualitative interview data. The study was conducted at Women’s College Hospital in Toronto, Canada, across four high-messaging specialty clinics: mental health, rheumatology, dermatology, and surgery. Quantitative data (Oct, 2019-Oct, 2022) detailing message volumes, response patterns, and timing. Semi-structured interviews explored messaging workflows, barriers, and facilitators. Data were analyzed separately, then converged to identify areas of convergence and divergence. RESULTS Message volumes surged post-pandemic, particularly in mental health. The monthly message rate per patient varied, with higher rates in mental health and rheumatology. Physicians reported negative experiences due to increased workload, lack of compensation, and inadequate integration into clinical workflows. High patient-to-physician ratios and limited nursing support for message triage were associated with a poor messaging experience. Patients and administrative staff valued messaging for its convenience, accessibility, and efficiency. A key finding was the poor engagement of all user groups in decisions regarding messaging implementation. CONCLUSIONS The study highlights a disconnect between the high perceived value of secure messaging for patients and administrative staff and the negative experiences of physicians. Successful implementation requires thoughtful integration into care models, clear guidelines for patient use, and proper triage and "channel management" to guide patients to appropriate visit modalities. Future research should explore triaging algorithms as part of a digital front door, specialty-specific variations and the crucial role of nursing staff in message management.

Background: Primary care providers (PCPs) face significant burnout due to increasing administrative and documentation demands, contributing to job dissatisfaction and impacting care quality. Artificial intelligence (AI) scribes have emerged as potential solutions to reduce administrative burden by automating clinical documentation of patient encounters. Although AI scribes are gaining popularity in primary care, there is limited information on their usability, effectiveness, and accuracy. Objective: This study aimed to develop and apply an evaluation framework to systematically assess the usability, technical performance, and accuracy of various AI scribes used in primary care settings across Canada and the United States. Methods: We conducted a systematic comparison of a suite of AI scribes using competitive analysis methods. An evaluation framework was developed using expert usability approaches and human factors engineering principles and comprises 3 domains: usability, effectiveness and technical performance, and accuracy and quality. Audio files from 4 standardized patient encounters were used to generate transcripts and SOAP (Subjective, Objective, Assessment, and Plan)–format medical notes from each AI scribe. A verbatim transcript, detailed case notes, and physician-written medical notes for each audio file served as a benchmark for comparison against the AI-generated outputs. Applicable items were rated on a 3-point Likert scale (1=poor, 2=good, 3=excellent). Additional insights were gathered from clinical experts, vendor questionnaires, and public resources to support usability, effectiveness, and quality findings. Results: In total, 6 AI scribes were evaluated, with notable performance differences. Most AI scribes could be accessed via various platforms (n=4) and launched within common electronic medical records, though data exchange capabilities were limited. Nearly all AI scribes generated SOAP-format notes in approximately 1 minute for a 15-minute standardized encounter (n=5), though documentation time increased with encounter length and topic complexity. While all AI scribes produced good to excellent quality medical notes, none were consistently error-free. Common errors included deletion, omission, and SOAP structure errors. Factors such as extraneous conversations and multiple speakers impacted the accuracy of both the transcript and medical note, with some AI scribes producing excellent notes despite minor transcript issues and vice versa. Limitations in usability, technical performance, and accuracy suggest areas for improvement to fully realize AI scribes’ potential in reducing administrative burden for PCPs. Conclusions: This study offers one of the first systematic evaluations of the usability, effectiveness, and accuracy of a suite of AI scribes currently used in primary care, providing benchmark data for further research, policy, and practice. While AI scribes show promise in reducing documentation burdens, improvements and ongoing evaluations are essential to ensure safe and effective use. Future studies should assess AI scribe performance in real-world settings across diverse populations to support equitable and reliable applications.

Government decisions about what digital health care programs will be funded, for how much money, for how many years, and for what use cases, need to occur rapidly based on the best information available at the time. These decisions also need to align with government funding timelines and strategic priorities. This rapid decision-making process contrasts with the traditional research process where programs are piloted, and evidence is generated regarding the program’s effectiveness from multiple perspectives over several years ... [continues in PDF / HTML]

Background: While the volume of virtual visits increased with the COVID-19 pandemic, little is still known regarding the quality of virtual visits. Furthermore, there is limited guidance on how best to evaluate the quality of virtual care initiatives. Objective: The objective of this study was to curate a balanced set of quality indicators grounded in the Quintuple Aim quality framework and the National Academy of Medicine domains of quality, including sustainability, to monitor the quality of virtual care in ambulatory environments. Methods: Phase 1 involved curating a list of ambulatory virtual care quality indicators from published and grey literature, along with knowledge user feedback generated through a pre-Delphi survey; these indicators were mapped and categorized to the Quintuple Aim and National Academy of Medicine (NAM) Quality Domains. In phase 2, a 19-member pan-Canadian panel was convened as part of a 3-round modified Delphi panel process to rate and rank the quality indicators. The panel included clinicians, individuals with lived experience with the health care system, policy makers, academics, and administrators. Panelists rated indicators using the Agency for Healthcare Research and Quality measure attributes on a 9-point Likert scale in round 1, followed by a virtual Delphi panel meeting to discuss indicators before round 2 of re-rating, then a final round 3 of ranking which assessed the importance of indicators within each quality domain and subdomain. To address gaps in the literature, novel quality indicators of virtual care were identified and discussed in panel discussions, patient group consultations, and a pre-Delphi survey. For indicator advancement during the rating exercises, 75% or more of the panelists’ responses in the top tertile (scores of 7-9) with a median composite score of 7 or greater was required. Results: There were 140 quality indicators identified in the literature which progressed through the 3 Delphi panel rounds. There was minimal attrition among Delphi panel members (17/19, 89% participated in all 3 rounds). After round 3, 25 indicators were included in the final scorecard. Aligned with the Quintuple Aim, there are 13 quality indicators on patient experience, 6 on provider experience, 2 on population health, 2 on health equity, and 2 on health system costs. Conclusions: A balanced set of 25 quality indicators of ambulatory virtual care was developed based on literature and consensus building from knowledge users across the health system. This curated set of indicators will support more comprehensive evaluations of virtual care. Organizations can use the set of indicators as part of a scorecard to compare across jurisdictions, identify priority areas, and ensure their virtual care initiatives are delivering high-quality care based on multiple domains of quality.