Karen Crotty

Data extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction. To compare an AI-assisted versus a traditional, human-only data extraction process. Study within reviews (SWAR) using a prospective, parallel-group comparison with blinded data adjudicators. Workflow validation within 6 ongoing systematic reviews of interventions under real-world conditions. Initial data extraction using an LLM (Claude, versions 2.1, 3.0 Opus, and 3.5 Sonnet) verified by a human reviewer. Concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis. The 6 systematic reviews in the SWAR yielded 9341 data elements from 63 studies. Concordance between the 2 methods was 77.2% (95% CI, 76.3% to 78.0%). Compared with the reference standard, the AI-assisted approach had an accuracy of 91.0% (CI, 90.4% to 91.6%) and the human-only approach an accuracy of 89.0% (CI, 88.3% to 89.6%). Sensitivities were 89.4% (CI, 88.6% to 90.1%) and 86.5% (CI, 85.7% to 87.3%), respectively, with positive predictive values of 99.2% (CI, 99.0% to 99.4%) and 98.9% (CI, 98.6% to 99.1%). Incorrect data were extracted in 9.0% (CI, 8.4% to 9.6%) of AI-assisted cases and 11.0% (CI, 10.4% to 11.7%) of human-only cases, with corresponding proportions of major errors of 2.5% (CI, 2.2% to 2.8%) versus 2.7% (CI, 2.4% to 3.1%). Missed data items were the most frequent error type in both approaches. The AI-assisted method reduced data extraction time by a median of 41 minutes per study. Assessing concordance and classifying errors required subjective judgment. Consistently tracking time on task was challenging. Data extraction assisted by AI may offer a viable, more efficient alternative to human-only methods. Agency for Healthcare Research and Quality and RTI International.

Background: There is a lack of consensus on the effectiveness of audio-based care to manage chronic conditions. This knowledge gap has implications for health policy decisions and for health equity, as underserved populations are more likely to access care by telephone. Objectives: We compared the effectiveness of audio-based care to usual care for managing chronic conditions (except diabetes). Design: We used systematic review methods to synthesize available evidence. Studies: We searched for English-language articles reporting on randomized controlled trials (RCTs) of adults diagnosed with a chronic condition published since 2012. Outcomes: We abstracted data on clinical outcomes, patient-reported health and quality of life, health care access and utilization, care quality and experience, and patient safety. Results: We included 40 RCTs evaluating audio-based care for a variety of chronic conditions, including cancer, heart failure, neurological disease, respiratory disease, musculoskeletal conditions, kidney disease, and others. There was significant heterogeneity across conditions and interventions. We generally found low to very low certainty of evidence of comparable effectiveness in the use of audio-based care to replace other care. Audio care as a supplement exhibited greater effectiveness in some outcomes, with generally low to very low certainty of evidence for most outcomes but moderate certainty for 2 groups of study outcomes. Conclusions: More research is needed to identify the conditions, populations, and intervention design combinations that improve outcomes and to determine when audio-based care can effectively replace other synchronous care.

Objectives: We compared the effectiveness of audio-based care, as a replacement or a supplement to usual care, for managing diabetes. Background: Diabetes is a chronic condition afflicting many in the United States. The impact of audio-based care on the health of individuals with diabetes is unclear, particularly for those at risk for disparities—many of whom may only be able to access telehealth services through telephone. Methods: We used systematic review methods to synthesize available evidence. We systematically searched for English-language articles from 2012 reporting randomized controlled trials of adults diagnosed with diabetes. We abstracted data on clinical outcomes (including A1c), patient-reported health and quality-of-life, health care access and utilization, care quality and experience, and patient safety. Results: Evidence for replacing in-person care with audio care was limited (n = 2), with low certainty of evidence for greater and comparable effectiveness for A1c and harms, respectively. Supplemental audio care (n = 23) had a positive effect on A1c (pooled mean difference A1c −0.20%; n = 8763; 95% CI: −0.36% to −0.04%), with moderate certainty of evidence. Stratified results indicated that audio interventions supplementing usual care performed more favorably in individuals with A1c ≤ 9%; populations not at risk of disparities; interventions with at least monthly contact; and interventions using remote monitoring tools. Conclusions: This evidence base reveals some promise for managing diabetes with audio-based care as a supplement to in-person care. Future studies could further investigate the effectiveness of audio-based care as a replacement and modify interventions to better serve individuals with poor glucose control and those at risk for disparities.

Replies to comments made by Mattos et al. (see record 2025-49982-003) on the original article (see record 2024-19816-001). Mattos et al. critiqued our assessments of the certainty of evidence as being overly permissive and not adhering to the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group's guidelines. GRADE has become an international standard to describe the level of confidence that investigators have in estimates of effects. Like the risk of bias evaluations, determining the certainty of evidence involves subjective judgment. The true value of GRADE is not in yielding a definitive evidence certainty rating but in its emphasis on transparency. While we acknowledge and respect the differing viewpoints of Mattos et al. regarding our ratings, we caution against the rigid and formulaic use of the GRADE methodology. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Borderline personality disorder (BPD) is the most common personality disorder, affecting 1.8% of the general population, 10% of psychiatric outpatients, and 15%-25% of psychiatric inpatients. Practice guidelines recommend psychotherapies as first-line treatments. However, psychotherapies commonly used for the treatment of BPD are numerous, and little is known about the comparative effectiveness of each individual psychotherapy versus treatment as usual (TAU) or other psychotherapies. To systematically assess the comparative effectiveness of commonly used psychotherapies versus TAU or versus other psychotherapies for BPD treatment. We conducted systematic literature searches in MEDLINE, EMBASE, the Cochrane Library, and APA PsycINFO up to July 14, 2022, and searched reference lists of pertinent articles and reviews. Inclusion criteria were (a) patients 13 years or older with a diagnosis of BPD, (b) treatment with commonly used psychotherapies, (c) comparison with TAU or another psychotherapy, (d) assessment of relevant BPD-related health outcomes, and (e) randomized or nonrandomized trials or controlled observational studies. Two investigators independently screened abstracts and full-text articles and graded the certainty of evidence using the Grading of Recommendations Assessment, Development, and Evaluation approach. We found 25 psychotherapy studies meeting inclusion criteria with data on 2,545 participants. Seventeen studies compared nine psychotherapies with TAU and nine studies compared eight psychotherapies with another psychotherapy for the treatment of BPD. Overall, both TAU and included psychotherapies were effective in treating the severity and symptoms of BPD. Moderate certainty of evidence suggests that systems training for emotional predictability and problem solving is more effective than TAU for the treatment of BPD; low certainty of evidence suggests that dialectical behavior therapy, schema therapy, transference-focused psychotherapy, acceptance and commitment therapy, manual-assisted cognitive therapy, and cognitive behavioral therapy are more effective than TAU for treating BPD. We were unable to draw conclusions from head-to-head comparisons of psychotherapies, which were limited to single studies with very low to low certainty of evidence. All commonly used psychotherapies improve BPD severity, symptoms, and functioning. Our assessment found no strong evidence suggesting that any one psychotherapy is more beneficial than another. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The authors summarize the evidence on the use of video teleconferencing when used as a replacement or as an adjunct for usual care. This article also provides an evidence map of the clinical contex... Background: Video teleconferencing (VTC) as a substitute for in-person health care or as an adjunct to usual care has increased in recent years. Purpose: To assess the benefits and harms of VTC vis...