Ewelina Türk

Background: Rare diseases, which affect millions of people worldwide, pose a major challenge, as it often takes years before an accurate diagnosis can be made. This delay results in substantial burdens for patients and health care systems, as misdiagnoses lead to inadequate treatment and increased costs. Artificial intelligence (AI)–powered symptom checkers (SCs) present an opportunity to flag rare diseases earlier in the diagnostic work-up. However, these tools are primarily based on published literature, which often contains incomplete data on rare diseases, resulting in compromised diagnostic accuracy. Integrating expert interview insights into SC models may enhance their performance, ensuring that rare diseases are considered sooner and diagnosed more accurately. Objective: The objectives of our study were to incorporate expert interview vignettes into AI-powered SCs, in addition to a traditional literature review, and to evaluate whether this novel approach improves diagnostic accuracy and user satisfaction for rare diseases, focusing on Fabry disease. Methods: This mixed methods prospective pilot study was conducted at Hannover Medical School, Germany. In the first phase, guided interviews were conducted with medical experts specialized in Fabry disease to create clinical vignettes that enriched the AI SC’s Fabry disease model. In the second phase, adult patients with a confirmed diagnosis of Fabry disease used both the original and optimized SC versions in a randomized order. The versions, containing either the original or the optimized Fabry disease model, were evaluated based on diagnostic accuracy and user satisfaction, which were assessed through questionnaires. Results: Three medical experts with extensive experience in lysosomal storage disorder Fabry disease contributed to the creation of 5 clinical vignettes, which were integrated into the AI-powered SC. The study compared the original and optimized SC versions in 6 patients with Fabry disease. The optimized version improved diagnostic accuracy, with Fabry disease identified as the top suggestion in 33% (2/6) of cases, compared to 17% (1/6) with the original model. Additionally, overall user satisfaction was higher for the optimized version, with participants rating it more favorably in terms of symptom coverage and completeness. Conclusions: This study demonstrates that integrating expert-derived clinical vignettes into AI-powered SCs can improve diagnostic accuracy and user satisfaction, particularly for rare diseases. The optimized SC version, which incorporated these vignettes, showed improved performance in identifying Fabry disease as a top diagnostic suggestion and received higher user satisfaction ratings compared to the original version. To fully realize the potential of this approach, it is crucial to include vignettes representing atypical presentations and to conduct larger-scale studies to validate these findings.

UNSTRUCTURED Background Africa has the lowest density of healthcare workers globally and digital tools using artificial intelligence (AI) could bridge that gap affordably. A partnership between Ada Health and Praekelt.org sought to integrate Ada for users in South Africa (SA). Challenges and solutions Three challenges were identified to improve the efficacy of health AI in the local setting: Localization: disease incidences and presentations of maternal and child health in SA differ from those in Europe and North America. We adapted Ada’s knowledge base to meet these challenges. For this project, 25 maternal and 25 pediatric health conditions were localized, and one new pediatric condition was created for users of the Ada app. This included adding region-specific incidences and adapting the knowledge base for variations in disease presentation. Readability: the existing readability score of Ada’s consumer-facing medical content was calculated using the Automated Readability Index as grade 11.0 (± 1.8, range = 5.8-17.5). Using Content Design London’s Readability Guidelines, the readability score of Ada’s content was lowered to below grade 8 (7.4 ± 0.8, range = 4.6-10.2) while maintaining medical accuracy. Different approaches to AI: most medical research is conducted in high-income countries and among people with high literacy levels, thereby leading to bias in the system. Using a white-box approach allows intelligent solutions to be delivered that incorporate the needs of the population, including in low and middle income countries. Conclusion By partnering with grassroots and local organizations, such as Praekelt.org, AI companies can reduce the burden on vulnerable healthcare systems, but should be designed and adapted to the needs of the local population to account for regional differences in incidences and disease presentations, and language barriers. Furthermore, wherever possible, efforts should be made to reduce bias in AI by using white-box systems. It is possible to use simple language for consumer-facing medical text without compromising on medical quality. With a few minor adaptations AI-technology can be localized to serve global public health needs.

Background: Low- and middle-income countries face difficulties in providing adequate health care. One of the reasons is a shortage of qualified health workers. Diagnostic decision support systems are designed to aid clinicians in their work and have the potential to mitigate pressure on health care systems. Objective: The Artificial Intelligence–Based Assessment of Health Symptoms in Tanzania (AFYA) study will evaluate the potential of an English-language artificial intelligence–based prototype diagnostic decision support system for mid-level health care practitioners in a low- or middle-income setting. Methods: This is an observational, prospective clinical study conducted in a busy Tanzanian district hospital. In addition to usual care visits, study participants will consult a mid-level health care practitioner, who will use a prototype diagnostic decision support system, and a study physician. The accuracy and comprehensiveness of the differential diagnosis provided by the diagnostic decision support system will be evaluated against a gold-standard differential diagnosis provided by an expert panel. Results: Patient recruitment started in October 2021. Participants were recruited directly in the waiting room of the outpatient clinic at the hospital. Data collection will conclude in May 2022. Data analysis is planned to be finished by the end of June 2022. The results will be published in a peer-reviewed journal. Conclusions: Most diagnostic decision support systems have been developed and evaluated in high-income countries, but there is great potential for these systems to improve the delivery of health care in low- and middle-income countries. The findings of this real-patient study will provide insights based on the performance and usability of a prototype diagnostic decision support system in low- or middle-income countries. Trial Registration: ClinicalTrials.gov NCT04958577; http://clinicaltrials.gov/ct2/show/NCT04958577 International Registered Report Identifier (IRRID): DERR1-10.2196/34298

Background: Continuously growing medical knowledge and the increasing amount of data make it difficult for medical professionals to keep track of all new information and to place it in the context of existing information. A variety of digital technologies and artificial intelligence–based methods are currently available as persuasive tools to empower physicians in clinical decision-making and improve health care quality. A novel diagnostic decision support system (DDSS) prototype developed by Ada Health GmbH with a focus on traceability, transparency, and usability will be examined more closely in this study. Objective: The aim of this study is to test the feasibility and functionality of a novel DDSS prototype, exploring its potential and performance in identifying the underlying cause of acute dyspnea in patients at the University Hospital Basel. Methods: A prospective, observational feasibility study was conducted at the emergency department (ED) and internal medicine ward of the University Hospital Basel, Switzerland. A convenience sample of 20 adult patients admitted to the ED with dyspnea as the chief complaint and a high probability of inpatient admission was selected. A study physician followed the patients admitted to the ED throughout the hospitalization without interfering with the routine clinical work. Routinely collected health-related personal data from these patients were entered into the DDSS prototype. The DDSS prototype’s resulting disease probability list was compared with the gold-standard main diagnosis provided by the treating physician. Results: The DDSS presented information with high clarity and had a user-friendly, novel, and transparent interface. The DDSS prototype was not perfectly suited for the ED as case entry was time-consuming (1.5-2 hours per case). It provided accurate decision support in the clinical inpatient setting (average of cases in which the correct diagnosis was the first diagnosis listed: 6/20, 30%, SD 2.10%; average of cases in which the correct diagnosis was listed as one of the top 3: 11/20, 55%, SD 2.39%; average of cases in which the correct diagnosis was listed as one of the top 5: 14/20, 70%, SD 2.26%) in patients with dyspnea as the main presenting complaint. Conclusions: The study of the feasibility and functionality of the tool was successful, with some limitations. Used in the right place, the DDSS has the potential to support physicians in their decision-making process by showing new pathways and unintentionally ignored diagnoses. The DDSS prototype had some limitations regarding the process of data input, diagnostic accuracy, and completeness of the integrated medical knowledge. The results of this study provide a basis for the tool’s further development. In addition, future studies should be conducted with the aim to overcome the current limitations of the tool and study design. Trial Registration: ClinicalTrials.gov NCT04827342; https://clinicaltrials.gov/ct2/show/NCT04827342

BACKGROUND Low- and middle-income countries face difficulties in providing adequate health care. One of the reasons is a shortage of qualified health workers. Diagnostic decision support systems are designed to aid clinicians in their work and have the potential to mitigate pressure on health care systems. OBJECTIVE The Artificial Intelligence–Based Assessment of Health Symptoms in Tanzania (AFYA) study will evaluate the potential of an English-language artificial intelligence–based prototype diagnostic decision support system for mid-level health care practitioners in a low- or middle-income setting. METHODS This is an observational, prospective clinical study conducted in a busy Tanzanian district hospital. In addition to usual care visits, study participants will consult a mid-level health care practitioner, who will use a prototype diagnostic decision support system, and a study physician. The accuracy and comprehensiveness of the differential diagnosis provided by the diagnostic decision support system will be evaluated against a gold-standard differential diagnosis provided by an expert panel. RESULTS Patient recruitment started in October 2021. Participants were recruited directly in the waiting room of the outpatient clinic at the hospital. Data collection will conclude in May 2022. Data analysis is planned to be finished by the end of June 2022. The results will be published in a peer-reviewed journal. CONCLUSIONS Most diagnostic decision support systems have been developed and evaluated in high-income countries, but there is great potential for these systems to improve the delivery of health care in low- and middle-income countries. The findings of this real-patient study will provide insights based on the performance and usability of a prototype diagnostic decision support system in low- or middle-income countries. CLINICALTRIAL ClinicalTrials.gov NCT04958577; http://clinicaltrials.gov/ct2/show/NCT04958577 INTERNATIONAL REGISTERED REPORT DERR1-10.2196/34298

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). N/A

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). We have several comments on the recent publication of [1], in which repeated testing of four symptom assessment applications with clinical vignettes was carried out to look for “hints of ‘non-locked learning algorithms’”. As the developer of one of the symptom assessment applications studied by [1], we are supportive of studies evaluating app performance, however there are important limitations in the methodology of the study. Most importantly, the methodology used in this study is not capable of addressing its main objective. The approach used to look for evidence of non-locked algorithms was the quantification of differences in performance using three ophthalmology vignettes, first in 2018 then in 2020. This methodology, although highly limited due to the use of only three vignettes in one medical specialism, could be used to detect changes in app performance over time. It however cannot be used to distinguish between non-locked algorithms and the manual updating of the apps’ medical intelligence, through the normal process of manual release of updated app versions.

Objectives: To compare breadth of condition coverage, accuracy of suggested conditions and appropriateness of urgency advice of 8 popular symptom assessment apps with each other and with 7 General Practitioners. Design: Clinical vignettes study. Setting: 200 clinical vignettes representing real-world scenarios in primary care. Intervention/comparator: Condition coverage, suggested condition accuracy, and urgency advice performance was measured against the vignettes’ gold-standard diagnoses and triage level. Primary outcome measures: Outcomes included (i) proportion of conditions “covered” by an app, i.e. not excluded because the patient was too young/old, pregnant, or comorbid, (ii) proportion of vignettes in which the correct primary diagnosis was amongst the top 3 conditions suggested, and, (iii) proportion of “safe” urgency level advice (i.e. at gold standard level, more conservative, or no more than one level less conservative). Results: Condition-suggestion coverage was highly variable, with some apps not offering a suggestion for many users: in alphabetical order, Ada: 99.0%; Babylon: 51.5%; Buoy: 88.5%; K Health: 74.5%; Mediktor: 80.5%; Symptomate: 61.5%; Your.MD: 64.5%. The top-3 suggestion accuracy (M3) of GPs was on average 82.i±5.2%. For the apps it was - Ada: 70.5%; Babylon: 32.0%; Buoy: 43.0%; K Health: 36.0%; Mediktor: 36.0%; Symptomate: 27.5%; WebMD: 35.5%; Your.MD: 23.5%. Some apps exclude certain user groups (e.g. younger users) or certain conditions - for these apps condition-suggestion performance is generally greater with exclusion of these vignettes. For safe urgency advice, tested GPs had an average of 97.0±2.5%. For the vignettes with advice provided, only three apps had safety performance within 1 S.D. of the GPs (mean) - Ada: 97.0%; Babylon: 95.i%; Symptomate: 97.8%. One app had a safety performance within 2 S.D.s of GPs - Your.MD: 92.6%. Three apps had a safety performance outside 2 S.D.s of GPs - Buoy: 80.0% ( p p p =1.3×10 ^-3 ). Conclusions: The utility of digital symptom assessment apps relies upon coverage, accuracy, and safety. While no digital tool outperformed GPs, some came close, and the nature of iterative improvements to software offers scalable improvements to care.

Rare disease diagnosis is often delayed by years. A primary factor for this delay is a lack of knowledge and awareness regarding rare diseases. Probabilistic diagnostic decision support systems (DDSSs) have the potential to accelerate rare disease diagnosis by suggesting differential diagnoses for physicians based on case input and incorporated medical knowledge. We examine the DDSS prototype Ada DX and assess its potential to provide accurate rare disease suggestions early in the course of rare disease cases. Ada DX suggested the correct disease earlier than the time of clinical diagnosis among the top five fit disease suggestions in 53.8% of cases (50 of 93), and as the top fit disease suggestion in 37.6% of cases (35 of 93). The median advantage of correct disease suggestions compared to the time of clinical diagnosis was 3 months or 50% for top five fit and 1 month or 21% for top fit. The correct diagnosis was suggested at the first documented patient visit in 33.3% (top 5 fit), and 16.1% of cases (top fit), respectively. Wilcoxon signed-rank test shows a significant difference between the time to clinical diagnosis and the time to correct disease suggestion for both top five fit and top fit (z-score -6.68, respective -5.71, α=0.05, p-value <0.001). Ada DX provided accurate rare disease suggestions in most rare disease cases. In many cases, Ada DX provided correct rare disease suggestions early in the course of the disease, sometimes at the very beginning of a patient journey. The interpretation of these results indicates that Ada DX has the potential to suggest rare diseases to physicians early in the course of a case. Limitations of this study derive from its retrospective and unblinded design, data input by a single user, and the optimization of the knowledge base during the course of the study. Results pertaining to the system’s accuracy should be interpreted cautiously. Whether the use of Ada DX reduces the time to diagnosis in rare diseases in a clinical setting should be validated in prospective studies.