Daniel Ben Joya
Daniel Ben Joya has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Dafna Eshel
Kahun Medical Ltd, Givatayim, Israel
3 shared publications - Eden Avnat
Faculty of Medicine, Tel Aviv University, Tel Aviv 69978, Israel
3 shared publications - Elia Yanko
The Azrieli Faculty of Medicine, Bar-Ilan University, Safed, Israel
3 shared publications - Shahar Ovadia
Kahun Medical Ltd, Givatayim, Israel
3 shared publications - Daniel Souroujon
Metaflow Ltd, HaArba'a St 30, Tel-Aviv, IL
2 shared publications - Eyal Schneider
Department of Otolaryngology-Head and Neck Surgery, Samson Assuta Ashdod University Hospital, Ashdod, Israel
2 shared publications
Latest Publications
- Open Access
- 14 July 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e64452
Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers
Background: Clinical problem-solving requires processing of semantic medical knowledge, such as illness scripts, and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate nonlanguage evidence-based answers to clinical questions is inherently limited by tokenization. Objective: This study aimed to evaluate LLMs’ performance on two question types: numeric (correlating findings) and semantic (differentiating entities), while examining differences within and between LLMs in medical aspects and comparing their performance to humans. Methods: To generate straightforward multichoice questions and answers (Q and As) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (containing data from more than 50,000 peer-reviewed studies) and created the EBM questions and answers (EBMQAs). EBMQA comprises 105,222 Q and As, categorized by medical topics (eg, medical disciplines) and nonmedical topics (eg, question length), and classified into numerical or semantic types. We benchmarked a dataset of 24,000 Q and As on two state-of-the-art LLMs, GPT-4 (OpenAI) and Claude 3 Opus (Anthropic). We evaluated the LLM’s accuracy on semantic and numerical question types and according to sublabeled topics. In addition, we examined the question-answering rate of LLMs by enabling them to choose to abstain from responding to questions. For validation, we compared the results for 100 unrelated numerical EBMQA questions between six human medical experts and the two language models. Results: In an analysis of 24,542 Q and As, Claude 3 and GPT-4 performed better on semantic Q and As (68.7%, n=1593 and 68.4%, n=1709), respectively. Then on numerical Q and As (61.3%, n=8583 and 56.7%, n=12,038), respectively, with Claude 3 outperforming GPT-4 in numeric accuracy (P<.001). A median accuracy gap of 7% (IQR 5%‐10%) was observed between the best and worst sublabels per topic, with different LLMs excelling in different sublabels. Focusing on Medical Discipline sublabels, Claude 3 performed well in neoplastic disorders but struggled with genitourinary disorders (69%, n=676 vs 58%, n=464; P<.0001), while GPT-4 excelled in cardiovascular disorders but struggled with neoplastic disorders (60%, n=1076 vs 53%, n=704; P=.0002). Furthermore, humans (82.3%, n=82.3) surpassed both Claude 3 (64.3%, n=64.3; P<.001) and GPT-4 (55.8%, n=55.8; P<.001) in the validation test. Spearman correlation between question-answering and accuracy rate in both Claude 3 and GPT-4 was insignificant (ρ=0.12, P=.69; ρ=0.43, P=.13). Conclusions: Both LLMs excelled more in semantic than numerical Q and As, with Claude 3 surpassing GPT-4 in numerical Q and As. However, both LLMs showed inter- and intramodel gaps in different medical aspects and remained inferior to humans. In addition, their ability to respond or abstain from answering a question does not reliably predict how accurately they perform when they do attempt to answer questions. Thus, their medical advice should be addressed carefully.
- 14 July 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e64452
Open AccessPerformance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers
Background: Clinical problem-solving requires processing of semantic medical knowledge, such as illness scripts, and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate nonlanguage evidence-based answers to clinical questions is inherently limited by tokenization. Objective: This study aimed to evaluate LLMs’ performance on two question types: numeric (correlating findings) and semantic (differentiating entities), while examining differences within and between LLMs in medical aspects and comparing their performance to humans. Methods: To generate straightforward multichoice questions and answers (Q and As) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (containing data from more than 50,000 peer-reviewed studies) and created the EBM questions and answers (EBMQAs). EBMQA comprises 105,222 Q and As, categorized by medical topics (eg, medical disciplines) and nonmedical topics (eg, question length), and classified into numerical or semantic types. We benchmarked a dataset of 24,000 Q and As on two state-of-the-art LLMs, GPT-4 (OpenAI) and Claude 3 Opus (Anthropic). We evaluated the LLM’s accuracy on semantic and numerical question types and according to sublabeled topics. In addition, we examined the question-answering rate of LLMs by enabling them to choose to abstain from responding to questions. For validation, we compared the results for 100 unrelated numerical EBMQA questions between six human medical experts and the two language models. Results: In an analysis of 24,542 Q and As, Claude 3 and GPT-4 performed better on semantic Q and As (68.7%, n=1593 and 68.4%, n=1709), respectively. Then on numerical Q and As (61.3%, n=8583 and 56.7%, n=12,038), respectively, with Claude 3 outperforming GPT-4 in numeric accuracy (P<.001). A median accuracy gap of 7% (IQR 5%‐10%) was observed between the best and worst sublabels per topic, with different LLMs excelling in different sublabels. Focusing on Medical Discipline sublabels, Claude 3 performed well in neoplastic disorders but struggled with genitourinary disorders (69%, n=676 vs 58%, n=464; P<.0001), while GPT-4 excelled in cardiovascular disorders but struggled with neoplastic disorders (60%, n=1076 vs 53%, n=704; P=.0002). Furthermore, humans (82.3%, n=82.3) surpassed both Claude 3 (64.3%, n=64.3; P<.001) and GPT-4 (55.8%, n=55.8; P<.001) in the validation test. Spearman correlation between question-answering and accuracy rate in both Claude 3 and GPT-4 was insignificant (ρ=0.12, P=.69; ρ=0.43, P=.13). Conclusions: Both LLMs excelled more in semantic than numerical Q and As, with Claude 3 surpassing GPT-4 in numerical Q and As. However, both LLMs showed inter- and intramodel gaps in different medical aspects and remained inferior to humans. In addition, their ability to respond or abstain from answering a question does not reliably predict how accurately they perform when they do attempt to answer questions. Thus, their medical advice should be addressed carefully.
- Open Access
- 17 June 2023
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e49570
The potential of evidence based clinical intake tools to discover or ground prevalence of symptoms using real-life virtual health encounters: a retrospective cohort study (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help healthcare providers make more informed decisions. The growing demand for personalized medicine, along with the big data revolution, have rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate and timely diagnosis, while contributing to the grounding of medical care. Objective: This work examines whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground real prevalence of symptoms in different disorders thereby expanding medical knowledge and further support medical diagnoses made by physicians. Methods: Between August 1, 2022, and January 15, 2023, patients who used the services of a virtual healthcare (VH) provider in the USA were first assessed by the Kahun EBCIT. Kahun platform gathered, documented, and analyzed the information from the sessions and its clinical findings. In this study, we estimated the prevalence of patients' symptoms in medical disorders, using two datasets. The first set analyzed symptoms prevalence, as determined by the Kahun's knowledge engine. The second set analyzed symptoms prevalence, relying solely on data from the VH patients gathered by Kahun. The difference in variance between these two prevalence datasets, helped us assess Kahun's ability to incorporate new data, while integrating existing knowledge. To analyze the comprehensiveness of the Kahun's knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NAMCS). To assess Kahun's diagnosis accuracy, physicians independently diagnosed 250 of Kahun-VH’s sessions. Their diagnoses were compared with Kahun's diagnoses. Results: As part of this work, 2,550 patients used Kahun to complete a full assessment, among them 1,714 females and 836 males. Kahun collected 314 different chief complaints and proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6,496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in NAMCS 2019. In 90% (224/250) of the sessions, at least one identical disorder suggested by both the physicians and Kahun, with total accuracy rate of 72% (367/507). Kahun’s engine yielded 519 prevalences while the Kahun-VH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both databases. Conclusions: ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnosis. Using this credible database, potential prevalence of symptoms in different disorders were discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.
- 17 June 2023
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e49570
Open AccessThe potential of evidence based clinical intake tools to discover or ground prevalence of symptoms using real-life virtual health encounters: a retrospective cohort study (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help healthcare providers make more informed decisions. The growing demand for personalized medicine, along with the big data revolution, have rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate and timely diagnosis, while contributing to the grounding of medical care. Objective: This work examines whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground real prevalence of symptoms in different disorders thereby expanding medical knowledge and further support medical diagnoses made by physicians. Methods: Between August 1, 2022, and January 15, 2023, patients who used the services of a virtual healthcare (VH) provider in the USA were first assessed by the Kahun EBCIT. Kahun platform gathered, documented, and analyzed the information from the sessions and its clinical findings. In this study, we estimated the prevalence of patients' symptoms in medical disorders, using two datasets. The first set analyzed symptoms prevalence, as determined by the Kahun's knowledge engine. The second set analyzed symptoms prevalence, relying solely on data from the VH patients gathered by Kahun. The difference in variance between these two prevalence datasets, helped us assess Kahun's ability to incorporate new data, while integrating existing knowledge. To analyze the comprehensiveness of the Kahun's knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NAMCS). To assess Kahun's diagnosis accuracy, physicians independently diagnosed 250 of Kahun-VH’s sessions. Their diagnoses were compared with Kahun's diagnoses. Results: As part of this work, 2,550 patients used Kahun to complete a full assessment, among them 1,714 females and 836 males. Kahun collected 314 different chief complaints and proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6,496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in NAMCS 2019. In 90% (224/250) of the sessions, at least one identical disorder suggested by both the physicians and Kahun, with total accuracy rate of 72% (367/507). Kahun’s engine yielded 519 prevalences while the Kahun-VH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both databases. Conclusions: ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnosis. Using this credible database, potential prevalence of symptoms in different disorders were discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.
- Open Access
- 17 June 2023
- Published by JMIR Publications Inc. in JMIR Preprints
The Potential of Evidence-Based Clinical Intake Tools to Discover or Ground Prevalence of Symptoms Using Real-Life Digital Health Encounters: Retrospective Cohort Study (Preprint)
BACKGROUND Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help health care providers make informed decisions. The growing demand for personalized medicine, along with the big data revolution, has rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate diagnosis, while contributing to the grounding of medical care. OBJECTIVE This work aims to examine whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, and thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground the real prevalence of symptoms in different disorders thereby expanding medical knowledge and further supporting medical diagnoses made by physicians. METHODS Between August 1, 2022, and January 15, 2023, patients who used the services of a digital health care (DH) provider in the United States were first assessed by the Kahun EBCIT. Kahun platform gathered and analyzed the information from the sessions. This study estimated the prevalence of patients’ symptoms in medical disorders using 2 data sets. The first data set analyzed symptom prevalence, as determined by Kahun’s knowledge engine. The second data set analyzed symptom prevalence, relying solely on data from the DH patients gathered by Kahun. The variance difference between these 2 prevalence data sets helped us assess Kahun’s ability to incorporate new data while integrating existing knowledge. To analyze the comprehensiveness of Kahun’s knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NMCAS). To assess Kahun’s diagnosis accuracy, physicians independently diagnosed 250 of Kahun-DH’s sessions. Their diagnoses were compared with Kahun’s diagnoses. RESULTS In this study, 2550 patients used Kahun to complete a full assessment. Kahun proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in the 2019 NMCAS. In 90% (224/250) of the sessions, both physicians and Kahun suggested at least one identical disorder, with a 72% (367/507) total accuracy rate. Kahun’s engine yielded 519 prevalences while the Kahun-DH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both data sets. CONCLUSIONS ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnoses. Using this credible database, the potential prevalence of symptoms in different disorders was discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.
- 17 June 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessThe Potential of Evidence-Based Clinical Intake Tools to Discover or Ground Prevalence of Symptoms Using Real-Life Digital Health Encounters: Retrospective Cohort Study (Preprint)
BACKGROUND Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help health care providers make informed decisions. The growing demand for personalized medicine, along with the big data revolution, has rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate diagnosis, while contributing to the grounding of medical care. OBJECTIVE This work aims to examine whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, and thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground the real prevalence of symptoms in different disorders thereby expanding medical knowledge and further supporting medical diagnoses made by physicians. METHODS Between August 1, 2022, and January 15, 2023, patients who used the services of a digital health care (DH) provider in the United States were first assessed by the Kahun EBCIT. Kahun platform gathered and analyzed the information from the sessions. This study estimated the prevalence of patients’ symptoms in medical disorders using 2 data sets. The first data set analyzed symptom prevalence, as determined by Kahun’s knowledge engine. The second data set analyzed symptom prevalence, relying solely on data from the DH patients gathered by Kahun. The variance difference between these 2 prevalence data sets helped us assess Kahun’s ability to incorporate new data while integrating existing knowledge. To analyze the comprehensiveness of Kahun’s knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NMCAS). To assess Kahun’s diagnosis accuracy, physicians independently diagnosed 250 of Kahun-DH’s sessions. Their diagnoses were compared with Kahun’s diagnoses. RESULTS In this study, 2550 patients used Kahun to complete a full assessment. Kahun proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in the 2019 NMCAS. In 90% (224/250) of the sessions, both physicians and Kahun suggested at least one identical disorder, with a 72% (367/507) total accuracy rate. Kahun’s engine yielded 519 prevalences while the Kahun-DH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both data sets. CONCLUSIONS ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnoses. Using this credible database, the potential prevalence of symptoms in different disorders was discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.