Hossein Motahari-Nezhad
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Interests
Hossein Motahari-Nezhad has a total of 33 co-authors. Here are the top co-authors by number of shared publications:
- László Gulácsi
Health Economics Research Center, University Research and Innovation Center, Óbuda University, Budapest, Hungary
11 shared publications - Márta Péntek
Health Economics Research Center, University Research and Innovation Center, Óbuda University, Budapest, Hungary
11 shared publications - Zsombor Zrubka
Health Economics Research Center, University Research and Innovation Center, Óbuda University, Budapest, Hungary
11 shared publications - Levente Kovács
Óbuda University
5 shared publications Aslan Sadeghdaghighi
Semmelweis Univ, Dept Med, Budapest, Hungary
4 shared publications- Meriem Fgaier
Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University Budapest Hungary
4 shared publications
Latest Publications
- Open Access
- 4 August 2025
- Published by Springer Nature in Research Square
SPECTRUM SRA: An R Shiny Application to Automate Systematic Reviews Using Artificial Intelligence and Large Language Models
Purpose: This study aims to develop an R Shiny application named SPECTRUM SRA (Systematic Review Automation) and to assess its feasibility. The application encompasses eight modules: Search, PRISMA flow diagram generation, dEduplication, sCreening, daTa extraction, Reporting, qUery, and suMmarization. It provides an end‑to‑end workflow to automate and semi‑automate multiple steps of systematic reviews, addressing growing time and labor constraints and accelerating evidence‑to‑practice translation. Design/methodology/approach: SPECTRUM SRA comprises eight interconnected modules and offers an interface for automated article search and retrieval from PubMed, Scopus, and Web of Science. Search functionality is implemented via API integration for PubMed and Scopus, with manual search and import for Web of Science results. Deduplication employs two strategies: DOI‑based matching and fuzzy string deduplication. The application also supports automated title/abstract and full‑text screening using large language model prompting. Automated reporting for both search and screening is incorporated through AI‑driven scripting. Additionally, SPECTRUM SRA automates PRISMA flow diagram generation. Summarization and data extraction of the final set of eligible studies are performed via LLM prompting. The feasibility of the application was assessed through a systematic review of randomized controlled trials comparing virtual reality interventions to non‑virtual‑reality approaches in dentistry. Findings: Keyword identification was completed in 12 seconds; search query generation took less than three seconds; and retrieval of 703 records from the databases required 4 minutes and 30 seconds. Deduplication removed 238 records in 3 seconds, leaving 465 records for screening. ChatGPT o4‑mini‑high and DeepSeek V3 R1 were selected as the first and second reviewers, respectively. Title/abstract screening took 84.5 minutes for Reviewer 1 and 131 minutes for Reviewer 2 (κ = 0.852). Full‑text screening of 43 articles required 43 minutes for Reviewer 1 and 90 minutes for Reviewer 2 (κ = 0.903). Automated summarization and data extraction of the 25 final eligible studies achieved 91% accuracy using ChatGPT in 25 minutes. Originality: To the best of our knowledge, SPECTRUM SRA is the first end‑to‑end Shiny application that supports multiple tasks in systematic reviews. It can reduce the time, labor, and energy required to conduct systematic reviews, thereby accelerating health policy and decision‑making.
- 4 August 2025
- Published by Springer Nature in Research Square
Open AccessSPECTRUM SRA: An R Shiny Application to Automate Systematic Reviews Using Artificial Intelligence and Large Language Models
Purpose: This study aims to develop an R Shiny application named SPECTRUM SRA (Systematic Review Automation) and to assess its feasibility. The application encompasses eight modules: Search, PRISMA flow diagram generation, dEduplication, sCreening, daTa extraction, Reporting, qUery, and suMmarization. It provides an end‑to‑end workflow to automate and semi‑automate multiple steps of systematic reviews, addressing growing time and labor constraints and accelerating evidence‑to‑practice translation. Design/methodology/approach: SPECTRUM SRA comprises eight interconnected modules and offers an interface for automated article search and retrieval from PubMed, Scopus, and Web of Science. Search functionality is implemented via API integration for PubMed and Scopus, with manual search and import for Web of Science results. Deduplication employs two strategies: DOI‑based matching and fuzzy string deduplication. The application also supports automated title/abstract and full‑text screening using large language model prompting. Automated reporting for both search and screening is incorporated through AI‑driven scripting. Additionally, SPECTRUM SRA automates PRISMA flow diagram generation. Summarization and data extraction of the final set of eligible studies are performed via LLM prompting. The feasibility of the application was assessed through a systematic review of randomized controlled trials comparing virtual reality interventions to non‑virtual‑reality approaches in dentistry. Findings: Keyword identification was completed in 12 seconds; search query generation took less than three seconds; and retrieval of 703 records from the databases required 4 minutes and 30 seconds. Deduplication removed 238 records in 3 seconds, leaving 465 records for screening. ChatGPT o4‑mini‑high and DeepSeek V3 R1 were selected as the first and second reviewers, respectively. Title/abstract screening took 84.5 minutes for Reviewer 1 and 131 minutes for Reviewer 2 (κ = 0.852). Full‑text screening of 43 articles required 43 minutes for Reviewer 1 and 90 minutes for Reviewer 2 (κ = 0.903). Automated summarization and data extraction of the 25 final eligible studies achieved 91% accuracy using ChatGPT in 25 minutes. Originality: To the best of our knowledge, SPECTRUM SRA is the first end‑to‑end Shiny application that supports multiple tasks in systematic reviews. It can reduce the time, labor, and energy required to conduct systematic reviews, thereby accelerating health policy and decision‑making.
- Open Access
- 1 January 2025
- Published by National Academy of Sciences of Tajikistan in Journal of Health Administration
- Vol. 27 (4) , 51-66
Readability evaluation of online health information on COVID-19: a survey of Persian websites
Introduction: The presence of readability challenges leads to difficulties in comprehending health-related online materials and has negative consequences, notably reducing the likelihood of disease prevention and healthcare utilization. The primary objective of this study is to assess the readability levels of Persian-language websites that provides information about COVID-19. Methods: Utilizing ...
- 1 January 2025
- Published by National Academy of Sciences of Tajikistan in Journal of Health Administration
- Vol. 27 (4) , 51-66
Open AccessReadability evaluation of online health information on COVID-19: a survey of Persian websites
Introduction: The presence of readability challenges leads to difficulties in comprehending health-related online materials and has negative consequences, notably reducing the likelihood of disease prevention and healthcare utilization. The primary objective of this study is to assess the readability levels of Persian-language websites that provides information about COVID-19. Methods: Utilizing ...
- 11 November 2024
The effect of virtual reality for anxiety and pain in dentistry: A systematic review and meta-analysis
Objectives: This study aimed to assess the impact of virtual reality (VR) on reducing anxiety and pain in dental patients across all age groups and dental procedures. Methods: Systematic review with comprehensive search of PubMed and Cochrane Library databases for randomized controlled trials (RCTs) comparing VR interventions with non-VR methods in dental settings up to April 2024. The selection followed the PRISMA-P guidelines. Inclusion criteria were based on the PICOS framework, focusing on studies involving dental patients of any age, utilizing VR during dental treatments, and reporting outcomes on anxiety and pain. Data extraction and quality appraisal were performed independently by two reviewers using the ROB-2 tool and GRADE methodology. Meta-analyses used a random-effects model. Results: Twenty-seven studies met the inclusion criteria, encompassing several dental treatments. In meta-analysis VR reduced anxiety in children (SMD -1.44, 95% CI -2.24 to -0.63) but not adults (SMD -0.35, 95% CI -1.11 to 0.4). For pain reduction, VR was effective in both children (SMD -1.11, 95% CI -1.65 to -0.57) and adults (SMD -0.59, 95% CI -1.187 to -0.001). Heterogeneity was high across studies, and evidence quality ranged from low to moderate. Conclusions: VR is a promising intervention for reducing anxiety and pain in children during dental procedures. Its effectiveness in adults is limited to pain reduction. High heterogeneity and risk of bias suggest that findings should be interpreted with caution. Further research is needed to standardize VR content and explore its varying impacts across different age groups and dental procedures.
- 11 November 2024
The effect of virtual reality for anxiety and pain in dentistry: A systematic review and meta-analysis
Objectives: This study aimed to assess the impact of virtual reality (VR) on reducing anxiety and pain in dental patients across all age groups and dental procedures. Methods: Systematic review with comprehensive search of PubMed and Cochrane Library databases for randomized controlled trials (RCTs) comparing VR interventions with non-VR methods in dental settings up to April 2024. The selection followed the PRISMA-P guidelines. Inclusion criteria were based on the PICOS framework, focusing on studies involving dental patients of any age, utilizing VR during dental treatments, and reporting outcomes on anxiety and pain. Data extraction and quality appraisal were performed independently by two reviewers using the ROB-2 tool and GRADE methodology. Meta-analyses used a random-effects model. Results: Twenty-seven studies met the inclusion criteria, encompassing several dental treatments. In meta-analysis VR reduced anxiety in children (SMD -1.44, 95% CI -2.24 to -0.63) but not adults (SMD -0.35, 95% CI -1.11 to 0.4). For pain reduction, VR was effective in both children (SMD -1.11, 95% CI -1.65 to -0.57) and adults (SMD -0.59, 95% CI -1.187 to -0.001). Heterogeneity was high across studies, and evidence quality ranged from low to moderate. Conclusions: VR is a promising intervention for reducing anxiety and pain in children during dental procedures. Its effectiveness in adults is limited to pain reduction. High heterogeneity and risk of bias suggest that findings should be interpreted with caution. Further research is needed to standardize VR content and explore its varying impacts across different age groups and dental procedures.
- Open Access
- 22 March 2024
- Published by Springer Nature in Research Square
The effect of virtual reality in dentistry: A protocol for a systematic review and meta-analysis
This systematic review and meta-analysis protocol outlines an investigation into the effects of Virtual Reality (VR) in dentistry. As a growing technological tool, VR offers immersive environments for patient education, anxiety and pain management, and professional skill enhancement in dental practice. Despite its growing use, empirical evidence on VR's efficacy remains fragmented and inconclusive. This review aims to consolidate current research, providing a comprehensive assessment of VR's impact across all health outcomes in dental care. Adhering to the PRISMA-P guidelines, the study will include English-language, two-arm randomized controlled trials, comparing VR interventions with non-VR controls. The review will consider all dental patients, regardless of demographics, and will encompass various dental treatment settings. A search strategy employing two databases including PubMed and the Cochrane Library will identify relevant studies, followed by a systematic screening and data extraction process conducted by independent reviewers. The review will conduct a risk of bias assessment and employ a random effects meta-analysis for evidence synthesis. A subgroup analysis and meta-regression will explore factors affecting VR's effectiveness. Publication bias will be assessed using Trim and Fill, Begg and Egger methods, and the GRADE tool will evaluate the quality of evidence. The review anticipates offering insights into the role of VR in enhancing patient-centered care. By synthesizing scattered evidence, it aims to identify specific health outcomes significantly impacted by VR applications. The findings are expected to guide future research and clinical practices, enhancing patient care in dentistry.
- 22 March 2024
- Published by Springer Nature in Research Square
Open AccessThe effect of virtual reality in dentistry: A protocol for a systematic review and meta-analysis
This systematic review and meta-analysis protocol outlines an investigation into the effects of Virtual Reality (VR) in dentistry. As a growing technological tool, VR offers immersive environments for patient education, anxiety and pain management, and professional skill enhancement in dental practice. Despite its growing use, empirical evidence on VR's efficacy remains fragmented and inconclusive. This review aims to consolidate current research, providing a comprehensive assessment of VR's impact across all health outcomes in dental care. Adhering to the PRISMA-P guidelines, the study will include English-language, two-arm randomized controlled trials, comparing VR interventions with non-VR controls. The review will consider all dental patients, regardless of demographics, and will encompass various dental treatment settings. A search strategy employing two databases including PubMed and the Cochrane Library will identify relevant studies, followed by a systematic screening and data extraction process conducted by independent reviewers. The review will conduct a risk of bias assessment and employ a random effects meta-analysis for evidence synthesis. A subgroup analysis and meta-regression will explore factors affecting VR's effectiveness. Publication bias will be assessed using Trim and Fill, Begg and Egger methods, and the GRADE tool will evaluate the quality of evidence. The review anticipates offering insights into the role of VR in enhancing patient-centered care. By synthesizing scattered evidence, it aims to identify specific health outcomes significantly impacted by VR applications. The findings are expected to guide future research and clinical practices, enhancing patient care in dentistry.
- Open Access
- 22 March 2024
- Published by Springer Nature in Research Square
The effect of telehealth on pain management: Protocol for a systematic review and meta-analysis
This protocol outlines a systematic review and meta-analysis aimed at evaluating the impact of telehealth on pain management. The inception of telehealth as a revolutionary method in healthcare delivery, particularly in pain management, has become increasingly prominent, especially during the COVID-19 pandemic. The review will follow the PRISMA-P guidelines, incorporating randomized controlled trials (RCTs) that compare telehealth interventions with non-telehealth controls for pain management in diverse populations, settings, and clinical conditions. Only English-language studies, published from 2018 onwards, will be included. Eligibility criteria are based on the PICOS framework, focusing on two-arm RCTs. The search will encompass major databases like PubMed and the Cochrane Library, with subsequent screening and data extraction conducted by independent reviewers. Risk of bias will be assessed using the ROB 2 tool. A random-effects meta-analysis will synthesize evidence, adjusting for heterogeneity and potential biases. The GRADE tool will evaluate the quality of evidence. This study aims to comprehensively assess telehealth's effectiveness in pain management, addressing knowledge gaps and guiding future research, policy development, and clinical practice. Its findings are expected to inform healthcare professionals and policymakers, aiding in the integration of telehealth into pain management strategies and adapting to the changing landscape of healthcare delivery in a digital age.
- 22 March 2024
- Published by Springer Nature in Research Square
Open AccessThe effect of telehealth on pain management: Protocol for a systematic review and meta-analysis
This protocol outlines a systematic review and meta-analysis aimed at evaluating the impact of telehealth on pain management. The inception of telehealth as a revolutionary method in healthcare delivery, particularly in pain management, has become increasingly prominent, especially during the COVID-19 pandemic. The review will follow the PRISMA-P guidelines, incorporating randomized controlled trials (RCTs) that compare telehealth interventions with non-telehealth controls for pain management in diverse populations, settings, and clinical conditions. Only English-language studies, published from 2018 onwards, will be included. Eligibility criteria are based on the PICOS framework, focusing on two-arm RCTs. The search will encompass major databases like PubMed and the Cochrane Library, with subsequent screening and data extraction conducted by independent reviewers. Risk of bias will be assessed using the ROB 2 tool. A random-effects meta-analysis will synthesize evidence, adjusting for heterogeneity and potential biases. The GRADE tool will evaluate the quality of evidence. This study aims to comprehensively assess telehealth's effectiveness in pain management, addressing knowledge gaps and guiding future research, policy development, and clinical practice. Its findings are expected to inform healthcare professionals and policymakers, aiding in the integration of telehealth into pain management strategies and adapting to the changing landscape of healthcare delivery in a digital age.
- Open Access
- 22 March 2024
- Published by Springer Nature in Research Square
Outcomes of smart glasses: Protocol for a systematic review and meta-analysis
Smart glasses have revolutionized healthcare delivery with their hands-free functionality, voice command operation, and ability to project data onto lenses. This protocol outlines a systematic review and meta-analysis to assess the outcomes of smart glasses in healthcare. Following PRISMA-P guidelines, this review will include peer-reviewed English randomized controlled trials (RCTs) comparing smart glasses to non-smart glass interventions across diverse patient populations and settings. Exclusion criteria comprise non-RCTs, non-peer-reviewed resources, and studies where smart glasses are not the primary intervention. The search, conducted in PubMed, Scopus, and the Web of Science, without time limitations, will be systematically managed and documented, including the selection process, represented through a PRISMA flow diagram. Data extraction will cover key study characteristics and outcomes. Risk of bias will be assessed using the ROB 2 tool. Studies with comparable outcomes will be pooled for meta-analysis, employing the DerSimonian and Laird method in R software, with heterogeneity assessed via the I-squared statistic. A meta-regression analysis will explore age-related heterogeneity, and subgroup analysis will investigate bias risk levels. Publication bias will be assessed using the Trim and Fill technique and Egger and Begg tests. The outcomes' quality will be evaluated using the GRADE framework. The findings are expected to guide future research, influence health policy, and inform healthcare professionals and policymakers about integrating smart glasses into healthcare systems.
- 22 March 2024
- Published by Springer Nature in Research Square
Open AccessOutcomes of smart glasses: Protocol for a systematic review and meta-analysis
Smart glasses have revolutionized healthcare delivery with their hands-free functionality, voice command operation, and ability to project data onto lenses. This protocol outlines a systematic review and meta-analysis to assess the outcomes of smart glasses in healthcare. Following PRISMA-P guidelines, this review will include peer-reviewed English randomized controlled trials (RCTs) comparing smart glasses to non-smart glass interventions across diverse patient populations and settings. Exclusion criteria comprise non-RCTs, non-peer-reviewed resources, and studies where smart glasses are not the primary intervention. The search, conducted in PubMed, Scopus, and the Web of Science, without time limitations, will be systematically managed and documented, including the selection process, represented through a PRISMA flow diagram. Data extraction will cover key study characteristics and outcomes. Risk of bias will be assessed using the ROB 2 tool. Studies with comparable outcomes will be pooled for meta-analysis, employing the DerSimonian and Laird method in R software, with heterogeneity assessed via the I-squared statistic. A meta-regression analysis will explore age-related heterogeneity, and subgroup analysis will investigate bias risk levels. Publication bias will be assessed using the Trim and Fill technique and Egger and Begg tests. The outcomes' quality will be evaluated using the GRADE framework. The findings are expected to guide future research, influence health policy, and inform healthcare professionals and policymakers about integrating smart glasses into healthcare systems.
- Open Access
- 23 February 2024
- Published by Skyler Publication in Frontiers in Health Informatics
- Vol. 13, 191
Readability and credibility evaluation of most-visited health websites based on eBizMBA and Alexa global ranking
Introduction: Online health information is one of the most important and widely used sources of information. Currently, a significant number of individuals use the internet for health-related information to learn about health, disease, health promotion, and threats to their health.Material and Methods: This research examined the top 15 health websites based on their popularity in eBizMBA and Alexa rank from January to February 2022. A total of 30 health websites (15 from each category) were evaluated in terms of credibility by the use of Journal of the American Medical Association (JAMA) and health on the net foundation (HON) code. Also, the readability of the mentioned websites was evaluated by the use of four readability tools.Results: Most of the websites ranked by both Alexa and eBizMBA met the “Authority” and” Complementarity" criteria. When it comes to the readability of the 30 most visited health websites according to eBizMBA and Alexa analytics in the world, the analysis demonstrated that the readability of highly-ranked health websites was higher than the recommended level.Conclusion: Some of the websites had deficiencies according to the HONcode and JAMA criteria, and the average readability of the websites did not meet the gold standard. Despite the increasing use of the internet for medical information, these resources' poor quality and readability remain a barrier to informed decision-making of patients.
- 23 February 2024
- Published by Skyler Publication in Frontiers in Health Informatics
- Vol. 13, 191
Open AccessReadability and credibility evaluation of most-visited health websites based on eBizMBA and Alexa global ranking
Introduction: Online health information is one of the most important and widely used sources of information. Currently, a significant number of individuals use the internet for health-related information to learn about health, disease, health promotion, and threats to their health.Material and Methods: This research examined the top 15 health websites based on their popularity in eBizMBA and Alexa rank from January to February 2022. A total of 30 health websites (15 from each category) were evaluated in terms of credibility by the use of Journal of the American Medical Association (JAMA) and health on the net foundation (HON) code. Also, the readability of the mentioned websites was evaluated by the use of four readability tools.Results: Most of the websites ranked by both Alexa and eBizMBA met the “Authority” and” Complementarity" criteria. When it comes to the readability of the 30 most visited health websites according to eBizMBA and Alexa analytics in the world, the analysis demonstrated that the readability of highly-ranked health websites was higher than the recommended level.Conclusion: Some of the websites had deficiencies according to the HONcode and JAMA criteria, and the average readability of the websites did not meet the gold standard. Despite the increasing use of the internet for medical information, these resources' poor quality and readability remain a barrier to informed decision-making of patients.
- Open Access
- 6 February 2024
- Published by Springer Nature in Scientometrics
- Vol. 129 (3) , 1739-1770
Ranking resilience: assessing the impact of scientific performance and the expansion of the Times Higher Education Word University Rankings on the position of Czech, Hungarian, Polish, and Slovak universities
University rankings are gaining importance worldwide. While the top-ranked universities are the subject of numerous academic publications, very little literature is available on universities in the lower two-thirds of the ranking list. In this study, we analysed the sensitivity of year to year changes in position in the Times Higher Education World University Rankings (THE) of universities from the Czech Republic, Hungary, Poland, and Slovakia using hypothetical scenarios of potential changes in their research outputs. A regression model was developed that quantifies the effect of the inclusion of institutions that are new to ranking as well as dropouts on the ranking list. We also developed a method to allow customized sensitivity analysis of the subsequent year's rank from an institution's projected research output, given its current rank and research output, assuming that other ranking indicators remain unchanged over 1–2 years. It should be noted that most universities in the region have ranked below 800, and while research output has been increasing, it was observed that ranks worsened for most universities from these four countries. The Field-Weighted Citation Impact and the total volume of publications had the highest effect on rankings, while the proportion of Scimago Q1 publications did not have a consistent effect on the projected next year's ranks. In most universities, the growth in research performance indicators were below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income. With due caution, our results might be generalized to most of the universities below 800 in the THE. The rank prediction tool presented in this article is available online at https://hecon.uni-obuda.hu/en/rank-calculator/.
- 6 February 2024
- Published by Springer Nature in Scientometrics
- Vol. 129 (3) , 1739-1770
Open AccessRanking resilience: assessing the impact of scientific performance and the expansion of the Times Higher Education Word University Rankings on the position of Czech, Hungarian, Polish, and Slovak universities
University rankings are gaining importance worldwide. While the top-ranked universities are the subject of numerous academic publications, very little literature is available on universities in the lower two-thirds of the ranking list. In this study, we analysed the sensitivity of year to year changes in position in the Times Higher Education World University Rankings (THE) of universities from the Czech Republic, Hungary, Poland, and Slovakia using hypothetical scenarios of potential changes in their research outputs. A regression model was developed that quantifies the effect of the inclusion of institutions that are new to ranking as well as dropouts on the ranking list. We also developed a method to allow customized sensitivity analysis of the subsequent year's rank from an institution's projected research output, given its current rank and research output, assuming that other ranking indicators remain unchanged over 1–2 years. It should be noted that most universities in the region have ranked below 800, and while research output has been increasing, it was observed that ranks worsened for most universities from these four countries. The Field-Weighted Citation Impact and the total volume of publications had the highest effect on rankings, while the proportion of Scimago Q1 publications did not have a consistent effect on the projected next year's ranks. In most universities, the growth in research performance indicators were below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income. With due caution, our results might be generalized to most of the universities below 800 in the THE. The rank prediction tool presented in this article is available online at https://hecon.uni-obuda.hu/en/rank-calculator/.
- Open Access
- 19 January 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e47430
The Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review
Background: Diabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. Objective: We aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. Methods: We searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. Results: After screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. Conclusions: The reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable.
- 19 January 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e47430
Open AccessThe Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review
Background: Diabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. Objective: We aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. Methods: We searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. Results: After screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. Conclusions: The reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable.
- Open Access
- 30 October 2023
- Published by Springer Nature in Research Square
Ranking resilience: assessing the impact of scientific performance and the expansion of the Times Higher Education Word University rankings on the position of Czech, Hungarian, Polish, and Slovak universities
University rankings are gaining importance worldwide. While the top-ranked universities are the subject of numerous academic publications, very little literature is available on universities in the lower two-thirds of the ranking list. In this study, we analysed the sensitivity of year to year changes in position in the Times Higher Education World University Rankings (THE) of universities from the Czech Republic, Hungary, Poland, and Slovakia using hypothetical scenarios of potential changes in their research outputs. A regression model was developed that quantifies the effect of the inclusion of institutions that are new to ranking as well as dropouts on the ranking list. We also developed a method to allow customized sensitivity analysis of the subsequent year's rank from an institution's projected research output, given its current rank and research output, assuming that other ranking indicators remain unchanged over 1–2 years. It should be noted that universities in the region have ranked below 800, and while research output has been increasing, it was observed that ranks worsen for most universities from these four countries. The field-weighted citation index and the total volume of publications had the highest effect on rankings, while the proportion of Scimago Q1 publications did not have a consistent effect on the projected next year's ranks. In most universities, the growth in research performance indicators were below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income. With due caution, our results might be generalized to most of the universities below 800 in the THE. The rank prediction tool presented in this article is available online at https://hecon.uni-obuda.hu/en/rank-calculator/. JEL code I21, I23, I28
- 30 October 2023
- Published by Springer Nature in Research Square
Open AccessRanking resilience: assessing the impact of scientific performance and the expansion of the Times Higher Education Word University rankings on the position of Czech, Hungarian, Polish, and Slovak universities
University rankings are gaining importance worldwide. While the top-ranked universities are the subject of numerous academic publications, very little literature is available on universities in the lower two-thirds of the ranking list. In this study, we analysed the sensitivity of year to year changes in position in the Times Higher Education World University Rankings (THE) of universities from the Czech Republic, Hungary, Poland, and Slovakia using hypothetical scenarios of potential changes in their research outputs. A regression model was developed that quantifies the effect of the inclusion of institutions that are new to ranking as well as dropouts on the ranking list. We also developed a method to allow customized sensitivity analysis of the subsequent year's rank from an institution's projected research output, given its current rank and research output, assuming that other ranking indicators remain unchanged over 1–2 years. It should be noted that universities in the region have ranked below 800, and while research output has been increasing, it was observed that ranks worsen for most universities from these four countries. The field-weighted citation index and the total volume of publications had the highest effect on rankings, while the proportion of Scimago Q1 publications did not have a consistent effect on the projected next year's ranks. In most universities, the growth in research performance indicators were below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income. With due caution, our results might be generalized to most of the universities below 800 in the THE. The rank prediction tool presented in this article is available online at https://hecon.uni-obuda.hu/en/rank-calculator/. JEL code I21, I23, I28