Ziyan Xiao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Ziyan Xiao has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Xiuping Han
Research Center Intelligent Equipment and Technology, Yangtze Delta Region Institute of Tsinghua University, Jiaxing, Zhejiang, China
2 shared publications - Celine Sze Ling Chui
School of Nursing, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong, China
1 shared publication - Daniel Yee Tak Fong
School of Nursing The University of Hong Kong Pokfulam Hong Kong
1 shared publication - Esther Wy Chan
The University of Hong Kong
1 shared publication - Francisco Tsz Tsun Lai
The University of Hong Kong
1 shared publication - Ian Chi Kei Wong
The University of Hong Kong
1 shared publication
Latest Publications
- Open Access
Large Language Model in Medical Information Extraction from Titles and Abstracts with Prompt Engineering Strategies: A Comparative Study of GPT-3.5 and GPT-4
Background Large language models (LLMs) have significantly enhanced the Natural Language Processing (NLP), offering significant potential in facilitating medical literature review. However, the accuracy, stability and prompt strategies associated with LLMs in extracting complex medical information have not been adequately investigated. Our study assessed the capabilities of GPT-3.5 and GPT-4.0 in extracting or summarizing seven crucial medical information items from the title and abstract of research papers. We also validated the impact of prompt engineering strategies and the effectiveness of evaluating metrics. Methodology We adopted a stratified sampling method to select 100 papers from the teaching schools and departments in the LKS Faculty of Medicine, University of Hong Kong, published between 2015 and 2023. GPT-3.5 and GPT-4.0 were instructed to extract seven pieces of information, including study design, sample size, data source, patient, intervention, comparison, and outcomes. The experiment incorporated three prompt engineering strategies: persona, chain-of-thought and few-shot prompting. We employed three metrics to assess the alignment between the GPT output and the ground truth: BERTScore, ROUGE-1 and a self-developed GPT-4.0 evaluator. Finally, we evaluated and compared the proportion of correct answers among different GPT versions and prompt engineering strategies. Results GPT demonstrated robust capabilities in accurately extracting medical information from titles and abstracts. The average accuracy of GPT-4.0, when paired with the optimal prompt engineering strategy, ranged from 0.688 to 0.964 among the seven items, with sample size achieving the highest score and intervention yielding the lowest. GPT version was shown to be a statistically significant factor in model performance, but prompt engineering strategies did not exhibit cumulative effects on model performance. Additionally, our results showed that the GPT-4.0 evaluator outperformed the ROUGE-1 and BERTScore in assessing the alignment of information (Accuracy: GPT-4.0 Evaluator: 0.9714, ROUGE-1: 0.9429, BERTScore: 0.8714). Conclusion Our result confirms the effectiveness of LLMs in extracting medical information, suggesting their potential as efficient tools for literature review. We recommend utilizing an advanced version of LLMs to enhance the model performance, while prompt engineering strategies should be tailored to the specific tasks. Additionally, LLMs show promise as an evaluation tool to assess the model performance related to complex information processing.
Open AccessLarge Language Model in Medical Information Extraction from Titles and Abstracts with Prompt Engineering Strategies: A Comparative Study of GPT-3.5 and GPT-4
Background Large language models (LLMs) have significantly enhanced the Natural Language Processing (NLP), offering significant potential in facilitating medical literature review. However, the accuracy, stability and prompt strategies associated with LLMs in extracting complex medical information have not been adequately investigated. Our study assessed the capabilities of GPT-3.5 and GPT-4.0 in extracting or summarizing seven crucial medical information items from the title and abstract of research papers. We also validated the impact of prompt engineering strategies and the effectiveness of evaluating metrics. Methodology We adopted a stratified sampling method to select 100 papers from the teaching schools and departments in the LKS Faculty of Medicine, University of Hong Kong, published between 2015 and 2023. GPT-3.5 and GPT-4.0 were instructed to extract seven pieces of information, including study design, sample size, data source, patient, intervention, comparison, and outcomes. The experiment incorporated three prompt engineering strategies: persona, chain-of-thought and few-shot prompting. We employed three metrics to assess the alignment between the GPT output and the ground truth: BERTScore, ROUGE-1 and a self-developed GPT-4.0 evaluator. Finally, we evaluated and compared the proportion of correct answers among different GPT versions and prompt engineering strategies. Results GPT demonstrated robust capabilities in accurately extracting medical information from titles and abstracts. The average accuracy of GPT-4.0, when paired with the optimal prompt engineering strategy, ranged from 0.688 to 0.964 among the seven items, with sample size achieving the highest score and intervention yielding the lowest. GPT version was shown to be a statistically significant factor in model performance, but prompt engineering strategies did not exhibit cumulative effects on model performance. Additionally, our results showed that the GPT-4.0 evaluator outperformed the ROUGE-1 and BERTScore in assessing the alignment of information (Accuracy: GPT-4.0 Evaluator: 0.9714, ROUGE-1: 0.9429, BERTScore: 0.8714). Conclusion Our result confirms the effectiveness of LLMs in extracting medical information, suggesting their potential as efficient tools for literature review. We recommend utilizing an advanced version of LLMs to enhance the model performance, while prompt engineering strategies should be tailored to the specific tasks. Additionally, LLMs show promise as an evaluation tool to assess the model performance related to complex information processing.
- Open Access
- 27 April 2023
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 25, e44256
Evaluation of the Effectiveness of Telehealth Chronic Disease Management System: Systematic Review and Meta-analysis
Background: Long-term daily health monitoring and management play a more significant role in telehealth management systems nowadays, which require evaluation indicators to present patients’ general health conditions and become applicable to multiple chronic diseases. Objective: This study aims to evaluate the effectiveness of subjective indicators of telehealth chronic disease management system (TCDMS). Methods: We selected Web of Science, ScienceDirect, Scopus, Cochrane library, IEEE, and Chinese National Knowledge Infrastructure and Wanfang, a Chinese medical database, and searched papers published from January 1, 2015, to July 1, 2022, regarding randomized controlled trials on the effectiveness of the telehealth system on patients with chronic diseases. The narrative review summarized the questionnaire indicators presented in the selected studies. In the meta-analysis, Mean Difference (MD) and Standardized Mean Difference (SMD) with a 95% CI were pooled depending on whether the measurements were the same. Subgroup analysis was conducted if the heterogeneity was significant, and the number of studies was sufficient. Results: Twenty RCTs with 4153 patients were included in the qualitative review. Seventeen different questionnaire-based outcomes were found, within which quality of life, psychological well-being (including depression, anxiety, and fatigue), self-management, self-efficacy, and medical adherence were most frequently used. Ten RCTs with 2095 patients remained in meta-analysis. Compared to usual care, telehealth system can significantly improve the quality of life (SMD 0.44; 95% CI 0.16-0.73; P=.002), whereas no significant effects were found on depression (SMD −0.25; 95% CI −0.72 to 0.23; P=.30), anxiety (SMD −0.10; 95% CI −0.27 to 0.07; P=.71), fatigue (SMD −0.36; 95% CI −1.06 to 0.34; P<.001), and self-care (SMD 0.77; 95% CI −0.28-1.81; P<.001). In the subdomains of quality of life, telehealth statistically significantly improved physical functioning (SMD 0.15; 95% CI 0.02 to 0.29; P=.03), mental functioning (SMD 0.37; 95% CI 0.13-0.60; P=.002), and social functioning (SMD 0.64; 95% CI 0.00-1.29; P=.05), while there was no difference on cognitive functioning (MD 8.31; 95% CI −7.33 to 23.95; P=.30) and role functioning (MD 5.30; 95% CI −7.80 to 18.39; P=.43). Conclusions: TCDMS positively affected patients’ physical, mental, and social quality of life across multiple chronic diseases. However, no significant difference was found in depression, anxiety, fatigue, and self-care. Subjective questionnaires had the potential ability to evaluate the effectiveness of long-term telehealth monitoring and management. However, further well-designed experiments are warranted to validate TCDMS’s effects on subjective outcomes, especially when tested among different chronically ill groups.
- 27 April 2023
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 25, e44256
Open AccessEvaluation of the Effectiveness of Telehealth Chronic Disease Management System: Systematic Review and Meta-analysis
Background: Long-term daily health monitoring and management play a more significant role in telehealth management systems nowadays, which require evaluation indicators to present patients’ general health conditions and become applicable to multiple chronic diseases. Objective: This study aims to evaluate the effectiveness of subjective indicators of telehealth chronic disease management system (TCDMS). Methods: We selected Web of Science, ScienceDirect, Scopus, Cochrane library, IEEE, and Chinese National Knowledge Infrastructure and Wanfang, a Chinese medical database, and searched papers published from January 1, 2015, to July 1, 2022, regarding randomized controlled trials on the effectiveness of the telehealth system on patients with chronic diseases. The narrative review summarized the questionnaire indicators presented in the selected studies. In the meta-analysis, Mean Difference (MD) and Standardized Mean Difference (SMD) with a 95% CI were pooled depending on whether the measurements were the same. Subgroup analysis was conducted if the heterogeneity was significant, and the number of studies was sufficient. Results: Twenty RCTs with 4153 patients were included in the qualitative review. Seventeen different questionnaire-based outcomes were found, within which quality of life, psychological well-being (including depression, anxiety, and fatigue), self-management, self-efficacy, and medical adherence were most frequently used. Ten RCTs with 2095 patients remained in meta-analysis. Compared to usual care, telehealth system can significantly improve the quality of life (SMD 0.44; 95% CI 0.16-0.73; P=.002), whereas no significant effects were found on depression (SMD −0.25; 95% CI −0.72 to 0.23; P=.30), anxiety (SMD −0.10; 95% CI −0.27 to 0.07; P=.71), fatigue (SMD −0.36; 95% CI −1.06 to 0.34; P<.001), and self-care (SMD 0.77; 95% CI −0.28-1.81; P<.001). In the subdomains of quality of life, telehealth statistically significantly improved physical functioning (SMD 0.15; 95% CI 0.02 to 0.29; P=.03), mental functioning (SMD 0.37; 95% CI 0.13-0.60; P=.002), and social functioning (SMD 0.64; 95% CI 0.00-1.29; P=.05), while there was no difference on cognitive functioning (MD 8.31; 95% CI −7.33 to 23.95; P=.30) and role functioning (MD 5.30; 95% CI −7.80 to 18.39; P=.43). Conclusions: TCDMS positively affected patients’ physical, mental, and social quality of life across multiple chronic diseases. However, no significant difference was found in depression, anxiety, fatigue, and self-care. Subjective questionnaires had the potential ability to evaluate the effectiveness of long-term telehealth monitoring and management. However, further well-designed experiments are warranted to validate TCDMS’s effects on subjective outcomes, especially when tested among different chronically ill groups.
- Open Access
- 12 November 2022
- Published by JMIR Publications Inc. in JMIR Preprints
Evaluation of the Effectiveness of Telehealth Chronic Disease Management System: Systematic Review and Meta-analysis (Preprint)
BACKGROUND Long-term daily health monitoring and management play a more significant role in telehealth management systems nowadays, which require evaluation indicators to present patients’ general health conditions and become applicable to multiple chronic diseases. OBJECTIVE This study aims to evaluate the effectiveness of subjective indicators of telehealth chronic disease management system (TCDMS). METHODS We selected Web of Science, ScienceDirect, Scopus, Cochrane library, IEEE, and Chinese National Knowledge Infrastructure and Wanfang, a Chinese medical database, and searched papers published from January 1, 2015, to July 1, 2022, regarding randomized controlled trials on the effectiveness of the telehealth system on patients with chronic diseases. The narrative review summarized the questionnaire indicators presented in the selected studies. In the meta-analysis, Mean Difference (MD) and Standardized Mean Difference (SMD) with a 95% CI were pooled depending on whether the measurements were the same. Subgroup analysis was conducted if the heterogeneity was significant, and the number of studies was sufficient. RESULTS Twenty RCTs with 4153 patients were included in the qualitative review. Seventeen different questionnaire-based outcomes were found, within which quality of life, psychological well-being (including depression, anxiety, and fatigue), self-management, self-efficacy, and medical adherence were most frequently used. Ten RCTs with 2095 patients remained in meta-analysis. Compared to usual care, telehealth system can significantly improve the quality of life (SMD 0.44; 95% CI 0.16-0.73; P=.002), whereas no significant effects were found on depression (SMD −0.25; 95% CI −0.72 to 0.23; P=.30), anxiety (SMD −0.10; 95% CI −0.27 to 0.07; P=.71), fatigue (SMD −0.36; 95% CI −1.06 to 0.34; P<.001), and self-care (SMD 0.77; 95% CI −0.28-1.81; P<.001). In the subdomains of quality of life, telehealth statistically significantly improved physical functioning (SMD 0.15; 95% CI 0.02 to 0.29; P=.03), mental functioning (SMD 0.37; 95% CI 0.13-0.60; P=.002), and social functioning (SMD 0.64; 95% CI 0.00-1.29; P=.05), while there was no difference on cognitive functioning (MD 8.31; 95% CI −7.33 to 23.95; P=.30) and role functioning (MD 5.30; 95% CI −7.80 to 18.39; P=.43). CONCLUSIONS TCDMS positively affected patients’ physical, mental, and social quality of life across multiple chronic diseases. However, no significant difference was found in depression, anxiety, fatigue, and self-care. Subjective questionnaires had the potential ability to evaluate the effectiveness of long-term telehealth monitoring and management. However, further well-designed experiments are warranted to validate TCDMS’s effects on subjective outcomes, especially when tested among different chronically ill groups.
- 12 November 2022
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessEvaluation of the Effectiveness of Telehealth Chronic Disease Management System: Systematic Review and Meta-analysis (Preprint)
BACKGROUND Long-term daily health monitoring and management play a more significant role in telehealth management systems nowadays, which require evaluation indicators to present patients’ general health conditions and become applicable to multiple chronic diseases. OBJECTIVE This study aims to evaluate the effectiveness of subjective indicators of telehealth chronic disease management system (TCDMS). METHODS We selected Web of Science, ScienceDirect, Scopus, Cochrane library, IEEE, and Chinese National Knowledge Infrastructure and Wanfang, a Chinese medical database, and searched papers published from January 1, 2015, to July 1, 2022, regarding randomized controlled trials on the effectiveness of the telehealth system on patients with chronic diseases. The narrative review summarized the questionnaire indicators presented in the selected studies. In the meta-analysis, Mean Difference (MD) and Standardized Mean Difference (SMD) with a 95% CI were pooled depending on whether the measurements were the same. Subgroup analysis was conducted if the heterogeneity was significant, and the number of studies was sufficient. RESULTS Twenty RCTs with 4153 patients were included in the qualitative review. Seventeen different questionnaire-based outcomes were found, within which quality of life, psychological well-being (including depression, anxiety, and fatigue), self-management, self-efficacy, and medical adherence were most frequently used. Ten RCTs with 2095 patients remained in meta-analysis. Compared to usual care, telehealth system can significantly improve the quality of life (SMD 0.44; 95% CI 0.16-0.73; P=.002), whereas no significant effects were found on depression (SMD −0.25; 95% CI −0.72 to 0.23; P=.30), anxiety (SMD −0.10; 95% CI −0.27 to 0.07; P=.71), fatigue (SMD −0.36; 95% CI −1.06 to 0.34; P<.001), and self-care (SMD 0.77; 95% CI −0.28-1.81; P<.001). In the subdomains of quality of life, telehealth statistically significantly improved physical functioning (SMD 0.15; 95% CI 0.02 to 0.29; P=.03), mental functioning (SMD 0.37; 95% CI 0.13-0.60; P=.002), and social functioning (SMD 0.64; 95% CI 0.00-1.29; P=.05), while there was no difference on cognitive functioning (MD 8.31; 95% CI −7.33 to 23.95; P=.30) and role functioning (MD 5.30; 95% CI −7.80 to 18.39; P=.43). CONCLUSIONS TCDMS positively affected patients’ physical, mental, and social quality of life across multiple chronic diseases. However, no significant difference was found in depression, anxiety, fatigue, and self-care. Subjective questionnaires had the potential ability to evaluate the effectiveness of long-term telehealth monitoring and management. However, further well-designed experiments are warranted to validate TCDMS’s effects on subjective outcomes, especially when tested among different chronically ill groups.