Zhenyu Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Zhenyu Zhang has a total of 44 co-authors. Here are the top co-authors by number of shared publications:
- Ping Yu
School of Computing and Information Technology, University of Wollongong, Wollongong, Australia
10 shared publications - Mengyang Yin
Opal Healthcare, Level 11/420 George St, Sydney NSW 2000, Australia
7 shared publications - Yunshu Zhu
School of Computing and Information Technology, University of Wollongong, Wollongong, Australia
7 shared publications - Chao Deng
School of Medical, Indigenous and Health Sciences, University of Wollongong, Wollongong, Australia
6 shared publications - Ting Song
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW 2522, Australia
6 shared publications - Dinithi Vithanage
School of Computing and Information Technology, University of Wollongong, Wollongong, Australia
3 shared publications
Latest Publications
- Open Access
- 20 February 2025
- Published by Springer Nature in Journal of Healthcare Informatics Research
- Vol. 9 (2) , 191-219
Adapting Generative Large Language Models for Information Extraction from Unstructured Electronic Health Records in Residential Aged Care: A Comparative Analysis of Training Approaches
Information extraction (IE) of unstructured electronic health records is challenging due to the semantic complexity of textual data. Generative large language models (LLMs) offer promising solutions to address this challenge. However, identifying the best training methods to adapt LLMs for IE in residential aged care settings remains underexplored. This research addresses this challenge by evaluating the effects of zero-shot and few-shot learning, both with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) using Llama 3.1-8B. The study performed named entity recognition (NER) to nursing notes from Australian aged care facilities (RACFs), focusing on agitation in dementia and malnutrition risk factors. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score. We used non-parametric statistical methods to compare if the differences were statistically significant. Results show that zero-shot and few-shot learning, whether combined with PEFT or RAG, achieve comparable performance across the clinical domains when the same prompting template is used. Few-shot learning significantly outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly improves model performance in both zero-shot and few-shot learning; however, RAG significantly improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning reaches a comparable level with few-shot learning. However, few-shot learning with RAG significantly outperforms zero-shot learning with RAG. We also found a similar level of performance between few-shot learning with RAG and zero-shot learning with PEFT. These findings provide valuable insights for researchers, practitioners, and stakeholders to optimize the use of generative LLMs in clinical IE.
- 20 February 2025
- Published by Springer Nature in Journal of Healthcare Informatics Research
- Vol. 9 (2) , 191-219
Open AccessAdapting Generative Large Language Models for Information Extraction from Unstructured Electronic Health Records in Residential Aged Care: A Comparative Analysis of Training Approaches
Information extraction (IE) of unstructured electronic health records is challenging due to the semantic complexity of textual data. Generative large language models (LLMs) offer promising solutions to address this challenge. However, identifying the best training methods to adapt LLMs for IE in residential aged care settings remains underexplored. This research addresses this challenge by evaluating the effects of zero-shot and few-shot learning, both with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) using Llama 3.1-8B. The study performed named entity recognition (NER) to nursing notes from Australian aged care facilities (RACFs), focusing on agitation in dementia and malnutrition risk factors. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score. We used non-parametric statistical methods to compare if the differences were statistically significant. Results show that zero-shot and few-shot learning, whether combined with PEFT or RAG, achieve comparable performance across the clinical domains when the same prompting template is used. Few-shot learning significantly outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly improves model performance in both zero-shot and few-shot learning; however, RAG significantly improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning reaches a comparable level with few-shot learning. However, few-shot learning with RAG significantly outperforms zero-shot learning with RAG. We also found a similar level of performance between few-shot learning with RAG and zero-shot learning with PEFT. These findings provide valuable insights for researchers, practitioners, and stakeholders to optimize the use of generative LLMs in clinical IE.
- Open Access
- 30 September 2024
- Published by MDPI AG in Psychiatry International
- Vol. 5 (4) , 642-659
COVID-19 and Its Influence on Prevalence of Dementia and Agitation in Australian Residential Aged Care: A Comparative Study
Agitation is one of the most common and persistent behavioral and psychological symptoms among persons with dementia (PWD) in residential aged care facilities (RACFs). While most studies have explored the general impact of COVID-19 on the mental health and well-being of aged care residents, there has been limited research on the pandemic’s impact on agitation in dementia within RACFs. This knowledge is crucial to ensuring that RACFs are better equipped to handle future public health emergencies. Therefore, this retrospective cohort study investigates the impact of the pandemic on agitation incidents within 40 Australian RACFs. Using Python, we extracted agitation symptoms from nursing notes and compared the frequency and percentage of symptom occurrence pre-pandemic versus during-pandemic. Chi-square tests examined any significant change in the prevalence of dementia and agitation in dementia between the comparative groups and periods. Dementia prevalence dropped significantly during the second year of the pandemic, with a concurrent increase in early-onset dementia cases. Overall, agitation symptoms decreased by 20.1%, but six symptoms significantly increased: resisting (28.98%), wandering (11.78%), restlessness (3.19%), complaining (10.1%), arguing (2.36%), and outbursts (1.74%). Conversely, pacing decreased by 15.88% and speaking loudly decreased by 10.9%. Over half of the care recipients with dementia experienced agitation symptoms 2–5 times each year, increasing from 50.56% in 2019 to 58.28% in 2021. Despite the co-occurrence of symptoms such as falls, confusion, and weakness, there was no evidence to suggest that these comorbidities were specific to COVID-19; rather, they appeared to be prevalent even before the pandemic. Persons with agitation in dementia had a significantly higher death rate during the COVID-19 pandemic than otherwise.
- 30 September 2024
- Published by MDPI AG in Psychiatry International
- Vol. 5 (4) , 642-659
Open AccessCOVID-19 and Its Influence on Prevalence of Dementia and Agitation in Australian Residential Aged Care: A Comparative Study
Agitation is one of the most common and persistent behavioral and psychological symptoms among persons with dementia (PWD) in residential aged care facilities (RACFs). While most studies have explored the general impact of COVID-19 on the mental health and well-being of aged care residents, there has been limited research on the pandemic’s impact on agitation in dementia within RACFs. This knowledge is crucial to ensuring that RACFs are better equipped to handle future public health emergencies. Therefore, this retrospective cohort study investigates the impact of the pandemic on agitation incidents within 40 Australian RACFs. Using Python, we extracted agitation symptoms from nursing notes and compared the frequency and percentage of symptom occurrence pre-pandemic versus during-pandemic. Chi-square tests examined any significant change in the prevalence of dementia and agitation in dementia between the comparative groups and periods. Dementia prevalence dropped significantly during the second year of the pandemic, with a concurrent increase in early-onset dementia cases. Overall, agitation symptoms decreased by 20.1%, but six symptoms significantly increased: resisting (28.98%), wandering (11.78%), restlessness (3.19%), complaining (10.1%), arguing (2.36%), and outbursts (1.74%). Conversely, pacing decreased by 15.88% and speaking loudly decreased by 10.9%. Over half of the care recipients with dementia experienced agitation symptoms 2–5 times each year, increasing from 50.56% in 2019 to 58.28% in 2021. Despite the co-occurrence of symptoms such as falls, confusion, and weakness, there was no evidence to suggest that these comorbidities were specific to COVID-19; rather, they appeared to be prevalent even before the pandemic. Persons with agitation in dementia had a significantly higher death rate during the COVID-19 pandemic than otherwise.
- Open Access
- 24 July 2024
- Published by MDPI AG in Preprints.org
COVID-19 and Its Influence on Prevalence of Dementia and Agitation in Australian Residential Age Care: A Comparative Study
The COVID-19 pandemic profoundly impacted older people in residential aged care facilities (RACFs), especially those with dementia. This retrospective cohort study investigates the impact of the pandemic on agitation incidents within this population in 40 Australian RACFs. We compared pre-pandemic data (January-October 2019) with during-pandemic data (same months in 2020 and 2021) from electronic health records (EHRs) using Python to extract information from nursing notes. Various statistical analyses, including descriptive statistics to summarize the data, and a chi-square test to examine any significant association between the COVID-19 pandemic and the prevalence of dementia and agitation in dementia. Dementia prevalence dropped by 1.98% in 2021 compared to 2019, and by 4.09% compared to 2020 (p < 0.05). Overall agitation symptoms decreased by 20.1% from 2019 to 2021 (p < 0.05), but six specific symptoms increased: resisting (28.98%), wandering (11.78%), restlessness (3.19%), complaining (10.1%), arguing (2.36%), and outbursts (1.74%) (p < 0.05). Conversely, pacing decreased by 15.88% and speaking loudly de-creased by 10.9% (p < 0.05). Over half of the care recipients experienced agitation symptoms 2-5 times each year, increasing from 50.56% in 2019 to 58.28% in 2021 (p < 0.05). It is important to exercise caution regarding the accuracy of these findings due to the reduced opportunity for busy care staff to record behavioral changes caused by the isolation of individuals in their rooms during the pandemic.
- 24 July 2024
- Published by MDPI AG in Preprints.org
Open AccessCOVID-19 and Its Influence on Prevalence of Dementia and Agitation in Australian Residential Age Care: A Comparative Study
The COVID-19 pandemic profoundly impacted older people in residential aged care facilities (RACFs), especially those with dementia. This retrospective cohort study investigates the impact of the pandemic on agitation incidents within this population in 40 Australian RACFs. We compared pre-pandemic data (January-October 2019) with during-pandemic data (same months in 2020 and 2021) from electronic health records (EHRs) using Python to extract information from nursing notes. Various statistical analyses, including descriptive statistics to summarize the data, and a chi-square test to examine any significant association between the COVID-19 pandemic and the prevalence of dementia and agitation in dementia. Dementia prevalence dropped by 1.98% in 2021 compared to 2019, and by 4.09% compared to 2020 (p < 0.05). Overall agitation symptoms decreased by 20.1% from 2019 to 2021 (p < 0.05), but six specific symptoms increased: resisting (28.98%), wandering (11.78%), restlessness (3.19%), complaining (10.1%), arguing (2.36%), and outbursts (1.74%) (p < 0.05). Conversely, pacing decreased by 15.88% and speaking loudly de-creased by 10.9% (p < 0.05). Over half of the care recipients experienced agitation symptoms 2-5 times each year, increasing from 50.56% in 2019 to 58.28% in 2021 (p < 0.05). It is important to exercise caution regarding the accuracy of these findings due to the reduced opportunity for busy care staff to record behavioral changes caused by the isolation of individuals in their rooms during the pandemic.
- Open Access
Evaluating approaches of training a generative large language model for multi-label classification of unstructured electronic health records
Multi-label classification of unstructured electronic health records (EHR) is challenging due to the semantic complexity of textual data. Identifying the most effective machine learning method for EHR classification is useful in real-world clinical settings. Advances in natural language processing (NLP) using large language models (LLMs) offer promising solutions. Therefore, this experimental research aims to test the effects of zero-shot and few-shot learning prompting, with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) of LLMs, on the multi-label classification of unstructured EHR data from residential aged care facilities (RACFs) in Australia. The four clinical tasks examined are agitation in dementia, depression in dementia, frailty index, and malnutrition risk factors, using the Llama 3.1-8B. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score, supported by non-parametric statistical analyses. Results indicate that both zero-shot and few-shot learning, regardless of the use of PEFT and RAG, demonstrate equivalent performance across the clinical tasks when using the same prompting template. Few-shot learning consistently outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly enhances model performance in both zero-shot and few-shot learning; however, RAG improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning is equal to that of few-shot learning across clinical tasks. Additionally, few-shot learning with RAG surpasses zero-shot learning with RAG, while no significant difference exists between few-shot learning with RAG and zero-shot learning with PEFT. These findings offer crucial insights into LLMs for researchers, practitioners, and stakeholders utilizing LLMs in clinical document analysis.
Open AccessEvaluating approaches of training a generative large language model for multi-label classification of unstructured electronic health records
Multi-label classification of unstructured electronic health records (EHR) is challenging due to the semantic complexity of textual data. Identifying the most effective machine learning method for EHR classification is useful in real-world clinical settings. Advances in natural language processing (NLP) using large language models (LLMs) offer promising solutions. Therefore, this experimental research aims to test the effects of zero-shot and few-shot learning prompting, with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) of LLMs, on the multi-label classification of unstructured EHR data from residential aged care facilities (RACFs) in Australia. The four clinical tasks examined are agitation in dementia, depression in dementia, frailty index, and malnutrition risk factors, using the Llama 3.1-8B. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score, supported by non-parametric statistical analyses. Results indicate that both zero-shot and few-shot learning, regardless of the use of PEFT and RAG, demonstrate equivalent performance across the clinical tasks when using the same prompting template. Few-shot learning consistently outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly enhances model performance in both zero-shot and few-shot learning; however, RAG improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning is equal to that of few-shot learning across clinical tasks. Additionally, few-shot learning with RAG surpasses zero-shot learning with RAG, while no significant difference exists between few-shot learning with RAG and zero-shot learning with PEFT. These findings offer crucial insights into LLMs for researchers, practitioners, and stakeholders utilizing LLMs in clinical document analysis.
- Open Access
- 25 January 2024
- Published by IOS Press
- Vol. 310, 109-113
A Five-Step Workflow to Manually Annotate Unstructured Data into Training Dataset for Natural Language Processing
Natural Language Processing (NLP) is a powerful technique for extracting valuable information from unstructured electronic health records (EHRs). However, a prerequisite for NLP is the availability of high-quality annotated datasets. To date, there is a lack of effective methods to guide the research effort of manually annotating unstructured datasets, which can hinder NLP performance. Therefore, this study develops a five-step workflow for manually annotating unstructured datasets, including (1) annotator training and familiarising with the text corpus, (2) vocabulary identification, (3) annotation schema development, (4) annotation execution, and (5) result validation. This framework was then applied to annotate agitation symptoms from the unstructured EHRs of 40 Australian residential aged care facilities. The annotated corpus achieved an accuracy rate of 96%. This suggests that our proposed annotation workflow can be used in manual data processing to develop annotated training corpus for developing NLP algorithms.
- 25 January 2024
- Published by IOS Press
- Vol. 310, 109-113
Open AccessA Five-Step Workflow to Manually Annotate Unstructured Data into Training Dataset for Natural Language Processing
Natural Language Processing (NLP) is a powerful technique for extracting valuable information from unstructured electronic health records (EHRs). However, a prerequisite for NLP is the availability of high-quality annotated datasets. To date, there is a lack of effective methods to guide the research effort of manually annotating unstructured datasets, which can hinder NLP performance. Therefore, this study develops a five-step workflow for manually annotating unstructured datasets, including (1) annotator training and familiarising with the text corpus, (2) vocabulary identification, (3) annotation schema development, (4) annotation execution, and (5) result validation. This framework was then applied to annotate agitation symptoms from the unstructured EHRs of 40 Australian residential aged care facilities. The annotated corpus achieved an accuracy rate of 96%. This suggests that our proposed annotation workflow can be used in manual data processing to develop annotated training corpus for developing NLP algorithms.
- Open Access
- 25 January 2024
- Published by IOS Press
- Vol. 310, 700-704
Extracting Symptoms of Agitation in Dementia from Free-Text Nursing Notes Using Advanced Natural Language Processing
Nursing staff record observations about older people under their care in free-text nursing notes. These notes contain older people’s care needs, disease symptoms, frequency of symptom occurrence, nursing actions, etc. Therefore, it is vital to develop a technique to uncover important data from these notes. This study developed and evaluated a deep learning and transfer learning-based named entity recognition (NER) model for extracting symptoms of agitation in dementia from the nursing notes. We employed a Clinical BioBERT model for word embedding. Then we applied bidirectional long-short-term memory (BiLSTM) and conditional random field (CRF) models for NER on nursing notes from Australian residential aged care facilities. The proposed NER model achieves satisfactory performance in extracting symptoms of agitation in dementia with a 75% F1 score and 78% accuracy. We will further develop machine learning models to recommend the optimal nursing actions to manage agitation.
- 25 January 2024
- Published by IOS Press
- Vol. 310, 700-704
Open AccessExtracting Symptoms of Agitation in Dementia from Free-Text Nursing Notes Using Advanced Natural Language Processing
Nursing staff record observations about older people under their care in free-text nursing notes. These notes contain older people’s care needs, disease symptoms, frequency of symptom occurrence, nursing actions, etc. Therefore, it is vital to develop a technique to uncover important data from these notes. This study developed and evaluated a deep learning and transfer learning-based named entity recognition (NER) model for extracting symptoms of agitation in dementia from the nursing notes. We employed a Clinical BioBERT model for word embedding. Then we applied bidirectional long-short-term memory (BiLSTM) and conditional random field (CRF) models for NER on nursing notes from Australian residential aged care facilities. The proposed NER model achieves satisfactory performance in extracting symptoms of agitation in dementia with a 75% F1 score and 78% accuracy. We will further develop machine learning models to recommend the optimal nursing actions to manage agitation.
- Open Access
- 13 December 2023
- Published by MDPI AG in Healthcare
- Vol. 11 (24) , 3163
Developing an Instrument to Evaluate the Quality of Dementia Websites
In today’s digital era, health information, especially for conditions like dementia, is crucial. This study aims to develop an instrument, demenTia wEbsite measSurement insTrument (TEST), through four steps: identifying existing instruments, determining criteria, selecting and revising measurement statements, and validating the instrument from March to August 2020. Five health informatics experts used the content validity ratio (CVR) test for validation. Thirteen evaluators compared Fleiss Kappa and intraclass correlation coefficient (ICC) values across four dementia websites using TEST and another tool, DISCERN. TEST consists of seven criteria and 25 measurement statements focusing on content quality (relevance, credibility, currency) and user experience (accessibility, interactivity, attractiveness, privacy). CVR = 1 confirmed all statements as essential. The TEST demonstrated stronger consistency and assessor agreement compared to DISCERN, measured by Fleiss Kappa and ICC. Overall, it is a robust tool for reliable and user-friendly dementia resources, ensuring health holistic information accessibility.
- 13 December 2023
- Published by MDPI AG in Healthcare
- Vol. 11 (24) , 3163
Open AccessDeveloping an Instrument to Evaluate the Quality of Dementia Websites
In today’s digital era, health information, especially for conditions like dementia, is crucial. This study aims to develop an instrument, demenTia wEbsite measSurement insTrument (TEST), through four steps: identifying existing instruments, determining criteria, selecting and revising measurement statements, and validating the instrument from March to August 2020. Five health informatics experts used the content validity ratio (CVR) test for validation. Thirteen evaluators compared Fleiss Kappa and intraclass correlation coefficient (ICC) values across four dementia websites using TEST and another tool, DISCERN. TEST consists of seven criteria and 25 measurement statements focusing on content quality (relevance, credibility, currency) and user experience (accessibility, interactivity, attractiveness, privacy). CVR = 1 confirmed all statements as essential. The TEST demonstrated stronger consistency and assessor agreement compared to DISCERN, measured by Fleiss Kappa and ICC. Overall, it is a robust tool for reliable and user-friendly dementia resources, ensuring health holistic information accessibility.
- Open Access
- 5 November 2023
- Published by Elsevier in IBRO Neuroscience Reports
- Vol. 15, S909
COGNITIVE PREDICTORS OF INDEPENDENCE IN DAILY ACTIVITIES AMONG COMMUNITY-DWELLING OLDER ADULTS
- 5 November 2023
- Published by Elsevier in IBRO Neuroscience Reports
- Vol. 15, S909
Open AccessCOGNITIVE PREDICTORS OF INDEPENDENCE IN DAILY ACTIVITIES AMONG COMMUNITY-DWELLING OLDER ADULTS
- 1 April 2022
- Published by SLACK, Inc. in Journal of Gerontological Nursing
- Vol. 48 (4) , 49-55
Developing an Intuitive Graph Representation of Knowledge for Nonpharmacological Treatment of Psychotic Symptoms in Dementia
Applying person-centered, nonpharmacological interventions to manage psychotic symptoms of dementia is promoted for health care professionals, particularly gerontological nurses, who are responsible for care of older adults in nursing homes. A knowledge graph is a graph consisting of a set of concepts that are linked together by their interrelationship and has been widely used as a formal representation of domain knowledge in health. However, there is lack of a knowledge graph for nonpharmacological treatment of psychotic symptoms in dementia. Therefore, we developed a comprehensive, human- and machine-understandable knowledge graph for this domain, named Dementia-Related Psychotic Symptom Nonpharmacological Treatment Ontology (DRPSNPTO). This graph was built by adopting the established NeOn methodology, a knowledge graph engineering method, to meet the quality standards for biomedical knowledge graphs. This intuitive graph representation of the domain knowledge sets a new direction for visualizing and computerizing gerontological knowledge to facilitate human comprehension and build intelligent aged care information systems. [Journal of Gerontological Nursing, 48(4), 49–55.]
- 1 April 2022
- Published by SLACK, Inc. in Journal of Gerontological Nursing
- Vol. 48 (4) , 49-55
Developing an Intuitive Graph Representation of Knowledge for Nonpharmacological Treatment of Psychotic Symptoms in Dementia
Applying person-centered, nonpharmacological interventions to manage psychotic symptoms of dementia is promoted for health care professionals, particularly gerontological nurses, who are responsible for care of older adults in nursing homes. A knowledge graph is a graph consisting of a set of concepts that are linked together by their interrelationship and has been widely used as a formal representation of domain knowledge in health. However, there is lack of a knowledge graph for nonpharmacological treatment of psychotic symptoms in dementia. Therefore, we developed a comprehensive, human- and machine-understandable knowledge graph for this domain, named Dementia-Related Psychotic Symptom Nonpharmacological Treatment Ontology (DRPSNPTO). This graph was built by adopting the established NeOn methodology, a knowledge graph engineering method, to meet the quality standards for biomedical knowledge graphs. This intuitive graph representation of the domain knowledge sets a new direction for visualizing and computerizing gerontological knowledge to facilitate human comprehension and build intelligent aged care information systems. [Journal of Gerontological Nursing, 48(4), 49–55.]