Donghyoung Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Donghyoung Lee has a total of 15 co-authors. Here are the top co-authors by number of shared publications:
- Sejin Nam
Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
4 shared publications - Keehyuck Lee
Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
3 shared publications - Borham Kim
Office of eHealth Research and Business, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
2 shared publications - Hyeongmin Cho
EzCaretech Research & Development Center, Seoul, Republic of Korea
2 shared publications - Jin-Haeng Chung
Department of Pathology, Seoul National University College of Medicine, Seoul, Korea
2 shared publications - Leonard Sunwoo
Seoul National University Bundang Hospital
2 shared publications
Latest Publications
- Open Access
- 31 July 2025
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 31 (3) , 218-225
Development and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System
Objectives: This study aimed to develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system designed to optimize hospital operations. By leveraging electronic medical record (EMR) manuals, the system seeks to streamline administrative workflows and enhance healthcare delivery. Methods: The system integrated fine-tuned multilingual embedding models (Multilingual-E5-Large and BGE-M3) for indexing and retrieving information from EMR manuals. A dataset comprising 5,931 question-document pairs was constructed through query augmentation and validated by domain experts. Fine-tuning was performed using contrastive learning to enhance semantic understanding, with performance assessed using top-k accuracy metrics. The Solar Mini Chat API was adopted for text generation, prioritizing Korean-language responses and cost efficiency. Results: The fine-tuned models demonstrated marked improvements in retrieval accuracy, with BGE-M3 achieving 97.6% and Multilingual-E5-Large reaching 89.7%. The chatbot achieved high performance, with query latency under 10 ms and robust retrieval precision, effectively addressing operational EMR queries. Key applications included administrative task support and billing process optimization, highlighting its potential to reduce staff workload and enhance healthcare service delivery. Conclusions: The RAG-based chatbot system successfully addressed critical challenges in healthcare administration, improving EMR usability and operational efficiency. Future research should focus on realworld deployment and longitudinal studies to further evaluate its impact on administrative burden reduction and workflow improvement.
- 31 July 2025
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 31 (3) , 218-225
Open AccessDevelopment and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System
Objectives: This study aimed to develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system designed to optimize hospital operations. By leveraging electronic medical record (EMR) manuals, the system seeks to streamline administrative workflows and enhance healthcare delivery. Methods: The system integrated fine-tuned multilingual embedding models (Multilingual-E5-Large and BGE-M3) for indexing and retrieving information from EMR manuals. A dataset comprising 5,931 question-document pairs was constructed through query augmentation and validated by domain experts. Fine-tuning was performed using contrastive learning to enhance semantic understanding, with performance assessed using top-k accuracy metrics. The Solar Mini Chat API was adopted for text generation, prioritizing Korean-language responses and cost efficiency. Results: The fine-tuned models demonstrated marked improvements in retrieval accuracy, with BGE-M3 achieving 97.6% and Multilingual-E5-Large reaching 89.7%. The chatbot achieved high performance, with query latency under 10 ms and robust retrieval precision, effectively addressing operational EMR queries. Key applications included administrative task support and billing process optimization, highlighting its potential to reduce staff workload and enhance healthcare service delivery. Conclusions: The RAG-based chatbot system successfully addressed critical challenges in healthcare administration, improving EMR usability and operational efficiency. Future research should focus on realworld deployment and longitudinal studies to further evaluate its impact on administrative burden reduction and workflow improvement.
- Open Access
- 1 June 2025
- Published by The Society of Korean Medicine in Journal of Korean Medicine
- Vol. 46 (2) , 51-62
Construction of Korean Medicine Knowledge Graph and Development of Prescription Recommendation System based on RippleNet
Objectives: We aim to propose an artificial intelligence system for recommending Korean medicine prescriptions based on the RippleNet algorithm using a knowledge graph.Methods: The Korean medicine knowledge graph was constructed with information on prescriptions, medicinal materials, chemical compounds, gene targets, and modern diseases from the TM-MC database. The RippleNet algorithm was trained with the knowledge graph and the prescription history of diseases from TM-MC. The optimized hyperparameter values of the algorithm were derived using the grid search method. The recommendation system was implemented with the knowledge graph and the parameterized model.Results: The Korean medicine knowledge graph contains 16,977 nodes and 142,924 edges, and the prescription histories of diseases include 2,101 true values and 3,413 false data points. The recommendation algorithm showed the best performance with an AUC of 0.919 when trained with parameters of 4 hops and a ripple set size of 32.Conclusions: In this study, we implemented a system that recommends prescriptions based on artificial intelligence algorithms using the knowledge graph. In the future, we plan to improve the quality of the knowledge graph and explore a variety of modern AI algorithms to enhance recommendations.
- 1 June 2025
- Published by The Society of Korean Medicine in Journal of Korean Medicine
- Vol. 46 (2) , 51-62
Open AccessConstruction of Korean Medicine Knowledge Graph and Development of Prescription Recommendation System based on RippleNet
Objectives: We aim to propose an artificial intelligence system for recommending Korean medicine prescriptions based on the RippleNet algorithm using a knowledge graph.Methods: The Korean medicine knowledge graph was constructed with information on prescriptions, medicinal materials, chemical compounds, gene targets, and modern diseases from the TM-MC database. The RippleNet algorithm was trained with the knowledge graph and the prescription history of diseases from TM-MC. The optimized hyperparameter values of the algorithm were derived using the grid search method. The recommendation system was implemented with the knowledge graph and the parameterized model.Results: The Korean medicine knowledge graph contains 16,977 nodes and 142,924 edges, and the prescription histories of diseases include 2,101 true values and 3,413 false data points. The recommendation algorithm showed the best performance with an AUC of 0.919 when trained with parameters of 4 hops and a ripple set size of 32.Conclusions: In this study, we implemented a system that recommends prescriptions based on artificial intelligence algorithms using the knowledge graph. In the future, we plan to improve the quality of the knowledge graph and explore a variety of modern AI algorithms to enhance recommendations.
- Open Access
- 20 December 2024
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 12 (1) , e67056
Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study
Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- 20 December 2024
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 12 (1) , e67056
Open AccessAutomated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study
Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- Open Access
- 30 September 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study (Preprint)
BACKGROUND Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. OBJECTIVE This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. METHODS We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual 8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. RESULTS Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. CONCLUSIONS The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- 30 September 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessAutomated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study (Preprint)
BACKGROUND Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. OBJECTIVE This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. METHODS We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual 8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. RESULTS Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. CONCLUSIONS The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.