Keehyuck Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Keehyuck Lee has a total of 91 co-authors. Here are the top co-authors by number of shared publications:
- Se Young Jung
Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam 13620, Republic of Korea
17 shared publications - Hee Hwang
Department of Pediatrics, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
14 shared publications - Sooyoung Yoo
Office of eHealth Research and Business, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
13 shared publications - Jong Soo Han
Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
8 shared publications - Ho-Young Lee
Department of Nuclear Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
7 shared publications - Woo Kyung Bae
Department of Family Medicine, Seoul National University Bundang Hospital, Seongnam, Korea
7 shared publications
Latest Publications
- Open Access
- 30 April 2026
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 32 (2) , 145-155
Transferable Migration Framework Derived from a Large-scale Tertiary Hospital EHR System
Objectives: Migrating legacy on-premise electronic health record (EHR) systems in tertiary hospitals to modern cloud-native platforms presents technical and strategic challenges. We aimed to establish an optimized roadmap for transitioning legacy monolithic systems to a microservice architecture-based cloud-native EHR (MCEHR).Methods: We conducted a 3-month strategic assessment and case study based on the modernization requirements of a global healthcare provider. The methodology incorporated semi-structured interviews with key stakeholders, including clinical informatics officers and system architects, to identify critical pain points such as .NET 4.0 end-of-support risks and performance bottlenecks during peak clinical hours. Phased hybrid migration was adopted, analyzing over 20 TB of Oracle-based legacy data and evaluating the technical feasibility of transitioning to .NET 9 and RESTful APIs. To ensure clinical safety, a proof-of-concept (PoC) environment was developed to simulate high-concurrency clinical workloads, emphasizing system resilience and transaction integrity during intensive order-entry periods.Results: The transition to .NET 9 and MCEHR demonstrated 100% transaction integrity across 1,012 complex clinical test cases. Frontend and backend modernization showed high feasibility; however, migration of business logic embedded within legacy Oracle views represented a primary technical bottleneck, necessitating targeted decoupling. The PoC confirmed that RESTful API-based services maintained stable throughput under heavy concurrent loads, significantly reducing the risk of system-induced delays in clinical workflows.Conclusions: Transitioning to an MCEHR architecture is complex but strategically essential. The proposed task force team roadmap outlines staged upgrades incorporating core technology modernization (.NET 9 and RESTful APIs), selective business component migration, and parallel DevOps adoption.
- 30 April 2026
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 32 (2) , 145-155
Open AccessTransferable Migration Framework Derived from a Large-scale Tertiary Hospital EHR System
Objectives: Migrating legacy on-premise electronic health record (EHR) systems in tertiary hospitals to modern cloud-native platforms presents technical and strategic challenges. We aimed to establish an optimized roadmap for transitioning legacy monolithic systems to a microservice architecture-based cloud-native EHR (MCEHR).Methods: We conducted a 3-month strategic assessment and case study based on the modernization requirements of a global healthcare provider. The methodology incorporated semi-structured interviews with key stakeholders, including clinical informatics officers and system architects, to identify critical pain points such as .NET 4.0 end-of-support risks and performance bottlenecks during peak clinical hours. Phased hybrid migration was adopted, analyzing over 20 TB of Oracle-based legacy data and evaluating the technical feasibility of transitioning to .NET 9 and RESTful APIs. To ensure clinical safety, a proof-of-concept (PoC) environment was developed to simulate high-concurrency clinical workloads, emphasizing system resilience and transaction integrity during intensive order-entry periods.Results: The transition to .NET 9 and MCEHR demonstrated 100% transaction integrity across 1,012 complex clinical test cases. Frontend and backend modernization showed high feasibility; however, migration of business logic embedded within legacy Oracle views represented a primary technical bottleneck, necessitating targeted decoupling. The PoC confirmed that RESTful API-based services maintained stable throughput under heavy concurrent loads, significantly reducing the risk of system-induced delays in clinical workflows.Conclusions: Transitioning to an MCEHR architecture is complex but strategically essential. The proposed task force team roadmap outlines staged upgrades incorporating core technology modernization (.NET 9 and RESTful APIs), selective business component migration, and parallel DevOps adoption.
- Open Access
- 17 September 2025
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 4 (1) , e75910
Comparative Diagnostic Performance of a Multimodal Large Language Model Versus a Dedicated Electrocardiogram AI in Detecting Myocardial Infarction From Electrocardiogram Images: Comparative Study
Background: Accurate and timely electrocardiogram (ECG) interpretation is critical for diagnosing myocardial infarction (MI) in emergency settings. Recent advances in multimodal large language models (LLMs), such as ChatGPT (OpenAI) and Gemini (Google DeepMind), have shown promise in clinical interpretation for medical imaging. However, whether these models analyze waveform patterns or simply rely on text cues remains unclear, underscoring the need for direct comparisons with dedicated ECG artificial intelligence (AI) tools. Objective: This study aimed to evaluate the diagnostic performance of ChatGPT and Gemini, a general-purpose LLM, in detecting MI from ECG images and to compare its performance with that of ECG Buddy (ARPI Inc) , a dedicated AI-driven ECG analysis tool. Methods: This retrospective study evaluated and compared AI models for classifying MI using a publicly available 12-lead ECG dataset from Pakistan, categorizing cases into MI-positive (239 images) and MI-negative (689 images). ChatGPT (GPT-4o, version November 20, 2024) and Gemini (Gemini 2.5 pro) were queried with 5 MI confidence options, whereas ECG Buddy for Microsoft Windows analyzed the images based on ST-elevation MI, acute coronary syndrome, and myocardial injury biomarkers. Results: Among 928 ECG recordings (239/928, 25.8% MI-positive), ChatGPT achieved an accuracy of 65.95% (95% CI 62.80‐69.00), area under the curve (AUC) of 57.34% (95% CI 53.44‐61.24), sensitivity of 36.40% (95% CI 30.30‐42.85), and specificity of 76.2% (95% CI 72.84‐79.33). With Gemini 2.5 Pro, accuracy dropped to 29.63% (95% CI 26.71‐32.69), AUC to 51.63% (95% CI 50.22‐53.04), and sensitivity rose to 97.07% (95% CI 94.06‐98.81), but specificity fell sharply to 6.24% (95% CI 4.55‐8.31). However, ECG Buddy reached an accuracy of 96.98% (95% CI 95.67‐97.99), AUC of 98.8% (95% CI 98.3‐99.43), sensitivity of 96.65% (95% CI 93.51‐98.54), and specificity of 97.10% (95% CI 95.55‐98.22). DeLong test confirmed that ECG Buddy significantly outperformed ChatGPT (all P <.001). In a qualitative error analysis of LLMs’ diagnostic explanations, GPT-4o produced fully accurate explanations in only 5% of cases (2/40), was partially accurate in 38% (15/40), and completely inaccurate in 58% (23/40). By contrast, Gemini 2.5 Pro yielded fully accurate explanations in 32% of cases (12/37), was partially accurate in 14% (5/37), and completely inaccurate in 54% (20/37). Conclusions: LLMs, such as ChatGPT and Gemini, underperform relative to specialized tools such as ECG Buddy in ECG image–based MI diagnosis. Further training may improve LLMs; however, domain-specific AI remains essential for clinical accuracy. The high performance of ECG Buddy underscores the importance of specialized models for achieving reliable and robust diagnostic outcomes.
- 17 September 2025
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 4 (1) , e75910
Open AccessComparative Diagnostic Performance of a Multimodal Large Language Model Versus a Dedicated Electrocardiogram AI in Detecting Myocardial Infarction From Electrocardiogram Images: Comparative Study
Background: Accurate and timely electrocardiogram (ECG) interpretation is critical for diagnosing myocardial infarction (MI) in emergency settings. Recent advances in multimodal large language models (LLMs), such as ChatGPT (OpenAI) and Gemini (Google DeepMind), have shown promise in clinical interpretation for medical imaging. However, whether these models analyze waveform patterns or simply rely on text cues remains unclear, underscoring the need for direct comparisons with dedicated ECG artificial intelligence (AI) tools. Objective: This study aimed to evaluate the diagnostic performance of ChatGPT and Gemini, a general-purpose LLM, in detecting MI from ECG images and to compare its performance with that of ECG Buddy (ARPI Inc) , a dedicated AI-driven ECG analysis tool. Methods: This retrospective study evaluated and compared AI models for classifying MI using a publicly available 12-lead ECG dataset from Pakistan, categorizing cases into MI-positive (239 images) and MI-negative (689 images). ChatGPT (GPT-4o, version November 20, 2024) and Gemini (Gemini 2.5 pro) were queried with 5 MI confidence options, whereas ECG Buddy for Microsoft Windows analyzed the images based on ST-elevation MI, acute coronary syndrome, and myocardial injury biomarkers. Results: Among 928 ECG recordings (239/928, 25.8% MI-positive), ChatGPT achieved an accuracy of 65.95% (95% CI 62.80‐69.00), area under the curve (AUC) of 57.34% (95% CI 53.44‐61.24), sensitivity of 36.40% (95% CI 30.30‐42.85), and specificity of 76.2% (95% CI 72.84‐79.33). With Gemini 2.5 Pro, accuracy dropped to 29.63% (95% CI 26.71‐32.69), AUC to 51.63% (95% CI 50.22‐53.04), and sensitivity rose to 97.07% (95% CI 94.06‐98.81), but specificity fell sharply to 6.24% (95% CI 4.55‐8.31). However, ECG Buddy reached an accuracy of 96.98% (95% CI 95.67‐97.99), AUC of 98.8% (95% CI 98.3‐99.43), sensitivity of 96.65% (95% CI 93.51‐98.54), and specificity of 97.10% (95% CI 95.55‐98.22). DeLong test confirmed that ECG Buddy significantly outperformed ChatGPT (all P <.001). In a qualitative error analysis of LLMs’ diagnostic explanations, GPT-4o produced fully accurate explanations in only 5% of cases (2/40), was partially accurate in 38% (15/40), and completely inaccurate in 58% (23/40). By contrast, Gemini 2.5 Pro yielded fully accurate explanations in 32% of cases (12/37), was partially accurate in 14% (5/37), and completely inaccurate in 54% (20/37). Conclusions: LLMs, such as ChatGPT and Gemini, underperform relative to specialized tools such as ECG Buddy in ECG image–based MI diagnosis. Further training may improve LLMs; however, domain-specific AI remains essential for clinical accuracy. The high performance of ECG Buddy underscores the importance of specialized models for achieving reliable and robust diagnostic outcomes.
- Open Access
- 31 July 2025
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 31 (3) , 218-225
Development and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System
Objectives: This study aimed to develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system designed to optimize hospital operations. By leveraging electronic medical record (EMR) manuals, the system seeks to streamline administrative workflows and enhance healthcare delivery. Methods: The system integrated fine-tuned multilingual embedding models (Multilingual-E5-Large and BGE-M3) for indexing and retrieving information from EMR manuals. A dataset comprising 5,931 question-document pairs was constructed through query augmentation and validated by domain experts. Fine-tuning was performed using contrastive learning to enhance semantic understanding, with performance assessed using top-k accuracy metrics. The Solar Mini Chat API was adopted for text generation, prioritizing Korean-language responses and cost efficiency. Results: The fine-tuned models demonstrated marked improvements in retrieval accuracy, with BGE-M3 achieving 97.6% and Multilingual-E5-Large reaching 89.7%. The chatbot achieved high performance, with query latency under 10 ms and robust retrieval precision, effectively addressing operational EMR queries. Key applications included administrative task support and billing process optimization, highlighting its potential to reduce staff workload and enhance healthcare service delivery. Conclusions: The RAG-based chatbot system successfully addressed critical challenges in healthcare administration, improving EMR usability and operational efficiency. Future research should focus on realworld deployment and longitudinal studies to further evaluate its impact on administrative burden reduction and workflow improvement.
- 31 July 2025
- Published by The Korean Society of Medical Informatics in Healthcare Informatics Research
- Vol. 31 (3) , 218-225
Open AccessDevelopment and Evaluation of a Retrieval-Augmented Generation-Based Electronic Medical Record Chatbot System
Objectives: This study aimed to develop and evaluate a retrieval-augmented generation (RAG)-based chatbot system designed to optimize hospital operations. By leveraging electronic medical record (EMR) manuals, the system seeks to streamline administrative workflows and enhance healthcare delivery. Methods: The system integrated fine-tuned multilingual embedding models (Multilingual-E5-Large and BGE-M3) for indexing and retrieving information from EMR manuals. A dataset comprising 5,931 question-document pairs was constructed through query augmentation and validated by domain experts. Fine-tuning was performed using contrastive learning to enhance semantic understanding, with performance assessed using top-k accuracy metrics. The Solar Mini Chat API was adopted for text generation, prioritizing Korean-language responses and cost efficiency. Results: The fine-tuned models demonstrated marked improvements in retrieval accuracy, with BGE-M3 achieving 97.6% and Multilingual-E5-Large reaching 89.7%. The chatbot achieved high performance, with query latency under 10 ms and robust retrieval precision, effectively addressing operational EMR queries. Key applications included administrative task support and billing process optimization, highlighting its potential to reduce staff workload and enhance healthcare service delivery. Conclusions: The RAG-based chatbot system successfully addressed critical challenges in healthcare administration, improving EMR usability and operational efficiency. Future research should focus on realworld deployment and longitudinal studies to further evaluate its impact on administrative burden reduction and workflow improvement.
- Open Access
- 1 May 2025
- Published by SAGE Publications in Digital Health
- Vol. 11
Machine learning-based fatigue classification using heart rate variability and cortisol: A multimodal approach to wearable health monitoring
Objective: Fatigue is a critical indicator in modern health management, and efficient, accurate methods for predicting fatigue levels using wearable devices have garnered increasing attention. Although recent advancements have enabled non-invasive cortisol measurement via wearable sensors, it remains unclear how effectively cortisol, in combination with other physiological biomarkers, predicts fatigue. Therefore, this study aimed to evaluate the effectiveness of a multimodal machine learning model that integrates cortisol levels and heart rate variability (HRV) for fatigue prediction. Methods: Data from 336 participants who completed the Fatigue Severity Scale (FSS) were analyzed. Missing data mechanisms for cortisol were examined, and multivariate imputation by chained equations (MICEs) were applied. A TabNet deep-learning model was used to predict low and high fatigue levels based on HRV and cortisol data. Results: The model using only HRV variables achieved a test AUC of 0.774, whereas the model incorporating both HRV and cortisol levels achieved 0.741, indicating a minimal overall performance difference. Feature importance analysis revealed that, in the cortisol-included model, predictions relied on a limited set of features. When feature selection was applied to this model, a reduced set of variables—age, cortisol, and logarithmic very low frequency—achieved comparable predictive performance (AUC = 0.759) without performance degradation. Conclusion: This study demonstrated that a fatigue prediction model based on cortisol and HRV can maintain significant predictive power with a reduced number of variables. These findings suggest the potential for practical implementation in wearable devices, enabling accurate fatigue monitoring while minimizing sensor count and computational burden.
- 1 May 2025
- Published by SAGE Publications in Digital Health
- Vol. 11
Open AccessMachine learning-based fatigue classification using heart rate variability and cortisol: A multimodal approach to wearable health monitoring
Objective: Fatigue is a critical indicator in modern health management, and efficient, accurate methods for predicting fatigue levels using wearable devices have garnered increasing attention. Although recent advancements have enabled non-invasive cortisol measurement via wearable sensors, it remains unclear how effectively cortisol, in combination with other physiological biomarkers, predicts fatigue. Therefore, this study aimed to evaluate the effectiveness of a multimodal machine learning model that integrates cortisol levels and heart rate variability (HRV) for fatigue prediction. Methods: Data from 336 participants who completed the Fatigue Severity Scale (FSS) were analyzed. Missing data mechanisms for cortisol were examined, and multivariate imputation by chained equations (MICEs) were applied. A TabNet deep-learning model was used to predict low and high fatigue levels based on HRV and cortisol data. Results: The model using only HRV variables achieved a test AUC of 0.774, whereas the model incorporating both HRV and cortisol levels achieved 0.741, indicating a minimal overall performance difference. Feature importance analysis revealed that, in the cortisol-included model, predictions relied on a limited set of features. When feature selection was applied to this model, a reduced set of variables—age, cortisol, and logarithmic very low frequency—achieved comparable predictive performance (AUC = 0.759) without performance degradation. Conclusion: This study demonstrated that a fatigue prediction model based on cortisol and HRV can maintain significant predictive power with a reduced number of variables. These findings suggest the potential for practical implementation in wearable devices, enabling accurate fatigue monitoring while minimizing sensor count and computational burden.
- Open Access
A Comparative Study of Predictive model (ECG Buddy) and ChatGPT-4o for Myocardial Infarction Diagnosis via ECG image Analysis: Performance, Accuracy, and Clinical Feasibility
Background: Accurate and timely electrocardiogram (ECG) interpretation is critical for diagnosing myocardial infarction (MI) in emergency settings. Recent advances in multimodal Large Language Models (LLMs), such as Chat Generative Pre-trained Transformer (ChatGPT), have shown promise in clinical interpretation for medical imaging. However, whether these models analyze waveform patterns or simply rely on text cues remains unclear, underscoring the need for direct comparisons with dedicated ECG artificial intelligence (AI) tools. Methods: This retrospective study evaluated and compared AI models for classifying MI using a publicly available 12-lead ECG dataset from Pakistan, categorizing cases into MI-positive (239 images) and MI-negative (689 images). ChatGPT (GPT-4o, version 2024-11-20) was queried with five MI confidence options, whereas ECG Buddy for Windows analyzed the images based on ST- elevation MI, acute coronary syndrome, and myocardial injury biomarkers. Results: Among 928 ECG recordings (25.8% MI-positive), ChatGPT achieved an accuracy of 65.95% (95% confidence interval [CI]: 62.80–69.00), area under the curve (AUC) of 57.34% (95% CI: 53.44–61.24), sensitivity of 36.40% (95% CI: 30.30–42.85), and specificity of 76.20% (95% CI: 72.84–79.33). However, ECG Buddy reached an accuracy of 96.98% (95% CI: 95.67–97.99), AUC of 98.8% (95% CI: 98.3–99.43), sensitivity of 96.65% (95% CI: 93.51–98.54), and specificity of 97.10% (95% CI: 95.55–98.22). DeLong’s test confirmed that ECG Buddy significantly outperformed ChatGPT (all P < .001). In an error analysis of 40 cases, ChatGPT provided clinically plausible explanations in only 7.5% of cases, whereas 35% were partially correct, 40% were completely incorrect, and 17.5% received no meaningful explanation. Conclusion: LLMs such as ChatGPT underperform relative to specialized tools such as ECG Buddy in ECG image-based MI diagnosis. Further training may improve ChatGPT; however, domain- specific AI remains essential for clinical accuracy. The high performance of ECG Buddy underscores the importance of specialized models for achieving reliable and robust diagnostic outcomes.
Open AccessA Comparative Study of Predictive model (ECG Buddy) and ChatGPT-4o for Myocardial Infarction Diagnosis via ECG image Analysis: Performance, Accuracy, and Clinical Feasibility
Background: Accurate and timely electrocardiogram (ECG) interpretation is critical for diagnosing myocardial infarction (MI) in emergency settings. Recent advances in multimodal Large Language Models (LLMs), such as Chat Generative Pre-trained Transformer (ChatGPT), have shown promise in clinical interpretation for medical imaging. However, whether these models analyze waveform patterns or simply rely on text cues remains unclear, underscoring the need for direct comparisons with dedicated ECG artificial intelligence (AI) tools. Methods: This retrospective study evaluated and compared AI models for classifying MI using a publicly available 12-lead ECG dataset from Pakistan, categorizing cases into MI-positive (239 images) and MI-negative (689 images). ChatGPT (GPT-4o, version 2024-11-20) was queried with five MI confidence options, whereas ECG Buddy for Windows analyzed the images based on ST- elevation MI, acute coronary syndrome, and myocardial injury biomarkers. Results: Among 928 ECG recordings (25.8% MI-positive), ChatGPT achieved an accuracy of 65.95% (95% confidence interval [CI]: 62.80–69.00), area under the curve (AUC) of 57.34% (95% CI: 53.44–61.24), sensitivity of 36.40% (95% CI: 30.30–42.85), and specificity of 76.20% (95% CI: 72.84–79.33). However, ECG Buddy reached an accuracy of 96.98% (95% CI: 95.67–97.99), AUC of 98.8% (95% CI: 98.3–99.43), sensitivity of 96.65% (95% CI: 93.51–98.54), and specificity of 97.10% (95% CI: 95.55–98.22). DeLong’s test confirmed that ECG Buddy significantly outperformed ChatGPT (all P < .001). In an error analysis of 40 cases, ChatGPT provided clinically plausible explanations in only 7.5% of cases, whereas 35% were partially correct, 40% were completely incorrect, and 17.5% received no meaningful explanation. Conclusion: LLMs such as ChatGPT underperform relative to specialized tools such as ECG Buddy in ECG image-based MI diagnosis. Further training may improve ChatGPT; however, domain- specific AI remains essential for clinical accuracy. The high performance of ECG Buddy underscores the importance of specialized models for achieving reliable and robust diagnostic outcomes.
- Open Access
- 20 December 2024
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 12 (1) , e67056
Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study
Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- 20 December 2024
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 12 (1) , e67056
Open AccessAutomated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study
Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- Open Access
- 30 September 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study (Preprint)
BACKGROUND Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. OBJECTIVE This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. METHODS We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual 8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. RESULTS Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. CONCLUSIONS The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- 30 September 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessAutomated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study (Preprint)
BACKGROUND Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. OBJECTIVE This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. METHODS We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual 8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. RESULTS Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. CONCLUSIONS The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types.
- Open Access
- 20 September 2022
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Medicine
- Vol. 43 (5) , 312-318
Impact of Sleep Duration on Decline in Kidney Function in Adult Patients with Hypertension: A Community-Based Prospective Cohort Study
Background: Sleep duration is associated with various health conditions, including chronic kidney disease. However, the association between sleep duration and decline in kidney function in the South Korean population remains unclear. We aimed to investigate the impact of sleep duration on kidney function decline in adult patients with hypertension.Methods: This cohort study was performed using data obtained from the Korean Genome and Epidemiology Study; 2,837 patients with hypertension who initially had normal kidney function were included. Glomerular filtration rates (GFRs) were estimated at baseline and throughout the 16 years of follow-up. A person was considered to have a decline in kidney function if they had a GFR <60 mL/min/1.73 m2. Sleep duration data were obtained through interviewer-assisted questionnaires. Sleep durations were classified as short (<6 hours), normal (≥6 hours but <9 hours), and long (≥9 hours). The Cox proportional hazards model was applied, with adjustments for covariates.Results: After adjusting for covariates, sleep duration was not associated with a decline in kidney function. However, among men with poorly controlled hypertension at baseline, compared to men with normal sleep durations, men with sleep durations <6 hours had a significantly higher risk of kidney function decline (hazard ratio, 1.56; 95% confidence interval, 1.02–2.36).Conclusion: Short sleep duration did not seem to be associated with an increased risk of decline in kidney function; however, it may be a risk factor for the decline in kidney function in men with poorly controlled hypertension.
- 20 September 2022
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Medicine
- Vol. 43 (5) , 312-318
Open AccessImpact of Sleep Duration on Decline in Kidney Function in Adult Patients with Hypertension: A Community-Based Prospective Cohort Study
Background: Sleep duration is associated with various health conditions, including chronic kidney disease. However, the association between sleep duration and decline in kidney function in the South Korean population remains unclear. We aimed to investigate the impact of sleep duration on kidney function decline in adult patients with hypertension.Methods: This cohort study was performed using data obtained from the Korean Genome and Epidemiology Study; 2,837 patients with hypertension who initially had normal kidney function were included. Glomerular filtration rates (GFRs) were estimated at baseline and throughout the 16 years of follow-up. A person was considered to have a decline in kidney function if they had a GFR <60 mL/min/1.73 m2. Sleep duration data were obtained through interviewer-assisted questionnaires. Sleep durations were classified as short (<6 hours), normal (≥6 hours but <9 hours), and long (≥9 hours). The Cox proportional hazards model was applied, with adjustments for covariates.Results: After adjusting for covariates, sleep duration was not associated with a decline in kidney function. However, among men with poorly controlled hypertension at baseline, compared to men with normal sleep durations, men with sleep durations <6 hours had a significantly higher risk of kidney function decline (hazard ratio, 1.56; 95% confidence interval, 1.02–2.36).Conclusion: Short sleep duration did not seem to be associated with an increased risk of decline in kidney function; however, it may be a risk factor for the decline in kidney function in men with poorly controlled hypertension.
- Open Access
- 19 July 2022
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Medicine
- Vol. 43 (4) , 225-230
Changes in Diagnostic Performance of Thyroid Cancer Screening before and after the Korean Thyroid Imaging Reporting and Data System Revision
Background: Since the era of “thyroid cancer epidemic,” many Korean academic societies discouraged the use of ultrasonography in healthy individuals and revised the Korean Thyroid Imaging Reporting and Data System to address the overscreening and overdiagnosis issues. This study aimed to evaluate the change in the diagnostic effectiveness of thyroid cancer screening over the last decade.Methods: This single-center, retrospective observational study analyzed the data of 125,962 thyroid nodules obtained during cancer screening at the health promotion center of Seoul National University Bundang Hospital from 2010 to 2019. Only 327 thyroid cancer cases pathologically confirmed by fine-needle aspiration (FNA) were included in the study. The strength of the association between the number of FNA and (1) the number of thyroid cancer diagnoses, (2) the positive predictive values (PPVs), and (3) the difference in PPV from the previous year were evaluated using Pearson’s correlation analysis.Results: The number of thyroid FNA biopsies as well as the thyroid cancer diagnoses decreased from 2010 to 2019 (166 to 48 [-71.1%] vs. 43 to 22 [-48.8%]). The PPV of FNA biopsies increased from 25.9% to 45.8% (+76.8%) and was negatively correlated with the number of FNA biopsies performed (R=-0.87, P<0.001). The difference in PPV from the previous year increased similarly but without statistical significance (R=-0.59, P=0.09).Conclusion: The diagnostic efficiency of thyroid cancer screening has increased over the last decade, as evidenced by the increasing PPV of FNA biopsies.
- 19 July 2022
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Medicine
- Vol. 43 (4) , 225-230
Open AccessChanges in Diagnostic Performance of Thyroid Cancer Screening before and after the Korean Thyroid Imaging Reporting and Data System Revision
Background: Since the era of “thyroid cancer epidemic,” many Korean academic societies discouraged the use of ultrasonography in healthy individuals and revised the Korean Thyroid Imaging Reporting and Data System to address the overscreening and overdiagnosis issues. This study aimed to evaluate the change in the diagnostic effectiveness of thyroid cancer screening over the last decade.Methods: This single-center, retrospective observational study analyzed the data of 125,962 thyroid nodules obtained during cancer screening at the health promotion center of Seoul National University Bundang Hospital from 2010 to 2019. Only 327 thyroid cancer cases pathologically confirmed by fine-needle aspiration (FNA) were included in the study. The strength of the association between the number of FNA and (1) the number of thyroid cancer diagnoses, (2) the positive predictive values (PPVs), and (3) the difference in PPV from the previous year were evaluated using Pearson’s correlation analysis.Results: The number of thyroid FNA biopsies as well as the thyroid cancer diagnoses decreased from 2010 to 2019 (166 to 48 [-71.1%] vs. 43 to 22 [-48.8%]). The PPV of FNA biopsies increased from 25.9% to 45.8% (+76.8%) and was negatively correlated with the number of FNA biopsies performed (R=-0.87, P<0.001). The difference in PPV from the previous year increased similarly but without statistical significance (R=-0.59, P=0.09).Conclusion: The diagnostic efficiency of thyroid cancer screening has increased over the last decade, as evidenced by the increasing PPV of FNA biopsies.
- Open Access
- 20 October 2021
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Practice
- Vol. 11 (5) , 393-399
Association of Housing Poverty and Depression in Korean Elderly
Minseol Jang, Jung-Ha Park, Ju Young Kim, Kiheon Lee, Woo Kyung Bae, Kee Hyuck Lee, Jong Soo Han, Se Young Jung, Hyejin Lee. Korean J Fam Pract 2021;11:393-9. https://doi.org/10.21215/kjfp.2021.11.5.393
- 20 October 2021
- Published by The Korean Academy of Family Medicine - M2PI in Korean Journal of Family Practice
- Vol. 11 (5) , 393-399
Open AccessAssociation of Housing Poverty and Depression in Korean Elderly
Minseol Jang, Jung-Ha Park, Ju Young Kim, Kiheon Lee, Woo Kyung Bae, Kee Hyuck Lee, Jong Soo Han, Se Young Jung, Hyejin Lee. Korean J Fam Pract 2021;11:393-9. https://doi.org/10.21215/kjfp.2021.11.5.393