Saeyoun Choi
Saeyoun Choi has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Dong Hun Lee
KOI Healthcare Co., Ltd., Seoul, Republic of Korea
2 shared publications - Donghyun Kim
Department of Industrial and Systems Engineering, Dongguk University, Seoul, Republic of Korea
2 shared publications - Eu Sun Lee
Department of Emergency Medicine, Korea University College of Medicine, Seoul, Republic of Korea
2 shared publications - Hyun Youk
Department of Emergency Medicine, Wonju College of Medicine, Yonsei University, Wonju 26426, Republic of Korea
2 shared publications - Ji-Hwan Jeon
MAIN Corp, 1 Gangwon-daehak-gil, Room 1201 (Bodeum-gwan), Chuncheon City, Gangwon Province, 24341, Republic of Korea, 82 10-9840-2120
2 shared publications - Yoon Ji Kim
Digital Health Laboratory, Yonsei University Wonju College of Medicine, Wonju 26417, Gangwon State, Republic of Korea
2 shared publications
Latest Publications
- Open Access
- 15 May 2025
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e72604
Korean Medical Consultation With Open-Weight Large Language Models: Pilot Comparative Evaluation of Retrieval-Augmented Generation With Metadata Filtering (Preprint)
This study develops an open-source large language model-based chatbot tailored for Korean health consultations. The chatbot was implemented using the retrieval-augmented generation (RAG) technique alongside metadata filtering to enhance its performance. This study aims to analyze and compare the performance of a RAG-based chatbot with other leading language models in the context of Korean health consultations. A 10.4 GB Korean medical document corpus (487,277 segments) was constructed from official websites of major Korean hospitals, public health sources, and medical textbooks. This study quantitatively compared 5 open-source large language models (Qwen3:4B, Mistral:7B, Llama-3.1:8B, Gpt-Oss:20B, and Gemma3:27B) in 3 configurations: baseline (model only), RAG-only, and RAG with metadata filtering. The RAG system used a specialized Korean embedding model (upskyy/bge-m3-korean) and an Elasticsearch store. Performance was assessed by an emergency medicine specialist using a validation set of 226 questions across 7 common diseases and scoring responses based on accuracy, safety, and helpfulness. The application of RAG alone failed to yield statistically significant performance improvements and, in some cases (Llama 3.1: 8B and Gemma 3: 27B), resulted in decreased scores. However, the combination of RAG with metadata filtering yielded statistically significant (P<.05) performance increases in most models. Notably, the average score for Mistral:7B increased from 3.79, SD 0.08, to 4.10, SD 0.10, and Gpt-Oss:20B increased from 4.43, SD 0.05, to 4.51, SD 0.04, with the latter achieving the highest safety score (4.61, SD 0.03). The Gemma3:27B model, which possessed a high baseline performance (4.42, SD 0.03), was an exception, exhibiting no significant improvement (P=.14) even with filtering. The effectiveness of RAG for specialized domains such as Korean medical consultation is highly dependent on a metadata filtering process that controls the quality of retrieved information; simple information augmentation is insufficient. Furthermore, the benefit of RAG is limited when a model's intrinsic knowledge (eg, Gemma3:27B) already meets or exceeds the quality of the external knowledge base. This finding indicates that performance enhancement strategies must account for both the retrieval mechanism's quality and the model's preexisting capabilities.
- 15 May 2025
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e72604
Open AccessKorean Medical Consultation With Open-Weight Large Language Models: Pilot Comparative Evaluation of Retrieval-Augmented Generation With Metadata Filtering (Preprint)
This study develops an open-source large language model-based chatbot tailored for Korean health consultations. The chatbot was implemented using the retrieval-augmented generation (RAG) technique alongside metadata filtering to enhance its performance. This study aims to analyze and compare the performance of a RAG-based chatbot with other leading language models in the context of Korean health consultations. A 10.4 GB Korean medical document corpus (487,277 segments) was constructed from official websites of major Korean hospitals, public health sources, and medical textbooks. This study quantitatively compared 5 open-source large language models (Qwen3:4B, Mistral:7B, Llama-3.1:8B, Gpt-Oss:20B, and Gemma3:27B) in 3 configurations: baseline (model only), RAG-only, and RAG with metadata filtering. The RAG system used a specialized Korean embedding model (upskyy/bge-m3-korean) and an Elasticsearch store. Performance was assessed by an emergency medicine specialist using a validation set of 226 questions across 7 common diseases and scoring responses based on accuracy, safety, and helpfulness. The application of RAG alone failed to yield statistically significant performance improvements and, in some cases (Llama 3.1: 8B and Gemma 3: 27B), resulted in decreased scores. However, the combination of RAG with metadata filtering yielded statistically significant (P<.05) performance increases in most models. Notably, the average score for Mistral:7B increased from 3.79, SD 0.08, to 4.10, SD 0.10, and Gpt-Oss:20B increased from 4.43, SD 0.05, to 4.51, SD 0.04, with the latter achieving the highest safety score (4.61, SD 0.03). The Gemma3:27B model, which possessed a high baseline performance (4.42, SD 0.03), was an exception, exhibiting no significant improvement (P=.14) even with filtering. The effectiveness of RAG for specialized domains such as Korean medical consultation is highly dependent on a metadata filtering process that controls the quality of retrieved information; simple information augmentation is insufficient. Furthermore, the benefit of RAG is limited when a model's intrinsic knowledge (eg, Gemma3:27B) already meets or exceeds the quality of the external knowledge base. This finding indicates that performance enhancement strategies must account for both the retrieval mechanism's quality and the model's preexisting capabilities.
- Open Access
- 15 May 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Implementation and Performance Comparison Study of a Retrieval-Augmented Generation (RAG)-based Chatbot for Korean Medical Consultation (Preprint)
BACKGROUND This study focuses on the development of a GPT-4-based chatbot specifically designed for health consultations in Korean. The chatbot is implemented using the Retrieval-Augmented Generation (RAG) technique alongside Metadata Filtering to enhance its performance. OBJECTIVE The objective of this research is to analyze and compare the performance of the RAG-based chatbot against other leading language models in the context of Korean health consultations. METHODS We developed a Korean medical consultation system integrating Retrieval-Augmented Generation (RAG) with GPT-4o. Medical consultation data were collected from open-source Korean medical resources and public health databases, then preprocessed through cleaning, terminology standardization, de-identification, and classification. The chatbot was fine-tuned for Korean medical dialogues and enhanced with context retention, structured conversation flows, and image-based information extraction. For retrieval, a Korean embedding model (upskyy/bge-m3-korean) and Elastic Search were employed to semantically retrieve relevant medical documents. Metadata Filtering and optimized document chunking improved query precision and retrieval efficiency. Model performance was evaluated using expert-created reference answers, with assessments based on medical accuracy, relevance, and clarity, alongside qualitative clinical validation by medical professionals. RESULTS The proposed model achieved the highest scores in accuracy, relevance, consistency, response time, and cultural appropriateness compared to other models. It showed particularly strong performance in accuracy (4.4) and relevance (4.8), significantly outperforming competing models. In clarity, it also led with a score of 4.9. In qualitative evaluation by medical experts, the model received an 89% approval rate, indicating high trust and reliability in medical consultations. A real-time web service was successfully implemented using an NVIDIA A100 GPU and tools such as Streamlit, Langchain, and Langsmith, enabling fast and accurate health consultations for both the general public and healthcare professionals. CONCLUSIONS This study demonstrates that a RAG-based GPT-4 health chatbot with Metadata Filtering improves the accuracy, relevance, and cultural alignment of Korean medical consultations. The model showed superior performance and clinical validity, though limitations remain in complex reasoning and coverage of lifestyle-related topics. Future efforts should enhance metadata extraction, optimize Korean prompts, and broaden the knowledge base to support more comprehensive healthcare applications.
- 15 May 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessImplementation and Performance Comparison Study of a Retrieval-Augmented Generation (RAG)-based Chatbot for Korean Medical Consultation (Preprint)
BACKGROUND This study focuses on the development of a GPT-4-based chatbot specifically designed for health consultations in Korean. The chatbot is implemented using the Retrieval-Augmented Generation (RAG) technique alongside Metadata Filtering to enhance its performance. OBJECTIVE The objective of this research is to analyze and compare the performance of the RAG-based chatbot against other leading language models in the context of Korean health consultations. METHODS We developed a Korean medical consultation system integrating Retrieval-Augmented Generation (RAG) with GPT-4o. Medical consultation data were collected from open-source Korean medical resources and public health databases, then preprocessed through cleaning, terminology standardization, de-identification, and classification. The chatbot was fine-tuned for Korean medical dialogues and enhanced with context retention, structured conversation flows, and image-based information extraction. For retrieval, a Korean embedding model (upskyy/bge-m3-korean) and Elastic Search were employed to semantically retrieve relevant medical documents. Metadata Filtering and optimized document chunking improved query precision and retrieval efficiency. Model performance was evaluated using expert-created reference answers, with assessments based on medical accuracy, relevance, and clarity, alongside qualitative clinical validation by medical professionals. RESULTS The proposed model achieved the highest scores in accuracy, relevance, consistency, response time, and cultural appropriateness compared to other models. It showed particularly strong performance in accuracy (4.4) and relevance (4.8), significantly outperforming competing models. In clarity, it also led with a score of 4.9. In qualitative evaluation by medical experts, the model received an 89% approval rate, indicating high trust and reliability in medical consultations. A real-time web service was successfully implemented using an NVIDIA A100 GPU and tools such as Streamlit, Langchain, and Langsmith, enabling fast and accurate health consultations for both the general public and healthcare professionals. CONCLUSIONS This study demonstrates that a RAG-based GPT-4 health chatbot with Metadata Filtering improves the accuracy, relevance, and cultural alignment of Korean medical consultations. The model showed superior performance and clinical validity, though limitations remain in complex reasoning and coverage of lifestyle-related topics. Future efforts should enhance metadata extraction, optimize Korean prompts, and broaden the knowledge base to support more comprehensive healthcare applications.