SCSaeyoun Choi

Saeyoun Choi

*This is an algorithmically generated author record based on publicly available information.

Saeyoun Choi has a total of 10 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Korean Medical Consultation With Open-Weight Large Language Models: Pilot Comparative Evaluation of Retrieval-Augmented Generation With Metadata Filtering (Preprint)

    by Saeyoun Choi, Donghyun Kim, Ji-Hwan Jeon, Minji Kim, Dong Hun Lee, DaeHwan Ahn, Eu Sun Lee, Yoon Ji Kim, Hyun Youk

    This study develops an open-source large language model-based chatbot tailored for Korean health consultations. The chatbot was implemented using the retrieval-augmented generation (RAG) technique alongside metadata filtering to enhance its performance. This study aims to analyze and compare the performance of a RAG-based chatbot with other leading language models in the context of Korean health consultations. A 10.4 GB Korean medical document corpus (487,277 segments) was constructed from official websites of major Korean hospitals, public health sources, and medical textbooks. This study quantitatively compared 5 open-source large language models (Qwen3:4B, Mistral:7B, Llama-3.1:8B, Gpt-Oss:20B, and Gemma3:27B) in 3 configurations: baseline (model only), RAG-only, and RAG with metadata filtering. The RAG system used a specialized Korean embedding model (upskyy/bge-m3-korean) and an Elasticsearch store. Performance was assessed by an emergency medicine specialist using a validation set of 226 questions across 7 common diseases and scoring responses based on accuracy, safety, and helpfulness. The application of RAG alone failed to yield statistically significant performance improvements and, in some cases (Llama 3.1: 8B and Gemma 3: 27B), resulted in decreased scores. However, the combination of RAG with metadata filtering yielded statistically significant (P<.05) performance increases in most models. Notably, the average score for Mistral:7B increased from 3.79, SD 0.08, to 4.10, SD 0.10, and Gpt-Oss:20B increased from 4.43, SD 0.05, to 4.51, SD 0.04, with the latter achieving the highest safety score (4.61, SD 0.03). The Gemma3:27B model, which possessed a high baseline performance (4.42, SD 0.03), was an exception, exhibiting no significant improvement (P=.14) even with filtering. The effectiveness of RAG for specialized domains such as Korean medical consultation is highly dependent on a metadata filtering process that controls the quality of retrieved information; simple information augmentation is insufficient. Furthermore, the benefit of RAG is limited when a model's intrinsic knowledge (eg, Gemma3:27B) already meets or exceeds the quality of the external knowledge base. This finding indicates that performance enhancement strategies must account for both the retrieval mechanism's quality and the model's preexisting capabilities.

  • Open Access

    Implementation and Performance Comparison Study of a Retrieval-Augmented Generation (RAG)-based Chatbot for Korean Medical Consultation (Preprint)

    by Saeyoun Choi, Donghyun Kim, Ji-Hwan Jeon, Minji Kim, Dong Hun Lee, DaeHwan Ahn, Eu Sun Lee, Yoon Ji Kim, Hyun Youk

    BACKGROUND This study focuses on the development of a GPT-4-based chatbot specifically designed for health consultations in Korean. The chatbot is implemented using the Retrieval-Augmented Generation (RAG) technique alongside Metadata Filtering to enhance its performance. OBJECTIVE The objective of this research is to analyze and compare the performance of the RAG-based chatbot against other leading language models in the context of Korean health consultations. METHODS We developed a Korean medical consultation system integrating Retrieval-Augmented Generation (RAG) with GPT-4o. Medical consultation data were collected from open-source Korean medical resources and public health databases, then preprocessed through cleaning, terminology standardization, de-identification, and classification. The chatbot was fine-tuned for Korean medical dialogues and enhanced with context retention, structured conversation flows, and image-based information extraction. For retrieval, a Korean embedding model (upskyy/bge-m3-korean) and Elastic Search were employed to semantically retrieve relevant medical documents. Metadata Filtering and optimized document chunking improved query precision and retrieval efficiency. Model performance was evaluated using expert-created reference answers, with assessments based on medical accuracy, relevance, and clarity, alongside qualitative clinical validation by medical professionals. RESULTS The proposed model achieved the highest scores in accuracy, relevance, consistency, response time, and cultural appropriateness compared to other models. It showed particularly strong performance in accuracy (4.4) and relevance (4.8), significantly outperforming competing models. In clarity, it also led with a score of 4.9. In qualitative evaluation by medical experts, the model received an 89% approval rate, indicating high trust and reliability in medical consultations. A real-time web service was successfully implemented using an NVIDIA A100 GPU and tools such as Streamlit, Langchain, and Langsmith, enabling fast and accurate health consultations for both the general public and healthcare professionals. CONCLUSIONS This study demonstrates that a RAG-based GPT-4 health chatbot with Metadata Filtering improves the accuracy, relevance, and cultural alignment of Korean medical consultations. The model showed superior performance and clinical validity, though limitations remain in complex reasoning and coverage of lifestyle-related topics. Future efforts should enhance metadata extraction, optimize Korean prompts, and broaden the knowledge base to support more comprehensive healthcare applications.