Hyunjae Kim
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Hyunjae Kim has a total of 90 co-authors. Here are the top co-authors by number of shared publications:
- Jaewoo Kang
Korea University
11 shared publications Zhiyong Lu
National Library of Medicine, National Institutes of Health, Bethesda, USA
3 shared publicationsHua Xu
2 shared publications- Mujeen Sung
Korea University
2 shared publications Qingyu Chen
2 shared publications- Qingyu Chen
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, USA
2 shared publications
Latest Publications
- Top Cited PapersOpen Access
- 19 June 2026
- Published by Springer Nature in Nature Communications
- Vol. 17 (1)
Memorization in large language models in medicine prevalence characteristics and implications
Large Language Models (LLMs) have demonstrated significant potential in medicine, with many studies adapting them through continued pretraining or fine-tuning on medical data. However, a key question remains: to what extent do LLMs memorize medical training data—that is, recall or regenerate content seen during continued pretraining or fine-tuning. In this work, we investigate memorization of LLMs in medicine, assessing its prevalence (frequency), characteristics (what is memorized), volume (how much), and potential downstream impacts. We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent and significantly higher than that in the general domain. Memorization has distinct characteristics during continued pretraining and fine-tuning, and it is persistent: up to 87% of content memorized during continued pretraining remains after fine-tuning. Memorization can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines), uninformative (e.g., templated language), and harmful (e.g., sensitive clinical content). We offer practical recommendations to facilitate beneficial memorization, minimize uninformative memorization, and mitigate harmful memorization to protect patient privacy and improve medical utility.
- 19 June 2026
- Published by Springer Nature in Nature Communications
- Vol. 17 (1)
Top Cited PapersOpen AccessMemorization in large language models in medicine prevalence characteristics and implications
Large Language Models (LLMs) have demonstrated significant potential in medicine, with many studies adapting them through continued pretraining or fine-tuning on medical data. However, a key question remains: to what extent do LLMs memorize medical training data—that is, recall or regenerate content seen during continued pretraining or fine-tuning. In this work, we investigate memorization of LLMs in medicine, assessing its prevalence (frequency), characteristics (what is memorized), volume (how much), and potential downstream impacts. We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent and significantly higher than that in the general domain. Memorization has distinct characteristics during continued pretraining and fine-tuning, and it is persistent: up to 87% of content memorized during continued pretraining remains after fine-tuning. Memorization can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines), uninformative (e.g., templated language), and harmful (e.g., sensitive clinical content). We offer practical recommendations to facilitate beneficial memorization, minimize uninformative memorization, and mitigate harmful memorization to protect patient privacy and improve medical utility.
- Open Access
- 12 June 2026
- Published by Elsevier in SSRN Electronic Journal
From AI Agents to AI Scientists in Health: Emerging Landscape, Challenges, and the Path Forward
AI Scientists are systems that pursue scientific questions by generating, evaluating, and refining evidence across extended research cycles, and they are beginning to move from proof-of-concept demonstrations toward early translational use. Recent systems have autonomously proposed biomedical hypotheses, designed computational pipelines, and produced findings later validated in wet-lab experiments. Health is both the most compelling and the most demanding domain for this paradigm: compelling because the need for faster, more adaptive scientific inquiry is acute across biomedical discovery, clinical research, healthcare delivery, and population health; demanding because failures do not remain confined to model outputs, but can propagate across scientific trajectories, clinical workflows, and the evidence base itself. We therefore argue that AI Scientists in health should be understood not simply as a new application of generative AI, but as an emerging biomedical engineering challenge. Their central risk is trajectory-level failure, a long-horizon failure mode in which errors self-reinforce across multi-stage reasoning cycles rather than appearing as isolated incorrect outputs, and can become embedded in the scientific record and clinical decisions before they are visible enough to correct. Existing frameworks for large language models and AI agents were not designed to detect or contain it. Progress should therefore be judged not primarily by autonomy, but by whether these systems can be engineered, evaluated, and governed as trustworthy collaborators in health. We characterize the emerging landscape of AI Scientists in health, identify five directions of opportunity and six major challenges, and propose the Workflow-driven, Evidence-grounded, Cross-modal, Accountable, Reproducible, and Embodied (WE CARE) framework as a practical roadmap for engineering, evaluating, and governing them.
- 12 June 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessFrom AI Agents to AI Scientists in Health: Emerging Landscape, Challenges, and the Path Forward
AI Scientists are systems that pursue scientific questions by generating, evaluating, and refining evidence across extended research cycles, and they are beginning to move from proof-of-concept demonstrations toward early translational use. Recent systems have autonomously proposed biomedical hypotheses, designed computational pipelines, and produced findings later validated in wet-lab experiments. Health is both the most compelling and the most demanding domain for this paradigm: compelling because the need for faster, more adaptive scientific inquiry is acute across biomedical discovery, clinical research, healthcare delivery, and population health; demanding because failures do not remain confined to model outputs, but can propagate across scientific trajectories, clinical workflows, and the evidence base itself. We therefore argue that AI Scientists in health should be understood not simply as a new application of generative AI, but as an emerging biomedical engineering challenge. Their central risk is trajectory-level failure, a long-horizon failure mode in which errors self-reinforce across multi-stage reasoning cycles rather than appearing as isolated incorrect outputs, and can become embedded in the scientific record and clinical decisions before they are visible enough to correct. Existing frameworks for large language models and AI agents were not designed to detect or contain it. Progress should therefore be judged not primarily by autonomy, but by whether these systems can be engineered, evaluated, and governed as trustworthy collaborators in health. We characterize the emerging landscape of AI Scientists in health, identify five directions of opportunity and six major challenges, and propose the Workflow-driven, Evidence-grounded, Cross-modal, Accountable, Reproducible, and Embodied (WE CARE) framework as a practical roadmap for engineering, evaluating, and governing them.
- Open Access
- 20 May 2026
- Published by Elsevier in SSRN Electronic Journal
Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology
Background: Multimodal large language models (MLLMs) have demonstrated promise on publicly available dermatology benchmarks. However, benchmark performance may not generalize to real-world dermatologic decision-making. Most prior evaluations rely on curated image-only tasks, often using dermoscopic or single-lesion or rash images obtained under standardized conditions. In contrast, real-world dermatology consultations commonly involve photographs captured in primary care or emergency department settings. Consultations are accompanied by brief requests with context written by referring clinicians; cases are reviewed by dermatologists, who triage patients for possible urgent evaluation based on likely differential diagnoses. We evaluated MLLM performance under conditions designed to reflect real-world dermatology consultation workflows.Methods: We evaluated four open-weight MLLMs (InternVL-Chat v1.5, LLaVA-Med v1.5, SkinGPT4 and MedGemma-4B-Instruct) and one commercial MLLM (GPT-4.1) across three publicly available dermatology datasets and a retrospective multi-site hospital-based dermatology consultation cohort comprising 5,811 cases and 46,405 clinical images. The real-world cohort paired clinical photographs with referring clinician consultation notes and dermatologist-authored differential diagnoses. Models were evaluated on two clinically relevant tasks: free-text differential diagnosis generation and severity-based triage of urgent dermatologic conditions. Diagnostic performance was assessed using BERTScore, LLM-based adjudication, and independent review by board-certified or board-eligible dermatologists for clinical intent, factuality, and specificity.Findings: Diagnostic performance was modest on public datasets and declined substantially in the real-world hospital consultation cohort. On public benchmarks, top-3 diagnostic accuracy reached 26.55% for the best open-weight model and 42.25% for GPT-4.1. On real-world consultation cases using images alone, top-3 diagnostic accuracy declined to 1.50-13.35% among open-weight models and 24.65% for GPT-4.1. Incorporating clinical context from referring clinician requests improved performance across all models, increasing top-3 diagnostic accuracy up to 28.75% among open-weight models and 38.93% for GPT-4.1. However, model outputs were highly sensitive to incomplete or erroneous consultation context, leading to substantial diagnostic variability across evaluation settings. For severity-based triage of dermatologic conditions, models achieved moderate sensitivity (generally exceeding 60%), suggesting potential utility for screening but insufficient reliability for clinical deployment overall.Interpretation: Current dermatology MLLMs showed lower performance under real-world consultation conditions than on public benchmark datasets. These findings suggest that limitations in visual grounding and context integration contribute substantially to current MLLM performance limitations. Future evaluation frameworks should prioritize workflow realism and deployment-oriented assessment rather than benchmark performance alone.
- 20 May 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessAre Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology
Background: Multimodal large language models (MLLMs) have demonstrated promise on publicly available dermatology benchmarks. However, benchmark performance may not generalize to real-world dermatologic decision-making. Most prior evaluations rely on curated image-only tasks, often using dermoscopic or single-lesion or rash images obtained under standardized conditions. In contrast, real-world dermatology consultations commonly involve photographs captured in primary care or emergency department settings. Consultations are accompanied by brief requests with context written by referring clinicians; cases are reviewed by dermatologists, who triage patients for possible urgent evaluation based on likely differential diagnoses. We evaluated MLLM performance under conditions designed to reflect real-world dermatology consultation workflows.Methods: We evaluated four open-weight MLLMs (InternVL-Chat v1.5, LLaVA-Med v1.5, SkinGPT4 and MedGemma-4B-Instruct) and one commercial MLLM (GPT-4.1) across three publicly available dermatology datasets and a retrospective multi-site hospital-based dermatology consultation cohort comprising 5,811 cases and 46,405 clinical images. The real-world cohort paired clinical photographs with referring clinician consultation notes and dermatologist-authored differential diagnoses. Models were evaluated on two clinically relevant tasks: free-text differential diagnosis generation and severity-based triage of urgent dermatologic conditions. Diagnostic performance was assessed using BERTScore, LLM-based adjudication, and independent review by board-certified or board-eligible dermatologists for clinical intent, factuality, and specificity.Findings: Diagnostic performance was modest on public datasets and declined substantially in the real-world hospital consultation cohort. On public benchmarks, top-3 diagnostic accuracy reached 26.55% for the best open-weight model and 42.25% for GPT-4.1. On real-world consultation cases using images alone, top-3 diagnostic accuracy declined to 1.50-13.35% among open-weight models and 24.65% for GPT-4.1. Incorporating clinical context from referring clinician requests improved performance across all models, increasing top-3 diagnostic accuracy up to 28.75% among open-weight models and 38.93% for GPT-4.1. However, model outputs were highly sensitive to incomplete or erroneous consultation context, leading to substantial diagnostic variability across evaluation settings. For severity-based triage of dermatologic conditions, models achieved moderate sensitivity (generally exceeding 60%), suggesting potential utility for screening but insufficient reliability for clinical deployment overall.Interpretation: Current dermatology MLLMs showed lower performance under real-world consultation conditions than on public benchmark datasets. These findings suggest that limitations in visual grounding and context integration contribute substantially to current MLLM performance limitations. Future evaluation frameworks should prioritize workflow realism and deployment-oriented assessment rather than benchmark performance alone.
- Top Cited PapersOpen Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 12739-12753
Rationale-Guided Retrieval Augmented Generation for Medical Question Answering
Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 12739-12753
Top Cited PapersOpen AccessRationale-Guided Retrieval Augmented Generation for Medical Question Answering
Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
- Open Access
- 4 October 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 159, 104731
Augmenting biomedical named entity recognition with general-domain resources
- 4 October 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 159, 104731
Open AccessAugmenting biomedical named entity recognition with general-domain resources
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 2175-2184
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing
Hyunjae Kim, Seunghyun Yoon, Trung Bui, Handong Zhao, Quan Tran, Franck Dernoncourt, Jaewoo Kang. Findings of the Association for Computational Linguistics: EACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 2175-2184
Open AccessFine-tuning CLIP Text Encoders with Two-step Paraphrasing
Hyunjae Kim, Seunghyun Yoon, Trung Bui, Handong Zhao, Quan Tran, Franck Dernoncourt, Jaewoo Kang. Findings of the Association for Computational Linguistics: EACL 2024. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 526-536
KU-DMIS at MEDIQA-CORR 2024: Exploring the Reasoning Capabilities of Small Language Models in Medical Error Correction
Hyeon Hwang, Taewhoo Lee, Hyunjae Kim, Jaewoo Kang. Proceedings of the 6th Clinical Natural Language Processing Workshop. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 526-536
Open AccessKU-DMIS at MEDIQA-CORR 2024: Exploring the Reasoning Capabilities of Small Language Models in Medical Error Correction
Hyeon Hwang, Taewhoo Lee, Hyunjae Kim, Jaewoo Kang. Proceedings of the 6th Clinical Natural Language Processing Workshop. 2024.
- Open Access
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (11) , 13014-13024
LIQUID: A Framework for List Question Answering Dataset Generation
Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (11) , 13014-13024
Open AccessLIQUID: A Framework for List Question Answering Dataset Generation
Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.
- Open Access
LIQUID: A Framework for List Question Answering Dataset Generation
Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.
Open AccessLIQUID: A Framework for List Question Answering Dataset Generation
Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.
- Open Access
- 1 January 2023
- Published by Oxford University Press (OUP) in Database: The Journal of Biological Databases and Curation
- Vol. 2023
Biomedical relation extraction with knowledge base–refined weak supervision
Biomedical relation extraction (BioRE) is the task of automatically extracting and classifying relations between two biomedical entities in biomedical literature. Recent advances in BioRE research have largely been powered by supervised learning and large language models (LLMs). However, training of LLMs for BioRE with supervised learning requires human-annotated data, and the annotation process often accompanies challenging and expensive work. As a result, the quantity and coverage of annotated data are limiting factors for BioRE systems. In this paper, we present our system for the BioCreative VII challenge—DrugProt track, a BioRE system that leverages a language model structure and weak supervision. Our system is trained on weakly labelled data and then fine-tuned using human-labelled data. To create the weakly labelled dataset, we combined two approaches. First, we trained a model on the original dataset to predict labels on external literature, which will become a model-labelled dataset. Then, we refined the model-labelled dataset using an external knowledge base. Based on our experiment, our approach using refined weak supervision showed significant performance gain over the model trained using standard human-labelled datasets. Our final model showed outstanding performance at the BioCreative VII challenge, achieving 3rd place (this paper focuses on our participating system in the BioCreative VII challenge). Database URL:http://wonjin.info/biore-yoon-et-al-2022
- 1 January 2023
- Published by Oxford University Press (OUP) in Database: The Journal of Biological Databases and Curation
- Vol. 2023
Open AccessBiomedical relation extraction with knowledge base–refined weak supervision
Biomedical relation extraction (BioRE) is the task of automatically extracting and classifying relations between two biomedical entities in biomedical literature. Recent advances in BioRE research have largely been powered by supervised learning and large language models (LLMs). However, training of LLMs for BioRE with supervised learning requires human-annotated data, and the annotation process often accompanies challenging and expensive work. As a result, the quantity and coverage of annotated data are limiting factors for BioRE systems. In this paper, we present our system for the BioCreative VII challenge—DrugProt track, a BioRE system that leverages a language model structure and weak supervision. Our system is trained on weakly labelled data and then fine-tuned using human-labelled data. To create the weakly labelled dataset, we combined two approaches. First, we trained a model on the original dataset to predict labels on external literature, which will become a model-labelled dataset. Then, we refined the model-labelled dataset using an external knowledge base. Based on our experiment, our approach using refined weak supervision showed significant performance gain over the model trained using standard human-labelled datasets. Our final model showed outstanding performance at the BioCreative VII challenge, achieving 3rd place (this paper focuses on our participating system in the BioCreative VII challenge). Database URL:http://wonjin.info/biore-yoon-et-al-2022