HKHyunjae Kim

Hyunjae Kim

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Hyunjae Kim has a total of 90 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers
    Open Access

    Memorization in large language models in medicine prevalence characteristics and implications

    by Anran Li, Lingfei Qian, Mengmeng Du, Yu Yin, Yan Hu, Zihao Sun, Yihang Fu, Hyunjae Kim, Erica Stutz, Xuguang Ai, et al.

    Large Language Models (LLMs) have demonstrated significant potential in medicine, with many studies adapting them through continued pretraining or fine-tuning on medical data. However, a key question remains: to what extent do LLMs memorize medical training data—that is, recall or regenerate content seen during continued pretraining or fine-tuning. In this work, we investigate memorization of LLMs in medicine, assessing its prevalence (frequency), characteristics (what is memorized), volume (how much), and potential downstream impacts. We systematically analyze common adaptation scenarios: (1) continued pretraining on medical corpora, (2) fine-tuning on standard medical benchmarks, and (3) fine-tuning on real-world clinical data, including over 13,000 unique inpatient records from Yale New Haven Health System. The results demonstrate that memorization is prevalent and significantly higher than that in the general domain. Memorization has distinct characteristics during continued pretraining and fine-tuning, and it is persistent: up to 87% of content memorized during continued pretraining remains after fine-tuning. Memorization can be categorized into three types: beneficial (e.g., accurate recall of clinical guidelines), uninformative (e.g., templated language), and harmful (e.g., sensitive clinical content). We offer practical recommendations to facilitate beneficial memorization, minimize uninformative memorization, and mitigate harmful memorization to protect patient privacy and improve medical utility.


  • Open Access

    From AI Agents to AI Scientists in Health: Emerging Landscape, Challenges, and the Path Forward

    by Qingyu Chen, Xin Wang, Rong Zhou, Hyunjae Kim, Shuai Wang, Wenjun Zhao, Tianyu Liu, Irbaz Riaz, Lifang He, Yize Zhao, et al.

    AI Scientists are systems that pursue scientific questions by generating, evaluating, and refining evidence across extended research cycles, and they are beginning to move from proof-of-concept demonstrations toward early translational use. Recent systems have autonomously proposed biomedical hypotheses, designed computational pipelines, and produced findings later validated in wet-lab experiments. Health is both the most compelling and the most demanding domain for this paradigm: compelling because the need for faster, more adaptive scientific inquiry is acute across biomedical discovery, clinical research, healthcare delivery, and population health; demanding because failures do not remain confined to model outputs, but can propagate across scientific trajectories, clinical workflows, and the evidence base itself. We therefore argue that AI Scientists in health should be understood not simply as a new application of generative AI, but as an emerging biomedical engineering challenge. Their central risk is trajectory-level failure, a long-horizon failure mode in which errors self-reinforce across multi-stage reasoning cycles rather than appearing as isolated incorrect outputs, and can become embedded in the scientific record and clinical decisions before they are visible enough to correct. Existing frameworks for large language models and AI agents were not designed to detect or contain it. Progress should therefore be judged not primarily by autonomy, but by whether these systems can be engineered, evaluated, and governed as trustworthy collaborators in health. We characterize the emerging landscape of AI Scientists in health, identify five directions of opportunity and six major challenges, and propose the Workflow-driven, Evidence-grounded, Cross-modal, Accountable, Reproducible, and Embodied (WE CARE) framework as a practical roadmap for engineering, evaluating, and governing them.


  • Open Access

    Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology

    by Ruoyi Jiang, Hyunjae Kim, Zhenyue Qin, Morten Lee, Margaret MacGibeny, Ailish Hanly, Angela Sadlowski, Shanin Chowdhury, Xuguang Ai, Jeffrey Gehlhausen, et al.

    Background: Multimodal large language models (MLLMs) have demonstrated promise on publicly available dermatology benchmarks. However, benchmark performance may not generalize to real-world dermatologic decision-making. Most prior evaluations rely on curated image-only tasks, often using dermoscopic or single-lesion or rash images obtained under standardized conditions. In contrast, real-world dermatology consultations commonly involve photographs captured in primary care or emergency department settings. Consultations are accompanied by brief requests with context written by referring clinicians; cases are reviewed by dermatologists, who triage patients for possible urgent evaluation based on likely differential diagnoses. We evaluated MLLM performance under conditions designed to reflect real-world dermatology consultation workflows.Methods: We evaluated four open-weight MLLMs (InternVL-Chat v1.5, LLaVA-Med v1.5, SkinGPT4 and MedGemma-4B-Instruct) and one commercial MLLM (GPT-4.1) across three publicly available dermatology datasets and a retrospective multi-site hospital-based dermatology consultation cohort comprising 5,811 cases and 46,405 clinical images. The real-world cohort paired clinical photographs with referring clinician consultation notes and dermatologist-authored differential diagnoses. Models were evaluated on two clinically relevant tasks: free-text differential diagnosis generation and severity-based triage of urgent dermatologic conditions. Diagnostic performance was assessed using BERTScore, LLM-based adjudication, and independent review by board-certified or board-eligible dermatologists for clinical intent, factuality, and specificity.Findings: Diagnostic performance was modest on public datasets and declined substantially in the real-world hospital consultation cohort. On public benchmarks, top-3 diagnostic accuracy reached 26.55% for the best open-weight model and 42.25% for GPT-4.1. On real-world consultation cases using images alone, top-3 diagnostic accuracy declined to 1.50-13.35% among open-weight models and 24.65% for GPT-4.1. Incorporating clinical context from referring clinician requests improved performance across all models, increasing top-3 diagnostic accuracy up to 28.75% among open-weight models and 38.93% for GPT-4.1. However, model outputs were highly sensitive to incomplete or erroneous consultation context, leading to substantial diagnostic variability across evaluation settings. For severity-based triage of dermatologic conditions, models achieved moderate sensitivity (generally exceeding 60%), suggesting potential utility for screening but insufficient reliability for clinical deployment overall.Interpretation: Current dermatology MLLMs showed lower performance under real-world consultation conditions than on public benchmark datasets. These findings suggest that limitations in visual grounding and context integration contribute substantially to current MLLM performance limitations. Future evaluation frameworks should prioritize workflow realism and deployment-oriented assessment rather than benchmark performance alone.

  • Top Cited Papers
    Open Access

    Rationale-Guided Retrieval Augmented Generation for Medical Question Answering

    by Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang

    Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.


  • Open Access

    Augmenting biomedical named entity recognition with general-domain resources

    by Yu Yin, Hyunjae Kim, Xiao Xiao, Chih Hsuan Wei, Jaewoo Kang, Zhiyong Lu, Hua Xu, Meng Fang, Qingyu Chen


  • Open Access

    Fine-tuning CLIP Text Encoders with Two-step Paraphrasing

    by Hyunjae Kim, Seunghyun Yoon, Trung Bui, Handong Zhao, Quan Hung Tran, Franck Dernoncourt, Jaewoo Kang

    Hyunjae Kim, Seunghyun Yoon, Trung Bui, Handong Zhao, Quan Tran, Franck Dernoncourt, Jaewoo Kang. Findings of the Association for Computational Linguistics: EACL 2024. 2024.


  • Open Access

    KU-DMIS at MEDIQA-CORR 2024: Exploring the Reasoning Capabilities of Small Language Models in Medical Error Correction

    by Hyeon Hwang, Taewhoo Lee, Hyunjae Kim, Jaewoo Kang

    Hyeon Hwang, Taewhoo Lee, Hyunjae Kim, Jaewoo Kang. Proceedings of the 6th Clinical Natural Language Processing Workshop. 2024.


  • Open Access

    LIQUID: A Framework for List Question Answering Dataset Generation

    by Seongyun Lee, Hyunjae Kim, Jaewoo Kang

    Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.


  • Open Access

    LIQUID: A Framework for List Question Answering Dataset Generation

    by Seongyun Lee, Hyunjae Kim, Jaewoo Kang

    Question answering (QA) models often rely on large-scale training datasets, which necessitates the development of a data generation framework to reduce the cost of manual annotations. Although several recent studies have aimed to generate synthetic questions with single-span answers, no study has been conducted on the creation of list questions with multiple, non-contiguous spans as answers. To address this gap, we propose LIQUID, an automated framework for generating list QA datasets from unlabeled corpora. We first convert a passage from Wikipedia or PubMed into a summary and extract named entities from the summarized text as candidate answers. This allows us to select answers that are semantically correlated in context and is, therefore, suitable for constructing list questions. We then create questions using an off-the-shelf question generator with the extracted entities and original passage. Finally, iterative filtering and answer expansion are performed to ensure the accuracy and completeness of the answers. Using our synthetic data, we significantly improve the performance of the previous best list QA models by exact-match F1 scores of 5.0 on MultiSpanQA, 1.9 on Quoref, and 2.8 averaged across three BioASQ benchmarks.

  • Open Access

    Biomedical relation extraction with knowledge base–refined weak supervision

    by Wonjin Yoon, Sean Yi, Richard Jackson, Hyunjae Kim, Sunkyu Kim, Jaewoo Kang

    Biomedical relation extraction (BioRE) is the task of automatically extracting and classifying relations between two biomedical entities in biomedical literature. Recent advances in BioRE research have largely been powered by supervised learning and large language models (LLMs). However, training of LLMs for BioRE with supervised learning requires human-annotated data, and the annotation process often accompanies challenging and expensive work. As a result, the quantity and coverage of annotated data are limiting factors for BioRE systems. In this paper, we present our system for the BioCreative VII challenge—DrugProt track, a BioRE system that leverages a language model structure and weak supervision. Our system is trained on weakly labelled data and then fine-tuned using human-labelled data. To create the weakly labelled dataset, we combined two approaches. First, we trained a model on the original dataset to predict labels on external literature, which will become a model-labelled dataset. Then, we refined the model-labelled dataset using an external knowledge base. Based on our experiment, our approach using refined weak supervision showed significant performance gain over the model trained using standard human-labelled datasets. Our final model showed outstanding performance at the BioCreative VII challenge, achieving 3rd place (this paper focuses on our participating system in the BioCreative VII challenge). Database URL:http://wonjin.info/biore-yoon-et-al-2022