When Does Instruction Tuning Work in Biomedical NLP? A Structural, Task-Aware, and Safety–Critical Evaluation of Large Language Models

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Instruction tuning has emerged as an effective mechanism for adapting large language models (LLMs) to biomedical natural language processing (BioNLP), enabling improved task generalization with reduced reliance on extensive domain-specific pretraining. However, empirical evidence shows that its effectiveness varies substantially across biomedical tasks, and the factors governing these variations remain insufficiently understood. In particular, prior work often treats instruction tuning as a uniform adaptation strategy, without systematically examining the roles of task structure, instruction design, and robustness. In this work, we present a comprehensive, structure-aware and task-aware evaluation of instruction tuning for biomedical NLP. We organize biomedical tasks according to output granularity and rigidity, and analyze how instruction structures interact with token-, sentence-, and document-level tasks across multiple model families. To move beyond performance-centric evaluation, we introduce diagnostic measures that capture instruction sensitivity, format violations, and safety-related failure modes. Our results reveal a consistent relationship between output rigidity and instruction tuning gain: tasks with flexible output spaces benefit more from instruction tuning than schema-rigid tasks. We further show that instruction sensitivity varies with task granularity and instruction structure, while robustness and safety considerations expose trade-offs, and that robustness and safety considerations expose trade-offs that are not reflected in standard accuracy metrics alone. Based on these findings, we derive practical design guidelines for instruction construction, model selection, and evaluation in biomedical settings. This study reframes instruction tuning as a structured design problem rather than a one-size-fits-all approach.

This publication has 27 references indexed in Scilit: