Decoupling Reasoning and Reward: A Modular Approach for Stable Alignment of Small Clinical Language Models
Preprint
Abstract
Deploying language models (LMs) in clinical settings requires navigating competing demands between accuracy, auditability, and on-device efficiency for privacy. While smaller LMs are desirable for this purpose, aligning them with methods like Group Relative Policy Optimization (GRPO) is often hindered by training instability and objective conflicts. Prior work has shown that Chain-of-Thought (CoT) supervision during supervised fine-tuning (SFT) can stabilize GRPO, but existing approaches typically entangle these objectives within a single, monolithic model. In this work, we introduce a modular, adapter-based alignment framework that decouples reasoning supervision and reward tuning into separate, composable parameter-efficient fine-tuning (PEFT) stages using LoRA adapters. We evaluate five alignment configurations on a medically grounded question answering dataset, using Qwen2.5 models from 0.5B to 7B to analyze how alignment stability, factual accuracy, and structural auditability scale with model size. Our findings demonstrate that this modular approach resolves key training instabilities, especially in smaller models, and produces structurally consistent, auditable reasoning without sacrificing accuracy. To support further research, we release, upon publication, (1) our dataset comprising over 100K clinically relevant QA pairs with CoT traces and (2) our multi-stage alignment codebase. We conclude that decoupling reasoning and reward offers a flexible and robust foundation for building privacy-preserving, verifiably aligned clinical LLMs that successfully address the competing demands in clinical AI.Keywords
This publication has 4 references indexed in Scilit:
- Privacy preserving strategies for electronic health records in the era of large language modelsnpj Digital Medicine, 2025
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021
- Transformers: State-of-the-Art Natural Language Processing2020
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018