Improving Medical Visual Reinforcement Fine-Tuning via Perception and Reasoning Augmentation
Authors:
Preprint
Abstract
While recent advances in Reinforcement Fine-Tuning (RFT) have shown that rule-based reward schemes can enable effective post-training for large language models, their extension to cross-modal, vision-centric domains remains largely underexplored. This limitation is especially pronounced in the medical imaging domain, where effective performance requires both robust visual perception and structured reasoning. In this work, we address this gap by proposing VRFT-Aug, a visual reinforcement fine-tuning framework tailored for the medical domain. VRFT-Aug introduces a series of training strategies designed to augment both perception and reasoning, including prior knowledge injection, perception-driven policy refinement, medically informed reward shaping, and behavioral imitation. Together, these methods aim to stabilize and improve the RFT process. Through extensive experiments across multiple medical datasets, we show that our approaches consistently outperform both standard supervised fine-tuning and RFT baselines. Moreover, we provide empirically grounded insights and practical training heuristics that can be generalized to other medical image tasks. We hope this work contributes actionable guidance and fresh inspiration for the ongoing effort to develop reliable, reasoning-capable models for high-stakes medical applications.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Prompt engineering for healthcare: Methodologies and applicationsMeta-Radiology, 2026
- MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning2025
- MedCapsNet: A modified Densenet201 model integrated with capsule network for heel disease detection and classificationHeliyon, 2024
- A prior knowledge-guided distributionally robust optimization-based adversarial training strategy for medical image classificationInformation Sciences, 2024
- A deep-learning-based framework for identifying and localizing multiple abnormalities and assessing cardiomegaly in chest X-rayNature Communications, 2024
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classificationScientific Data, 2023
- CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations2022
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesionsScientific Data, 2018
- Microsoft COCO: Common Objects in Context2014