The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models
Preprint
Abstract
Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and OutlookACM Computing Surveys, 2025
- HybridFlow: A Flexible and Efficient RLHF Framework2025
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information Systems, 2025
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models2023
- Med-HALT: Medical Domain Hallucination Test for Large Language Models2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- Exploration in deep reinforcement learning: A surveyInformation Fusion, 2022
- Training Language Models to Follow Instructions with Human Feedback2022
- Diving Deep into Modes of Fact Hallucinations in Dialogue Systems2022