Mitigating Overthinking in Large Reasoning Models via Difficulty-aware Reinforcement Learning
Authors:
Preprint
Abstract
Large Reasoning Models (LRMs) achieve explicit chain-of-thought expansion by imitating deep thinking behaviors of humans, demonstrating excellent performance in complex task scenarios. However, the deep-thinking mode often leads to unnecessarily lengthy reasoning and resource inefficiency when handling simple tasks. This overthinking phenomenon may arise from the generation preference triggered by the reward function during post-training. Existing research attempts to mitigate overthinking from the perspective of prompt design or model training, but generally underestimates the importance of task difficulty awareness, which makes it difficult for LRMs to effectively allocate reasoning resources. In this paper, we propose Difficulty-aware Policy Optimization (DiPO), a reinforcement learning-based LRM training framework. DiPO encourages LRM to spontaneously model task complexity, and integrates them into reinforcement learning framework to adjust the generation preferences introduced by post-training. A difficulty modeling method based on model self-reasoning is proposed, which significantly reduces the dependence on manual annotation and formalize task complexity. We further develop a difficulty-signal-enhanced reward function that incorporates a penalty for lengthy reasoning while considering reasoning performance and output format. Experimental results indicate that DiPO enables the model to spontaneously adjust inference overhead, significantly reducing redundant tokens without losing performance due to thought compression.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- C3oT: Generating Shorter Chain-of-Thought Without Compromising EffectivenessProceedings of the AAAI Conference on Artificial Intelligence, 2025
- HybridFlow: A Flexible and Efficient RLHF Framework2025
- TokenSkip: Controllable Chain-of-Thought Compression in LLMs2025
- Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching2025
- Self-Training Elicits Concise Reasoning in Large Language Models2025
- SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs2025
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Large Language Models Are Zero-Shot Reasoners2022
- Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models2022