Aligning Few-Step Diffusion Models With Dense Reward Difference Learning
Top Cited Papers
- 1 July 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (7) , 7375-7386
Abstract
Few-step diffusion models enable efficient high-resolution image synthesis but struggle to align with specific downstream objectives due to limitations of existing reinforcement learning (RL) methods in low-step regimes with limited state spaces and suboptimal sample quality. To address this, we propose Stepwise Diffusion Policy Optimization (SDPO), a novel RL framework tailored for few-step diffusion models. SDPO introduces a dual-state trajectory sampling mechanism, tracking both noisy and predicted clean states at each step to provide dense reward feedback and enable low-variance, mixed-step optimization. For further efficiency, we develop a latent similarity-based dense reward prediction strategy to minimize costly dense reward queries. Leveraging these dense rewards, SDPO optimizes a dense reward difference learning objective that enables more frequent and granular policy updates. Additional refinements, including stepwise advantage estimates, temporal importance weighting, and step-shuffled gradient updates, further enhance long-term dependency, low-step priority, and gradient stability. Our experiments demonstrate that SDPO consistently delivers superior reward-aligned results across diverse few-step settings and tasks.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62576364)
- Shenzhen Basic Research (JCYJ20241202124430041)
- CCF-Tencent Rhino-Bird Open Research Fund (CCF-Tencent RAGR20250114)
- CCF-Tencent Rhino-Bird Open Research Fund (JR2025TEG002)
- NTU RSR
- National Natural Science Foundation of China (62225113)
- National Natural Science Foundation of China (U23A20318)
- National Natural Science Foundation of China (62276195)
- Science and Technology Major Project of Hubei Province (2025BCB026)
- Science and Technology Major Project of Hubei Province (2024BAB046)
- Foundation for Innovative Research Groups of Hubei Province (2024AFA017)
This publication has 27 references indexed in Scilit:
- Large-Scale Reinforcement Learning for Diffusion Models2024
- Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models2024
- Adversarial Diffusion Distillation2024
- Parrot: Pareto-Optimal Multi-reward Reinforcement Learning Framework for Text-to-Image Generation2024
- PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models2024
- One-Step Diffusion with Distribution Matching Distillation2024
- On Distillation of Guided Diffusion Models2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Simple statistical gradient-following algorithms for connectionist reinforcement learningMachine Learning, 1992
- Maximum entropy samplingJournal of Applied Statistics, 1987