Aligning Text-to-Image Diffusion Models With Constrained Reinforcement Learning
- 18 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (11) , 9550-9562
Abstract
Reward finetuning has emerged as a powerful technique for aligning diffusion models with specific downstream objectives or user preferences. However, current approaches suffer from a persistent challenge of reward overoptimization, where models exploit imperfect reward feedback at the expense of overall performance. In this work, we identify three key contributors to overoptimization: (1) a granularity mismatch between the multi-step diffusion process and sparse rewards; (2) a loss of plasticity that limits the model’s ability to adapt and generalize; and (3) an overly narrow focus on a single reward objective that neglects complementary performance criteria. Accordingly, we introduce Constrained Diffusion Policy Optimization (CDPO), a novel reinforcement learning framework that addresses reward overoptimization from multiple angles. Firstly, CDPO tackles the granularity mismatch through a temporal policy optimization strategy that delivers step-specific rewards throughout the entire diffusion trajectory, thereby reducing the risk of overfitting to sparse final-step rewards. Then we incorporate a neuron reset strategy that selectively resets overactive neurons in the model, preventing overoptimization induced by plasticity loss. Finally, to avoid overfitting to a narrow reward objective, we integrate constrained reinforcement learning with auxiliary reward objectives serving as explicit constraints, ensuring a balanced optimization across diverse performance metrics.Keywords
Funding Information
- National Key Research and Development Program of China (2023YFC2705700)
- National Natural Science Foundation of China (U23A20318, U2336211, 62276195, 62225113)
- Science and Technology Major Project of Hubei Province (2024BAB046)
- Foundation for Innovative Research Groups of Hubei Province (2024AFA017)
- National Research Foundation Singapore and DSO National Laboratories
- AI Singapore Programme (AISG2-GC-2023-006)
- National Research Foundation Singapore (NRF-P2024-001)
This publication has 11 references indexed in Scilit:
- A Diffusion Model Translator for Efficient Image-to-Image TranslationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Stimulating Diffusion Model for Image Denoising via Adaptive Embedding and EnsemblingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- DifFace: Blind Face Restoration With Diffused Error ContractionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Measurement Guidance in Diffusion Models: Insight from Medical Image SynthesisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- MotionDiffuse: Text-Driven Human Motion Generation With Diffusion ModelIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- AnyFace++: A Unified Framework for Free-style Text-to-Face Synthesis and ManipulationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Fast Graph Generation via Spectral DiffusionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference2023
- Reproducible Scaling Laws for Contrastive Language-Image Learning2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022