PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7423-7433
Abstract
Reward finetuning has emerged as a promising approach to aligning foundation models with downstream objectives. Remarkable success has been achieved in the language domain by using reinforcement learning (RL) to maximize rewards that reflect human preference. However, in the vision domain, existing RL-based reward finetuning methods are limited by their instability in large-scale training, rendering them incapable of generalizing to complex, unseen prompts. In this paper, we propose Proximal Reward Difference Prediction (PRDP), enabling stable black-box reward finetuning for diffusion models for the first time on large-scale prompt datasets with over 100K prompts. Our key innovation is the Reward Difference Prediction (RDP) objective that has the same optimal solution as the RL objective while enjoying better training stability. Specifically, the RDP objective is a supervised regression objective that tasks the diffusion model with predicting the reward difference of generated image pairs from their denoising trajectories. We theoretically prove that the diffusion model that obtains perfect reward difference prediction is exactly the maximizer of the RL objective. We further develop an online algorithm with proximal updates to stably optimize the RDP objective. In experiments, we demonstrate that PRDP can match the reward maximization ability of well-established RL-based methods in small-scale training. Furthermore, through large-scale training on text prompts from the Human Preference Dataset v2 and the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a diverse set of complex, unseen prompts whereas RL-based methods completely fail.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference2023
- End-to-End Diffusion Latent Optimization Improves Classifier Guidance2023
- VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining2023
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- 3D Shape Generation and Completion through Point-Voxel Diffusion2021
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement LearningMachine Learning, 1992