Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation

Authors:
  • SLSeung Hyun Lee
    Seung Hyun Lee
  • YLYinxiao Li
    Yinxiao Li
  • JKJunjie Ke
    Junjie Ke
  • IYInnfarn Yoo
    Innfarn Yoo
  • HZHan Zhang
    Han Zhang
  • JYJiahui Yu
    Jiahui Yu
  • QWQifei Wang
    Qifei Wang
  • FDFei Deng
    Fei Deng
  • GEGlenn Entis
    Glenn Entis
  • JHJunfeng He
    Junfeng He
  • GLGang Li
    Gang Li
  • SKSangpil Kim
    Sangpil Kim
Preprint
Publisher website
Google Scholar
Abstract
Recent works have demonstrated that using reinforcement learning (RL) with multiple quality rewards can improve the quality of generated images in text-to-image (T2I) generation. However, manually adjusting reward weights poses challenges and may cause over-optimization in certain metrics. To solve this, we propose Parrot, which addresses the issue through multi-objective optimization and introduces an effective multi-reward optimization strategy to approximate Pareto optimal. Utilizing batch-wise Pareto optimal selection, Parrot automatically identifies the optimal trade-off among different rewards. We use the novel multi-reward optimization algorithm to jointly optimize the T2I model and a prompt expansion network, resulting in significant improvement of image quality and also allow to control the trade-off of different rewards using a reward related prompt during inference. Furthermore, we introduce original prompt-centered guidance at inference time, ensuring fidelity to user input after prompt expansion. Extensive experiments and a user study validate the superiority of Parrot over several baselines across various quality criteria, including aesthetics, human preference, text-image alignment, and image sentiment.