Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation

Authors:
  • SLSeung Hyun Lee
    Seung Hyun Lee
  • YLYinxiao Li
    Yinxiao Li
  • JKJunjie Ke
    Junjie Ke
  • IYInnfarn Yoo
    Innfarn Yoo
  • HZHan Zhang
    Han Zhang
  • JYJiahui Yu
    Jiahui Yu
  • QWQifei Wang
    Qifei Wang
  • FDFei Deng
    Fei Deng
  • GEGlenn Entis
    Glenn Entis
  • JHJunfeng He
    Junfeng He
  • GLGang Li
    Gang Li
  • SKSangpil Kim
    Sangpil Kim
Preprint
Publisher website
Google Scholar
Abstract
Recent works demonstrate that using reinforcement learning (RL) with quality rewards can enhance the quality of generated images in text-to-image (T2I) generation. However, a simple aggregation of multiple rewards may cause over-optimization in certain metrics and degradation in others, and it is challenging to manually find the optimal weights. An effective strategy to jointly optimize multiple rewards in RL for T2I generation is highly desirable. This paper introduces Parrot, a novel multi-reward RL framework for T2I generation. Through the use of the batch-wise Pareto optimal selection, Parrot automatically identifies the optimal trade-off among different rewards during the RL optimization of the T2I generation. Additionally, Parrot employs a joint optimization approach for the T2I model and the prompt expansion network, facilitating the generation of quality-aware text prompts, thus further enhancing the final image quality. To counteract the potential catastrophic forgetting of the original user prompt due to prompt expansion, we introduce original prompt centered guidance at inference time, ensuring that the generated image remains faithful to the user input. Extensive experiments and a user study demonstrate that Parrot outperforms several baseline methods across various quality criteria, including aesthetics, human preference, image sentiment, and text-image alignment.