MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 22535-22545
Abstract
The advent of open-source AI communities has produced a cornucopia of powerful text-guided diffusion models that are trained on various datasets. While few explorations have been conducted on ensembling such models to combine their strengths. In this work, we propose a simple yet effective method called Saliency-aware Noise Blending (SNB) that can empower the fused text-guided diffusion models to achieve more controllable generation. Specifically, we experimentally find that the responses of classifier-free guidance are highly related to the saliency of generated images. Thus we propose to trust different models in their areas of expertise by blending the predicted noises of two diffusion models in a saliency-aware manner. SNB is training-free and can be completed within a DDIM sampling process. Additionally, it can automatically align the semantics of two noise spaces without requiring additional annotations such as masks. Extensive experiments show the impressive effectiveness of SNB in various applications. The project page is available at https://magicfusion.github.io/.Keywords
Funding Information
- National Natural Science Foundation of China
This publication has 23 references indexed in Scilit:
- Compositional Visual Generation with Composable Diffusion Models2022
- Cross-Modal Contrastive Learning for Text-to-Image Generation2021
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis2019
- MirrorGAN: Learning Text-To-Image Generation by Redescription2019
- Object-Driven Text-To-Image Synthesis via Adversarial Training2019
- Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network2018
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Ensemble methods for wind and solar power forecasting—A state-of-the-art reviewRenewable and Sustainable Energy Reviews, 2015
- A large-scale car dataset for fine-grained categorization and verification2015
- Ensemble-based classifiersArtificial Intelligence Review, 2009