LLMGA: Multimodal Large Language Model based Generation Assistant
Authors:
- Yitong WangYWYitong Wang
Preprint
Abstract
In this paper, we introduce a Multimodal Large Language Model-based Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and proficiency in reasoning, comprehension, and response inherent in Large Language Models (LLMs) to assist users in image generation and editing. Diverging from existing approaches where Multimodal Large Language Models (MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our LLMGA provides a detailed language generation prompt for precise control over SD. This not only augments LLM context understanding but also reduces noise in generation prompts, yields images with more intricate and precise content, and elevates the interpretability of the network. To this end, we curate a comprehensive dataset comprising prompt refinement, similar image generation, inpainting $\&$ outpainting, and visual question answering. Moreover, we propose a two-stage training scheme. In the first stage, we train the MLLM to grasp the properties of image generation and editing, enabling it to generate detailed prompts. In the second stage, we optimize SD to align with the MLLM's generation prompts. Additionally, we propose a reference-based restoration network to alleviate texture, brightness, and contrast disparities between generated and preserved regions during image editing. Extensive results show that LLMGA has promising generative capabilities and can enable wider applications in an interactive manner.
Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- All are Worth Words: A ViT Backbone for Diffusion Models2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- RePaint: Inpainting using Denoising Diffusion Probabilistic Models2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Vector Quantized Diffusion Model for Text-to-Image Synthesis2022
- DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation2022
- Microsoft COCO: Common Objects in Context2014