SmartEdit: Exploring Complex Instruction-Based Image Editing with Multimodal Large Language Models
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8362-8371
Abstract
Current instruction-based image editing methods, such as InstructPix2Pix, often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder in diffusion models. To rectify this, this paper introduces SmartEdit, a novel approach of instruction-based image editing that leverages Multimodal Large Language Models (MLLMs) to enhance its understanding and reasoning capabilities. However, direct integration of these elements still faces challenges in situations requiring complex reasoning. To mitigate this, we propose a Bidirectional Interaction Module (BIM) that enables comprehensive bidirectional information interactions between the input image and the MLLM output. During training, we initially incorporate perception data to boost the perception and understanding capabilities of diffusion models. Subsequently, we demonstrate that a small amount of complex instruction editing data can effectively stimulate SmartEdit’ s editing capabilities for more complex instructions. We further construct a new evaluation dataset, Reason-Edit, specifically tailored for complex instruction-based image editing. Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.Keywords
All Related Versions
Funding Information
- Young Scientists Fund of the National Natural Science Foundation of China (62106154)
- Natural Science Foundation of Guangdong Province, China (General Program) (2022A1515011524)
- The Chinese University of Hong Kong (Shenzhen)
This publication has 14 references indexed in Scilit:
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- InstructPix2Pix: Learning to Follow Image Editing Instructions2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- GRES: Generalized Referring Expression Segmentation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- End-to-End Object Detection with Transformers2020
- COCO-Stuff: Thing and Stuff Classes in Context2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Modeling Context in Referring Expressions2016
- Vicuna2015