Emu Edit: Precise Image Editing via Recognition and Generation Tasks
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8871-8879
Abstract
Instruction-based image editing holds immense potential for a variety of applications, as it enables users to perform any editing operation using a natural language instruction. However, current models in this domain often struggle with accurately executing user instructions. We present Emu Edit, a multitask image editing model which sets state-of-the-art results in instruction-based image editing. To develop Emu Edit we train it to multitask across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks, all of which are formulated as generative tasks. Additionally, to enhance Emu Edit's multitask learning abilities, we provide it with learned task embeddings which guide the generation process towards the correct edit type. Both these elements are essential for Emu Edit's outstanding performance. Furthermore, we show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples. This capability offers a significant advantage in scenarios where high-quality samples are scarce. Lastly, to facilitate a more rigorous and informed assessment of instructable image editing models, we release a new challenging and versatile benchmark that includes seven different image editing tasks. 1Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- InstructPix2Pix: Learning to Follow Image Editing Instructions2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Blended Diffusion for Text-driven Editing of Natural Images2022
- Microsoft COCO: Common Objects in Context2014