In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer
Authors:
Preprint
Abstract
Instruction-based image editing enables robust image modification via natural language prompts, yet current methods face a precision-efficiency tradeoff. Fine-tuning methods demand significant computational resources and large datasets, while training-free techniques struggle with instruction comprehension and edit quality. We resolve this dilemma by leveraging large-scale Diffusion Transformer (DiT)' enhanced generation capacity and native contextual awareness. Our solution introduces three contributions: (1) an in-context editing framework for zero-shot instruction compliance using in-context prompting, avoiding structural changes; (2) a LoRA-MoE hybrid tuning strategy that enhances flexibility with efficient adaptation and dynamic expert routing, without extensive retraining; and (3) an early filter inference-time scaling method using vision-language models (VLMs) to select better initial noise early, improving edit quality. Extensive evaluations demonstrate our method's superiority: it outperforms state-of-the-art approaches while requiring only 0.5% training data and 1% trainable parameters compared to conventional baselines. This work establishes a new paradigm that enables high-precision yet efficient instruction-guided editing. Codes and demos can be found in https://river-zhang.github.io/ICEdit-gh-pages/.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling2025
- InsightEdit: Towards Better Instruction Following for Image Editing2025
- Stable Flow: Vital Layers for Training-Free Image Editing2025
- Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator2025
- A Survey on Mixture of Experts in Large Language ModelsIEEE Transactions on Knowledge and Data Engineering, 2025
- Emu Edit: Precise Image Editing via Recognition and Generation Tasks2024
- VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation2024
- MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing2023
- CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023