SOEDiff: Efficient Distillation for Small Object Editing
Open Access
- 10 November 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (11) , 1-19
Abstract
In this article, we delve into a new task known as Small Object Editing (SOE), which focuses on text-based image inpainting within a constrained, small-sized area. Despite the remarkable success have been achieved by current image inpainting approaches, their application to the SOE task generally results in failure cases such as Object Missing, Text-Image Mismatch, and Distortion . These failures stem from the limited use of small-sized objects in training datasets and the down-sampling operations employed by U-Net models, which hinders accurate generation. To overcome these challenges, we introduce a novel training-based approach, SOEDiff, aimed at enhancing the capability of baseline models like StableDiffusion in editing small-sized objects while minimizing training costs. Specifically, our method involves two key components: SO-LoRA , which efficiently fine-tunes low-rank matrices, and Cross-scale score distillation , which leverages high-resolution predictions from the pre-trained teacher diffusion model. Our method presents significant improvements on the test dataset collected from MSCOCO and OpenImage, validating the effectiveness of our proposed method in SOE. In particular, when comparing SOEDiff with SD-I model on the OpenImage-small-val dataset, we observe a 0.99 improvement in CLIP-Score and a reduction of 2.87 in FID.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62202431, 62036009, and 2020YFB1707700)
This publication has 26 references indexed in Scilit:
- LoMOE: Localized Multi-Object Editing via Multi-Diffusion2024
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Blended Latent DiffusionACM Transactions on Graphics, 2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation2022
- Cross-Modal Contrastive Learning for Text-to-Image Generation2021
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis2019
- Large-Scale Interactive Object Segmentation With Human Annotators2019
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Microsoft COCO: Common Objects in Context2014