ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation
Preprint
Abstract
Text-to-image diffusion models excel at generating single-instance scenes but struggle with multi-instance scenarios, often merging or omitting objects. Unlike previous training-free approaches that rely solely on semantic-level guidance without addressing instance individuation, our training-free method, Instance-to-Semantic Attention Control (ISAC), explicitly resolves incomplete instance formation and semantic entanglement through an instance-first modeling approach. This enables ISAC to effectively leverage a hierarchical, tree-structured prompt mechanism, disentangling multiple object instances and individually aligning them with their corresponding semantic labels. Without employing any external models, ISAC achieves up to 52% average multi-class accuracy and 83% average multi-instance accuracy by effectively forming disentangled instances. The code will be made available upon publication.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Make It Count: Text-to-Image Generation with an Accurate Number of Objects2025
- Diffusion Model is Secretly a Training-Free Open Vocabulary Semantic SegmenterIEEE Transactions on Image Processing, 2025
- T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image GenerationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Be Yourself: Bounded Attention for Multi-subject Text-to-Image Generation2024
- InstanceDiffusion: Instance-Level Control for Image Generation2024
- Initno: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization2024
- Training-Free Layout Control with Cross-Attention Guidance2024
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion ModelsACM Transactions on Graphics, 2023
- Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment2023
- Microsoft COCO: Common Objects in Context2014