Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping
Preprint
Abstract
While modern text-to-image models excel at prompt-based generation, they often lack the fine-grained control necessary for specific user requirements like spatial layouts or subject appearances. Multi-condition control addresses this, yet its integration into Diffusion Transformers (DiTs) is bottlenecked by the conventional ``concatenate-and-attend'' strategy, which suffers from quadratic computational and memory overhead as the number of conditions scales. Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant. To this end, we propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies. Specifically, Position-Aligned Attention (PAA) linearizes spatial control by enforcing localized patch alignment, while Keyword-Scoped Attention (KSA) prunes irrelevant subject-driven interactions via semantic-aware masking. To facilitate efficient learning, we further introduce a Conditional Sensitivity-Aware Sampling (CSAS) strategy that reweights the training objective towards critical denoising phases, drastically accelerating convergence and enhancing conditional fidelity. Empirically, PKA delivers a 10.0$\times$ inference speedup and a 5.1$\times$ VRAM saving, providing a scalable and resource-friendly solution for high-fidelity multi-conditioned generation.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Diffusion Models for Image Restoration and Enhancement: A Comprehensive SurveyInternational Journal of Computer Vision, 2025
- DreamArtist: Controllable One-Shot Text-to-Image Generation via Positive-Negative AdapterInternational Journal of Computer Vision, 2025
- Fast Sampling Through The Reuse Of Attention Maps In Diffusion ModelsInternational Journal of Computer Vision, 2025
- UniCanvas: Affordance-Aware Unified Real Image Editing via Customized Text-to-Image GenerationInternational Journal of Computer Vision, 2025
- AutoStory: Generating Diverse Storytelling Images with Minimal Human EffortsInternational Journal of Computer Vision, 2024
- DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild2023
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004