LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models
Top Cited Papers
- 20 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 8145-8158
Abstract
Customization generation techniques have significantly advanced the synthesis of specific concepts across varied contexts. Multi-concept customization emerges as the challenging task within this domain. Existing approaches often rely on training a fusion matrix of multiple Low-Rank Adaptations (LoRAs) to merge various concepts into a single image. However, we identify this straightforward method faces two major challenges: 1) concept confusion, where the model struggles to preserve distinct individual characteristics, and 2) concept vanishing, where the model fails to generate the intended subjects. To address these issues, we introduce LoRA-Composer, a training-free framework designed for seamlessly integrating multiple LoRAs, thereby enhancing the harmony among different concepts within generated images. LoRA-Composer addresses concept vanishing through concept injection constraints, enhancing visibility via an expanded cross-attention mechanism. To combat concept confusion, concept isolation constraints are introduced, refining the self-attention computation. Furthermore, we propose two inference techniques to accelerate inference speed without performance degradation and enhance the accuracy of the generated region, respectively. Extensive experiments demonstrate that LoRA-Composer significantly outperforms standard baselines, especially in scenarios without image-based conditions such as canny edge or pose estimation.Keywords
Funding Information
- National Natural Science Foundation of China (62402417)
- National Natural Science Foundation of China (62273302)
- National Natural Science Foundation of China (62036009)
- National Natural Science Foundation of China (61936006)
- Key Research and Development Program of Ningbo (2024Z115)
- Yongjiang Talent Introduction Program (2023A-197-G)
- China Computer Federation (CCF)-Tencent Open Fund (JR2023TEG012)
This publication has 18 references indexed in Scilit:
- LoRACLR: Contrastive Adaptation for Customization of Diffusion Models2025
- OMG: Occlusion-Friendly Personalized Multi-concept Generation in Diffusion Models2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- SVDiff: Compact Parameter Space for Diffusion Fine-Tuning2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Paint by Example: Exemplar-based Image Editing with Diffusion Models2023
- Multi-Concept Customization of Text-to-Image Diffusion2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition2019