Qi Zhu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Qi Zhu has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Pengzhi Chu
Student Innovation Center, Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Weili Zeng
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Weiming Zhao
Student Innovation Center, Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Xiaokang Yang
Shanghai Jiao Tong University
1 shared publication - Yichao Yan
Shanghai Jiao Tong University
1 shared publication - Zhuo Chen
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China
1 shared publication
Latest Publications
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 3568-3577
Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting
Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. However, existing works still face a main challenge, concept overfitting. To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited diversities, i.e, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively. Drawing from the analysis, we propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training diversities, while preserving non-customized knowledge. Remarkably, Infusion achieves this feat with remarkable efficiency, requiring a mere 11KB of trained parameters. Extensive experiments also demonstrate that our approach outperforms state-of-the-art methods in both single and multi-concept customized generation. Project page: https://zwl666666.github.io/infusion/.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 3568-3577
Open AccessInfusion: Preventing Customized Text-to-Image Diffusion from Overfitting
Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. However, existing works still face a main challenge, concept overfitting. To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited diversities, i.e, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively. Drawing from the analysis, we propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training diversities, while preserving non-customized knowledge. Remarkably, Infusion achieves this feat with remarkable efficiency, requiring a mere 11KB of trained parameters. Extensive experiments also demonstrate that our approach outperforms state-of-the-art methods in both single and multi-concept customized generation. Project page: https://zwl666666.github.io/infusion/.