Enhanced Facial Realism in Personalized Diffusion Models: A Memory-Optimized DreamBooth Implementation for Consumer Hardware

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Despite significant progress in general-purpose diffusion-based models capable of producing high-quality media, this approach is still too difficult to implement on consumer/gamer hardware. We present here a memory-optimized DreamBooth framework designed for consumer-grade GPUs with 16 GB of VRAM, that allows for end-to-end image personalization and addresses some of the limitations of existing solutions. Our system reduces peak GPU memory from 22 GB (baseline DreamBooth) to 14.2 GB through novel hierarchical memory management, including attention slicing, Variational Autoencoder (VAE) tiling, gradient accumulation, and gradient checkpointing integrated within the Hugging Face Accelerate ecosystem. The framework further incorporates state-of-the-art techniques for preserving facial features and a comprehensive automated quality management system. The result is a complete end-to-end pipeline achieving a peak memory of 14.2 GB, with quantitative performance (LPIPS: 0.139, SSIM: 0.879, identity: 0.852, and FID: 23.1) competitive with methods requiring significantly more hardware resources.
Funding Information
  • Natural Sciences and Engineering Research Council of Canada Discovery Grant (RGPIN-2022-05122)

This publication has 36 references indexed in Scilit: