EdgeDiff: Energy-Efficient Multi-Modal Few-Step Diffusion Model Accelerator Using Mixed-Precision and Reordered Group Quantization
- 26 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 61 (1) , 168-182
Abstract
Recent advances in diffusion models (DMs)—such as few-step denoising and multi-modal conditioning—have significantly improved computational efficiency and functional flexibility, but they also introduce new hardware challenges. In particular, the elimination of inter-timestep redundancy, increased encoder/decoder workload, and heightened sensitivity to quantization demand a new class of accelerator. We present EdgeDiff, the first processor to support end-to-end, few-step, and multi-modal DM inference. EdgeDiff introduces a unified solution named condition-aware reordered group mixed precision (CRMP) with several novel microarchitectures: compress-and-add (CAA) processing elements (PEs) with bit-shuffle trees (BSTs) for efficient low-bit multiply-accumulate (MAC), a tiered accumulation unit (TAU) to reduce floating-point (FP) accumulation energy, and a grid-based quantization unit (GQU) to eliminate expensive FP division. Fabricated in 28-nm CMOS, EdgeDiff achieves up to 34.4-TOPS/W energy efficiency and reduces generation energy to 418.4 mJ/image for one-step text-to-image (T2I) generation— $3.3\times $ lower than prior state of the art. Despite aggressive quantization, EdgeDiff maintains output quality comparable to FP inference across Fréchet Inception Distance (FID), contrastive language–image pretraining (CLIP), and peak signal-to-noise ratio (PSNR) metrics, establishing it as a compelling solution for energy-efficient, real-time generative artificial intelligence (AI) on edge platforms.Keywords
Funding Information
- Institute of Information and Communications Technology Planning and Evaluation
- Korea Government (MSIT), PIM Semiconductor Design Research Center (2025-0-01170)
This publication has 20 references indexed in Scilit:
- A 28nm 4.96 TOPS/W End-to-End Diffusion Accelerator with Reconfigurable Hyper-Precision and Unified Non-Matrix Processing Engine2024
- LUTein: Dense-Sparse Bit-Slice Architecture With Radix-4 LUT-Based Slice-Tensor Processing Units2024
- 20.2 A 28nm 74.34TFLOPS/W BF16 Heterogenous CIM-Based Accelerator Exploiting Denoising-Similarity for Diffusion Models2024
- 22.9 A 12nm 18.1TFLOPs/W Sparse Transformer Processor with Entropy-Based Early Exit, Mixed-Precision Predication and Fine-Grained Power Management2023
- Sibia: Signed Bit-slice Architecture for Dense DNN Acceleration with Slice-level Sparsity Exploitation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- A 28nm 27.5TOPS/W Approximate-Computing-Based Transformer Processor with Asymptotic Sparsity Speculating and Out-of-Order Computing2022
- 7.7 LNPU: A 25.3TFLOPS/W Sparse Deep-Neural-Network Learning Processor with Fine-Grained Mixed Precision of FP8-FP162019
- Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network2018
- Microsoft COCO: Common Objects in Context2014