Cambricon-D: Full-Network Differential Acceleration for Diffusion Models
Authors:
- 29 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 903-914
Abstract
Diffusion models have made significant progress in current image generation tasks, thus becoming a prominent area of research. Diffusion models necessitate repetitive iterations on minimally altered input data across timesteps, each timestep requiring the recalculation of the entire model, resulting in a remarkable computational redundancy and substantial hardware expenditures.Performing differential computing on input data seems to be a feasible approach for addressing such computational redundancy and improving hardware efficacy. However, non-linear operations (particularly activation functions) necessitate the merging of deltas (i.e., differential values) with raw inputs repeatedly to ensure computational correctness, leading to significant memory access for loading raw inputs, which fragmentedly blocks the forwarding of deltas throughout the network and undermines performance.To solve this problem, we propose Cambricon-D, a fullnetwork differential computing architecture with concise memory access. While maintaining the computational efficiency brought by differential computing, Cambricon-D employs a sign-mask dataflow, which requires only the loading of 1-bit signs (instead of large bitwidth raw inputs), thereby facilitating the seamless forwarding of deltas and effectively mitigating memory access overheads. Experimental results show that, compared to Diffy, Cambricon-D’s dataflow reduces 66% ~ 82% off-chip memory access. In total, Cambricon-D achieves 1.46× ~ 2.38× speedup over A100 on various diffusion models with different resolutions.Keywords
This publication has 17 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- DeepScaleTool: A Tool for the Accurate Estimation of Technology Scaling in the Deep-Submicron Era2021
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- Group Normalization2018
- Diffy: a Déjà vu-Free Differential Deep Neural Network Accelerator2018
- Energy-Efficient Neural Network Accelerator Based on Outlier-Aware Low-Precision Computation2018
- Rethinking the Inception Architecture for Computer Vision2016
- Ramulator: A Fast and Extensible DRAM SimulatorIEEE Computer Architecture Letters, 2015
- Image Quality Assessment Using the SSIM and the Just Noticeable Difference Paradigm2013
- ImageNet: A large-scale hierarchical image database2009