Zhuoran Song

Diffusion models (DMs) exhibit remarkable generative performance within AI. Despite their outstanding generative capability, DMs still face two major limitations in fast and energy-efficient edge deployment: 1) the large number of parameters in DMs introduces substantial computational overhead during a single inference; 2) The iterative denoising property of DMs requires multiple inference steps, which further amplifies both computational and memory overhead. To alleviate the bottlenecks of DMs described above, we propose an importance-aware dynamic compression framework for hardware-efficient inference of DMs. We design importance metrics for steps, blocks, and layers in DMs based on the observation that not all denoising modules are equally important for a specific input. Guided by importance metrics, we propose a dynamic pruning method to eliminate redundant blocks during iterative denoising, while simultaneously applying binary quantization to the weights and mixed-precision quantization to the activations. To address the accuracy degradation introduced by the compression framework, we further propose tailored supervision losses and novel connection modes to improve the semantic fidelity and perceptual quality of the generated images. Given the binary and mixed-precision operations, along with the dynamic changes in model topology and computation flow due to pruning, we design a dedicated accelerator to more effectively supports our compression framework. Experimental results demonstrate that the proposed framework achieves a maximum pruning acceleration of 1.21× and a quantization bit-width of W2.05A4.59. The hardware implementation achieves 2.3× speedup and 76% energy savings compared to the baseline.

Mixture of Experts (MoE) architectures significantly enhance the capacity of LLMs without proportional increases in computation, but at the cost of a vast parameter size. Offloading MoE expert parameters to host memory and leveraging both CPU and GPU computation has recently emerged as a promising direction to support such models on resourceconstrained local PC platforms. While promising, we notice that existing approaches mismatch the dynamic nature of expert workloads, which leads to three fundamental inefficiencies: (1) Static expert assignment causes severe CPUGPU load imbalance, underutilizing CPU and GPU resources; (2) Existing prefetching techniques fail to accurately predict high-workload experts, leading to costly inaccurate prefetches; (3) GPU cache policies neglect workload dynamics, resulting in poor hit rates and limited effectiveness. To address these challenges, we propose DALI, a workloaDAware offLoadIng framework for efficient MoE inference on local PCs. To fully utilize hardware resources, DALI first dynamically assigns experts to CPU or GPU by modeling assignment as a 0-1 integer optimization problem and solving it efficiently using a Greedy Assignment strategy at runtime. To improve prefetching accuracy, we develop a Residual-Based Prefetching method leveraging inter-layer residual information to accurately predict high-workload experts. Additionally, we introduce a Workload-Aware Cache Replacement policy that exploits temporal correlation in expert activations to improve GPU cache efficiency. By evaluating across various MoE models and settings, DALI achieves significant speedups in the both prefill and decoding phases over the state-of-the-art offloading frameworks.

Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attention computation. While existing sparse attention methods rely on predefined patterns or inaccurate estimations to approximate attention behavior, they often fail to fully capture the true dynamics of attention, resulting in reduced efficiency and compromised accuracy. Instead, we propose a highly accurate sparse attention mechanism that shares similar yet precise attention patterns across heads, enabling a more realistic capture of the dynamic behavior of attention. Our approach is grounded in two key observations: (1) attention patterns demonstrate strong inter-head similarity, and (2) this similarity remains remarkably consistent across diverse inputs. By strategically sharing computed accurate patterns across attention heads, our method effectively captures actual patterns while requiring full attention computation for only a small subset of heads. Comprehensive evaluations demonstrate that our approach achieves superior or comparable speedup relative to state-of-the-art methods while delivering the best overall accuracy.