Dezun Dong

The rapid advancement of deep learning models has led to their widespread application across various domains; however, this has resulted in a significant increase in both hardware and software costs, necessitating optimizations for resource efficiency. To address this issue, performance prediction techniques evaluate the performance impact of corresponding optimizations on deep learning systems to maximize performance gains. This paper presents a comprehensive survey of performance prediction techniques for deep learning systems, categorizing them into allocation, cluster, and accelerator levels based on the system level at which their corresponding optimizations are situated, ranging from high to low. Allocation-level techniques primarily construct the relationships between resource configurations and their performance, cluster-level techniques model the optimizations within deep learning frameworks at given configurations, while accelerator-level techniques assess the efficiency of specific hardware. We point out that prediction techniques at different levels exhibit a hierarchical relationship rather than operating in isolation. High-level prediction techniques can incorporate the results from low-level techniques into their performance models to accommodate more low-level optimizations and improve their prediction accuracy. However, this will slow down the construction and execution speed of performance models, requiring trade-offs for specific use cases. Finally, we discuss future challenges and research directions in performance prediction techniques for deep learning systems.

Vector-wise pruning is widely used to reduce the inference cost of deep neural networks (DNNs), but it turns key operators into sparse convolution (SpConv) and sparse matrix-matrix multiplication (SpMM) kernels that are difficult to optimize on multi-core CPUs. Although prior sparse-kernel acceleration has largely centered on GPUs, CPUs remain a first-order inference platform in CPU-only deployments and in heterogeneous serving when GPUs are scarce or memory-constrained. We present TurboSparse, an approach for accelerating vector-wise sparse DNN kernels on modern multi-core CPUs. TurboSparse (1) eliminates im2col by executing SpConv directly over a DNN-oriented sparse format to reduce data movement and improve reuse, (2) introduces row-merging micro-kernels guided by an analytical SIMD-utilization model to increase vector-lane occupancy, and (3) fuses consecutive sparse kernels when their combined working set fits in private caches to reuse intermediates and cut memory traffic. Across four CPUs (x86 and ARM) and representative pruned DNN workloads, TurboSparse outperforms five expert-tuned CPU libraries, achieving up to 4.05 × speedup for consecutive SpConvs and sustaining 15–80% of each processor’s theoretical peak throughput.

While Large Language Models (LLMs) have demonstrated exceptional performance in code generation, their autoregressive decoding paradigm typically results in low inference efficiency. Speculative decoding has emerged as a solution to speed up inference through its draft–verification mechanism, where multiple draft tokens are generated efficiently and then verified in parallel by LLMs. Current speculative decoding approaches for code generation are broadly categorized into two types: model-based approaches that generate drafts using a lightweight draft model, and retrieval-based approaches that reuse code snippets from external corpora. In this work, we first conduct empirical experiments to analyze the strengths and weaknesses of retrieval-based and model-based drafting. We find that retrieval-based approach is efficient but usually low-quality, while model-based one is high-quality but the drafting time increases with inference steps of the draft model. Inspired by the cache mechanism in computer architecture, we propose a novel framework CacheCoder. In our design, retrieval acts as a high-speed Level-1 (L1) cache for rapid draft acquisition, while the draft model functions as a more capable Level-2 (L2) cache to compensate for L1 misses, enabling us to capitalize on the speed of reuse when possible and rely on the precision of generation when necessary. To combine these two hierarchical processes, we propose two customized strategies, i.e., Preliminary Prediction and Draft Expansion, which utilize the draft model to pre-filter retrieved contents and expand them with tokens generated by the draft model to form drafts with higher length. Experimental results show that CacheCoder achieves up to a 2.64× speedup compared to autoregressive decoding, significantly outperforming state-of-the-art approaches.

Training large-scale graphs with GNNs on multi-GPU platforms faces substantial feature loading overhead, leading to low resource utilization and inefficient training. Overcoming such communication bottlenecks is crucial, and current solutions achieve this by overlapping computation and communication through pipelining. However, current designs often focus only on limited operation scheduling and overlook workload imbalance within the pipeline, restricting their performance optimization in heterogeneous environments. This article introduces LEO , a novel system to accelerate full-graph GNN training on multiple GPUs. The core of LEO lies in introducing hierarchical caching to construct a hybrid-grained software pipeline, enabling efficient computation-communication overlap both within and across GPU kernels. First, LEO introduces a hierarchical cache policy that effectively balances the workload between local and remote operations. Second, LEO tailors a cache-based pipeline and hybrid workload mapping to facilitate operation overlap. Extensive evaluations show that LEO outperforms state-of-the-art full-graph GNN systems in various settings: on averaged 7.50×, 11.90×, and 3.25× faster than DGL, LEO-UVM, and MGG, respectively.

Large language models (LLMs) have achieved remarkable performance in software engineering (SE), and fine-tuning LLM for specific SE tasks has gradually become a new paradigm. However, storing fine-tuned checkpoints for multiple tasks incurs heavy storage and deployment complexity. Model fusion, which operates on fine-tuned parameters, offers excellent parameter compression and scalability, yet its effectiveness in the SE domain remains underexplored, making such an investigation essential for guiding the development of customized fusion techniques for the SE domain. To bridge this gap, we conduct a systematic study of model fusion in the SE contexts and reveal the following major findings: (1) when fusing programming languages (PLs) within the same task, model fusion usually works well and can enhance the performance of PLs with fewer data when PLs share similar features. (2) when fusing SE tasks of the same category within a same PL, all methods except TALL-Masks generally suffer substantial performance degradation on specific tasks; (3) when fusing SE tasks of different categories across different PLs, all existing model fusion methods exhibit significant performance degradation on certain tasks. In our evaluation results, TALL-Masks, which introduces a mask for each task to extract the most relevant dimensions from the fusion parameters, achieves promising performance. However, during parameters fusion, weak features (i.e., small variation in fine-tuned parameters) are easily overshadowed by strong ones (i.e., large variation in fine-tuned parameters) during parameter fusion, causing the constructed masks to fail to extract the most relevant parameters. To overcome this situation, we propose an improved version of TALL-Masks, called Scaling-Masks. The key idea is to amplify weak features to prevent them from being overshadowed by strong ones, which is achieved by scaling the value range of weak features to match that of strong features. Experimental results demonstrate that Scaling-Masks can significantly improve fusion performance for tasks with extremely weak features without affecting other tasks, with normalized accuracy improved by 63.49% for vulnerability detection when fusing SE tasks of different categories and 24.02% for PHP when fusing PLs in the code repair task.

Sparse matrix–matrix multiplication (SpMM) is a fundamental operation in scientific computing with broad applications across numerous domains. Tiling is a key optimization technique for improving data locality and is widely adopted in high-performance computing. However, the irregular data access patterns inherent to SpMM make it challenging to exploit tiling effectively for data reuse. In this article, we propose MaSpMM , a memory-aware SpMM framework that integrates cache-aware tiling with a segment-oriented data layout. MaSpMM stores matrices as continuous segments to enhance data locality within each tile. Moreover, since many sparse matrices in real-world applications exhibit symmetry, we further develop MaSpMM-Sym, an extension that recursively partitions symmetric matrices to eliminate write conflicts and further improve locality. To adapt to diverse scenarios, we finally introduce MaSpMM-Adap, which adaptively selects the most suitable approach for each input matrix. Comprehensive evaluations on both x86 and ARM CPUs demonstrate that MaSpMM-Adap achieves average speedups of up to 1.86× over Intel oneMKL, 1.84× over ASpT, and 1.75× over J-Stream.

NVMe over Fabrics (NVMeoF) is a next-generation storage disaggregation protocol tailored for high-performance data centers utilizing NVMe SSDs and interconnected networks. Its performance advantages stem from leveraging multiple deep queue pairs to enable parallel PDUs transfers. However, as NVMe SSDs and network infrastructure continue to advance, achieving near-ideal performance in NVMeoF systems becomes increasingly challenging. This paper investigates key performance limitations in the native NVMeoF queue design, particularly focusing on issues related to the unified read-write communication pipeline, as well as intra- and inter-queue I/O performance interference. To address these challenges, we propose AEQ, a co-designed NVMeoF queue pair architecture aimed at enhancing the efficiency of data and command transmission for remote storage access. Specifically, we introduce AQ-RI, a switchable queue design that optimizes differentiated read-write request processing; EQ-RI, an end-to-end packet processing system with dedicated I/O pipelines for distinct application types; and AQ-PI, a PDU isolation mechanism that ensures intra-queue I/O isolation. Experimental results show that AEQ significantly enhances high-concurrency performance in NVMeoF storage networks, while also improving both latency and throughput quality of service.

The General Matrix Multiplication (GEMM) is a crucial subprogram in high-performance computing (HPC). With the increasing importance of power and energy consumption, modern Digital Signal Processors (DSPs) are being integrated into general-purpose HPC systems. However, due to architecture disparities, traditional optimizations for CPUs and GPUs are not easily applicable to modern DSPs. This paper shares our experience of optimizing the GEMM operation using a CPU-DSP platform as a case study. Our work employs a set of strategies to improve the performance and scalability of GEMM. These strategies focus on developing micro-kernels based on heterogeneous on-chip memory, addressing the memory access bottleneck in multi-core parallelism, and facilitating efficient transpose-GEMM. These approaches, collectively referred to as an efficient and practical library (a.k.a. mtGEMM), maximize computational capabilities and bandwidth utilization of multi-core DSPs, while achieving high performance for variously-shaped GEMMs. Our experimental results demonstrate that mtGEMM can attain between 92% and 96% of the hardware peak, with the multi-core scalability being almost linear.