Li Jiang

Vision Transformers (ViTs) achieve state-of-the-art performance across various vision tasks but pose significant deployment challenges due to their high computational complexity and memory footprint. Post-training quantization (PTQ) offers an efficient, retraining-free compression strategy; however, existing solutions typically enforce a uniform quantization scheme (fixed bit-width and data format) across all layers, constraining the potential accuracy-efficiency trade-off. We propose LOAD-ViT , a novel PTQ framework that leverages both mixed-precision and mixed-data-format quantization to allocate resources at a fine granularity based on layer sensitivity. LOAD-ViT optimizes per-layer configurations via a calibration-based sensitivity metric and a constrained search algorithm, minimizing accuracy loss under strict hardware budgets. To support this scheme, we design a dedicated hardware accelerator featuring heterogeneous compute engines and lightweight format-conversion units, facilitating the efficient execution of mixed formats. Extensive evaluations on various ViT models and benchmarks demonstrate that LOAD-ViT achieves up to 5.3 × speedup and 4.2 × energy savings compared to state-of-the-art quantization accelerators, with negligible accuracy degradation.

DRAM-based Processing-in-Memory (PIM) addresses the “memory wall” by executing computations directly inside main memory. However, memory interleaving and virtual memory limit contiguous data size visible to PIM units, constraining PIM task granularity. Fine-grained PIM tasks incur significant offloading overhead that negates PIM performance benefits. To mitigate this, existing PIM systems drastically isolate PIM memory or disable memory interleaving. These design choices, however, decrease the CPU memory bandwidth and introduce extra data transfer, leading to an additional “system memory wall” that degrades CPU performance and must be resolved to realize PIM’s full potential. In this work, we propose GUMPIM, a PIM system that allows interleaved CPU pages and non-interleaved PIM pages to coexist in a U nitary and M alleable memory space with G uaranteed PIM page allocation. GUMPIM enables zero-copy during PIM task offloading and maintains CPU memory bandwidth while ensuring low PIM offloading overhead. First, we propose a dual-track memory management mechanism consisting of independent page allocation and address translation for CPU and PIM pages. Second, we design GUMPIM interface hardware on PIM-enabled DRAMs to provide a dynamic address mapping for the different data layouts of CPU and PIM pages. Third, we propose a PIM-assisted page migration mechanism that transparently migrates pages while preserving CPU access bandwidth, thereby enabling guaranteed and accelerated PIM page allocation. GUMPIM requires no changes to commodity DRAM standards; all hardware modifications are limited to the DRAM side, ensuring full compatibility with existing CPUs and enabling immediate deployment on current HBMx- and LPDDRx-based PIM platforms. Our results show only <0.1% performance degradation for CPU workloads on GUMPIM, in contrast to the 25.8% degradation on PIM systems with memory interleaving turned off. For PIM workloads, GUMPIM reduces memory allocation and CPU-part computation times by 2.7× and 4.93×, respectively, yielding an end-to-end 2.3× speedup over a state-of-the-art baseline system.

Mixed-precision quantization has shown great potential in reducing memory requirements and improving inference efficiency and has received extensive attention. However, narrowing bit lengths does not always translate into significant efficiency gains for various reasons. Firstly, computation kernels often lack support for mixed-precision operations, limiting their utility in systems that rely on such kernels. Secondly, as DNN computation involves multiple stages, the precision pattern can vary significantly across these stages. This complicates the development of precision-aware optimizations for achieving end-to-end efficiency gains. Ultimately, effective precision-aware optimization necessitates additional support throughout the entire vertical stack. This article proposes Mix-PC, a framework that enables efficient mixed-precision compilation for high-performance DNN inference. On one hand, we delve into the cause of inefficiencies in the current compilation for supporting mixed-precision operators. On the other hand, we propose Mix-PC, a general-purpose compilation framework that leverages adaptive patterns for different numeric precision. It leverages adaptive patterns tailored for different numeric precision and progressive fusion strategies, enabling enhanced compatibility with diverse models. The results show that compared to the SOTA quantization efforts, Mix-PC achieves 1.70×, 1.34×, and 2.13× speedup, respectively, in BERT, ViT, and HuBERT.

The exponential scaling of Large Language Models (LLMs) has exposed a growing mismatch between computational demands and hardware efficiency. Although model compression is essential for mitigating this gap, two bottlenecks fundamentally limit its effectiveness in practice: (1) high-magnitude activation outliers that degrade the accuracy of conventional uniform quantization, and (2) dynamically varying activation sparsity, that is, difficult to exploit with rigid hardware datapaths. Existing index-based schemes, such as mixed-precision or non-uniform quantization, often incur prohibitive energy and area overheads due to the complex decoding logic required at runtime. In this article, we propose NICE, an index-assisted algorithm-hardware co-design framework that achieves high-efficiency DNN acceleration by synergistically exploiting unstructured weight sparsity and centroid-based activation indexing. The core philosophy of NICE is to reformulate computation-intensive Multiply-Accumulate (MAC) operations into a unified table-lookup (LUT) mechanism. Specifically, we design a new hardware primitive that treats static weight sparsity and dynamic activation indices as lookup coordinates, enabling the direct retrieval of pre-computed or quantized results. Building on this, we develop the NICE Architecture, a systolic array-based accelerator that introduces novel dimensions of data-and-index reuse, effectively alleviating the data-movement bottlenecks of traditional architectures. Experiments show that NICE can achieve 51.3% energy reduction and 3.83x speedup on average.

Transformer-based large language models (LLMs) have achieved remarkable success. However, their growing size presents challenges due to the increasing mismatch between model scale and hardware capacity. Model compression techniques have been proposed to address this issue, but existing methods often struggle to cope with the large dynamic range of model parameters, including both activations and weights. Additionally, current outlier-aware encoding schemes introduce complex logic, which limits their effectiveness in both compression ratio and hardware efficiency. To overcome these challenges, we propose SPADE, a co-designed algorithm–architecture solution for post-training quantized INT8 large language models, introducing a hybrid-precision encoding scheme. SPADE employs variable-length data representation guided by metadata, enabling local, value-dependent encoding with only 3.66% hardware area overhead while delivering up to 4.88× speedup and 70.5% energy reduction compared with state-of-the-art encoding-based accelerators. Our key insight is that only a small portion of activations are outliers that require high-precision such as INT8, while the vast majority can be accurately represented with INT4. This allows us to exploit the sparsity of high-significance bits across the data. By assigning an identifier bit to each data element, SPADE dynamically selects between compact INT4 encoding for normal values and higher INT8 encoding for rare outliers, eliminating unnecessary bit-width and reducing data redundancy. Moreover, we analyze the limitations of existing variable-length encoding methods, which typically require serialization and deserialization due to uncertainty in the precision format of loaded data blocks. SPADE integrates precision information directly into its encoding rules, enabling parallel decoding and computation on systolic arrays. This design significantly boosts computational throughput and helps realize the theoretical performance advantages of structured encoding. We evaluate SPADE-based accelerators against state-of-the-art encoding-based accelerators, and our results demonstrate that the SPADE-based accelerator achieves up to 4.88× speedup and 70.5% energy reduction, while maintaining superior model accuracy. The codes are available at https://github.com/jlsbz/SPADE.