Jingyu Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jingyu Wang has a total of 45 co-authors. Here are the top co-authors by number of shared publications:
- Yongpan Liu
Tsinghua University
12 shared publications - Huazhong Yang
Tsinghua University
11 shared publications - Jinshan Yue
Institute of Microelectronics of the Chinese Academy of Sciences, Beijing, China
7 shared publications Yifan He
Department of Electronic Engineering, Tsinghua University, Beijing, China
7 shared publicationsZhe Yuan
Department of Electronic Engineering, Tsinghua University, Beijing, China
5 shared publications- Lu Zhang
Department of Computer Science, Northwestern Polytechnical University, Xian, China
4 shared publications
Latest Publications
- 26 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (10) , 2916-2929
RE-Specter: Examining the Architectural Features of Configurable CNN With Power Side-Channel
As domain-specific training data is recognized as valuable intellectual property, acquiring well-trained weights in convolutional neural networks (CNN) has emerged as a new threat to the neural network design community. To design a CNN accelerator that is resilient to side-channel threats, it is crucial to have an accurate and efficient security-driven framework at the early design stage. However, there is no standard way to perform root-cause analysis on the power side channel that exists in FPGA-based CNN accelerators. Therefore, we build RE-Specter, a framework that facilitates security-driven design space exploration (DSE) across various building components, combination patterns, and parallelism configurations in CNNs. The goal is to fully understand the power side-channel effects resulting from architectural modifications or optimization decisions. We further compare the benchmarks considering precision, resource utilization, and power side-channel leakage. Finally, we experimentally explore the design space of various architectural features. The experimental results show that low-bit precision delivers more secure architectures ($68.9\times $ among DSPs,$2439\times $ among LUTs) in measurement-to-disclosure (MTD), but mixed-precision strategies are necessary to maintain the model accuracy. For loop optimization, in 16-parallel scenario, accumulator-based architecture outperforms the architecture featuring an adder tree with the improvements of$8.28\times $ in MTD and$1.38\times $ in PST.- 26 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (10) , 2916-2929
RE-Specter: Examining the Architectural Features of Configurable CNN With Power Side-Channel
As domain-specific training data is recognized as valuable intellectual property, acquiring well-trained weights in convolutional neural networks (CNN) has emerged as a new threat to the neural network design community. To design a CNN accelerator that is resilient to side-channel threats, it is crucial to have an accurate and efficient security-driven framework at the early design stage. However, there is no standard way to perform root-cause analysis on the power side channel that exists in FPGA-based CNN accelerators. Therefore, we build RE-Specter, a framework that facilitates security-driven design space exploration (DSE) across various building components, combination patterns, and parallelism configurations in CNNs. The goal is to fully understand the power side-channel effects resulting from architectural modifications or optimization decisions. We further compare the benchmarks considering precision, resource utilization, and power side-channel leakage. Finally, we experimentally explore the design space of various architectural features. The experimental results show that low-bit precision delivers more secure architectures ($68.9\times $ among DSPs,$2439\times $ among LUTs) in measurement-to-disclosure (MTD), but mixed-precision strategies are necessary to maintain the model accuracy. For loop optimization, in 16-parallel scenario, accumulator-based architecture outperforms the architecture featuring an adder tree with the improvements of$8.28\times $ in MTD and$1.38\times $ in PST.- Open Access
- 22 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of the Solid-State Circuits Society
- Vol. 4, 25-42
A Survey of Computing-in-Memory Processor: From Circuit to Application
The computing-in-memory (CIM) technique is emerging with the evolvement of big data and artificial intelligence (AI) application. The manuscript presents a systematic review of existing CIM works in a bottom-up view from circuit to application. Various types of CIM circuits based on different volatile/non-volatile devices are introduced. The micro CIM architectures are illustrated to support multi-bit precision computation. After that, several types of processor-level CIM chips are analyzed to reveal the system architecture design considerations. The corresponding CIM tool chains and applications beyond AI applications are also introduced. From circuit to application levels, this manuscript analyzes the design trade-offs, remained challenges and possible future design trends at different design hierarchies of CIM processors.
- 22 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Open Journal of the Solid-State Circuits Society
- Vol. 4, 25-42
Open AccessA Survey of Computing-in-Memory Processor: From Circuit to Application
The computing-in-memory (CIM) technique is emerging with the evolvement of big data and artificial intelligence (AI) application. The manuscript presents a systematic review of existing CIM works in a bottom-up view from circuit to application. Various types of CIM circuits based on different volatile/non-volatile devices are introduced. The micro CIM architectures are illustrated to support multi-bit precision computation. After that, several types of processor-level CIM chips are analyzed to reveal the system architecture design considerations. The corresponding CIM tool chains and applications beyond AI applications are also introduced. From circuit to application levels, this manuscript analyzes the design trade-offs, remained challenges and possible future design trends at different design hierarchies of CIM processors.
- 1 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (3) , 892-905
ULSeq-TA: Ultra-Long Sequence Attention Fusion Transformer Accelerator Supporting Grouped Sparse Softmax and Dual-Path Sparse LayerNorm
Transformer networks have been increasingly successful in various fields. The input sequence lengths have become much larger as the algorithm and task complexity develops, which is challenging due to high computational and storage cost. Softmax and LayerNorm are bottleneck nonlinear operators in ultra-long sequence Transformer networks. To improve the efficiency of Softmax, assumption-based and quantization-based Softmax approaches are introduced. However, the sparsity potential to accelerate Softmax itself is not fully discovered. To improve the efficiency of LayerNorm, some works reduce the input size, and some works explore the pipeline. However, the sparsity potential is also not yet explored. To address these challenges, this article presents the ULSeq-TA software–hardware co-design framework. The software includes 1) the grouped sparse Softmax method to leverage the data magnifying characteristic to explore the middle and post-Softmax sparse processing and 2) the dual-path sparse LayerNorm method which explores the dimensional significance for sparse calculation. The hardware includes 1) an attention fusion architecture which reduces the on-chip memory with fused operators; 2) the grouped sparse Softmax core; and 3) the dual-path sparse LayerNorm core. Experiments show that the software achieves$4.45\times $ and$7.59\times $ computation reduction with little output difference for Softmax and LayerNorm, respectively. The hardware architecture supports at most 32768 sequence length with only 186-kB on-chip memory and achieves$1.75\times -1.98\times $ and$3.22\times -4.32\times $ speedups for sparse Softmax core and sparse LayerNorm core with little accuracy loss, respectively.- 1 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (3) , 892-905
ULSeq-TA: Ultra-Long Sequence Attention Fusion Transformer Accelerator Supporting Grouped Sparse Softmax and Dual-Path Sparse LayerNorm
Transformer networks have been increasingly successful in various fields. The input sequence lengths have become much larger as the algorithm and task complexity develops, which is challenging due to high computational and storage cost. Softmax and LayerNorm are bottleneck nonlinear operators in ultra-long sequence Transformer networks. To improve the efficiency of Softmax, assumption-based and quantization-based Softmax approaches are introduced. However, the sparsity potential to accelerate Softmax itself is not fully discovered. To improve the efficiency of LayerNorm, some works reduce the input size, and some works explore the pipeline. However, the sparsity potential is also not yet explored. To address these challenges, this article presents the ULSeq-TA software–hardware co-design framework. The software includes 1) the grouped sparse Softmax method to leverage the data magnifying characteristic to explore the middle and post-Softmax sparse processing and 2) the dual-path sparse LayerNorm method which explores the dimensional significance for sparse calculation. The hardware includes 1) an attention fusion architecture which reduces the on-chip memory with fused operators; 2) the grouped sparse Softmax core; and 3) the dual-path sparse LayerNorm core. Experiments show that the software achieves$4.45\times $ and$7.59\times $ computation reduction with little output difference for Softmax and LayerNorm, respectively. The hardware architecture supports at most 32768 sequence length with only 186-kB on-chip memory and achieves$1.75\times -1.98\times $ and$3.22\times -4.32\times $ speedups for sparse Softmax core and sparse LayerNorm core with little accuracy loss, respectively.- 30 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (5) , 1612-1627
An Energy-Efficient Computing-in-Memory NN Processor With Set-Associate Blockwise Sparsity and Ping-Pong Weight Update
Computing-in-memory (CIM) chips have demonstrated the potential high energy efficiency for low-power neural network (NN) processors. Even with energy-efficient CIM macros, the existing system-level CIM chips still lack deep exploration on sparsity and large models, which prevents a higher system energy efficiency. This work presents a CIM NN processor with more sufficient support of sparsity and higher utilization rate. Three key innovations are proposed. First, a set-associate blockwise sparsity strategy is designed, which simultaneously saves execution time, power, and storage space. Second, a ping-pong weight update mechanism is proposed for a higher utilization rate, enabling simultaneous execution of CIM and write operations. Third, an efficient CIM macro is implemented with adaptive analog-digital converter (ADC) precision for better sparsity utilization and performance-accuracy trade-off. The 65-nm fabricated chip shows 9.5-TOPS/W system energy efficiency at 4-bit precision, with 6.25 $\times $ actual improvement compared with a state-of-the-art CIM chip. Besides, this work supports high CIM execution accuracy on the ImageNet dataset.
- 30 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 59 (5) , 1612-1627
An Energy-Efficient Computing-in-Memory NN Processor With Set-Associate Blockwise Sparsity and Ping-Pong Weight Update
Computing-in-memory (CIM) chips have demonstrated the potential high energy efficiency for low-power neural network (NN) processors. Even with energy-efficient CIM macros, the existing system-level CIM chips still lack deep exploration on sparsity and large models, which prevents a higher system energy efficiency. This work presents a CIM NN processor with more sufficient support of sparsity and higher utilization rate. Three key innovations are proposed. First, a set-associate blockwise sparsity strategy is designed, which simultaneously saves execution time, power, and storage space. Second, a ping-pong weight update mechanism is proposed for a higher utilization rate, enabling simultaneous execution of CIM and write operations. Third, an efficient CIM macro is implemented with adaptive analog-digital converter (ADC) precision for better sparsity utilization and performance-accuracy trade-off. The 65-nm fabricated chip shows 9.5-TOPS/W system energy efficiency at 4-bit precision, with 6.25 $\times $ actual improvement compared with a state-of-the-art CIM chip. Besides, this work supports high CIM execution accuracy on the ImageNet dataset.
- 10 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (3) , 1179-1183
A Weight-Reload-Eliminated Compute-in-Memory Accelerator for 60 fps 4K Super-Resolution
Convolutional neural networks (CNNs) have significantly enhanced the quality of image super-resolution (SR). However, deploying CNN with deeper layers for real-time 60fps and higher resolution SR, e.g. 4K, brings higher performance requirement, incurring huge area and power overhead as well. In this paper, a high performance compute-in-memory (CIM) accelerator with high energy and area efficiency is proposed for deep-layer CNN-based SR (CNNSR). It adopts alternate time division serial (ATS) dataflow to guarantee the high performance, which eliminates frequent and time-consuming weight reload operation of CIM macros. Moreover, considering the extremely high ratio of computation quantity to weight amount in the deep-layer CNNSR scenario, an energy and area efficiency enhanced digital CIM macro with high computation-to-memory ratio (CMR) is proposed. The proposed accelerator achieves 33.3TOPS peak performance, supporting 4K SR at no less than 60fps. It is evaluated under TSMC 28nm process, achieving 13.96TOPS/W energy efficiency and 1.85TOPS/mm area efficiency, which are 2.79× and 2.2× better than the state-of-the-art CNNSR accelerator, respectively.
- 10 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (3) , 1179-1183
A Weight-Reload-Eliminated Compute-in-Memory Accelerator for 60 fps 4K Super-Resolution
Convolutional neural networks (CNNs) have significantly enhanced the quality of image super-resolution (SR). However, deploying CNN with deeper layers for real-time 60fps and higher resolution SR, e.g. 4K, brings higher performance requirement, incurring huge area and power overhead as well. In this paper, a high performance compute-in-memory (CIM) accelerator with high energy and area efficiency is proposed for deep-layer CNN-based SR (CNNSR). It adopts alternate time division serial (ATS) dataflow to guarantee the high performance, which eliminates frequent and time-consuming weight reload operation of CIM macros. Moreover, considering the extremely high ratio of computation quantity to weight amount in the deep-layer CNNSR scenario, an energy and area efficiency enhanced digital CIM macro with high computation-to-memory ratio (CMR) is proposed. The proposed accelerator achieves 33.3TOPS peak performance, supporting 4K SR at no less than 60fps. It is evaluated under TSMC 28nm process, achieving 13.96TOPS/W energy efficiency and 1.85TOPS/mm area efficiency, which are 2.79× and 2.2× better than the state-of-the-art CNNSR accelerator, respectively.
- 11 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (3) , 1124-1128
Pareto Frequency-Aware Power Side-Channel Countermeasure Exploration on CNN Systolic Array
Neural Network (NN) accelerators are widely used in many scenarios, including computer vision, speech recognition, automatic driving, etc. The trained weights of NNs are considered confidential properties due to their commercial value, thus becoming the potential targets for hardware reverse-engineering intruders. In this paper, we propose a low-cost, high-security, easy-to-deploy countermeasure for convolutional neural network (CNN) systolic array against weight reverse-engineering through power side channels. This countermeasure utilizes the clock frequency-switching and phase-shifting techniques to mitigate the power side channels. We further make a design space exploration to trade-off between Side-Channel Analysis (SCA) security and performance by leveraging the non-ideality of CMOS device. The experimental results show that the proposed countermeasure achieves an 81.6% absolute correlation reduction, thus incurring more than 100× power trace consumption (more secure) while maintaining the MAC (multiply-and-accumulate) blocks work at the Pareto-optimal frequencies (Pareto faster).
- 11 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (3) , 1124-1128
Pareto Frequency-Aware Power Side-Channel Countermeasure Exploration on CNN Systolic Array
Neural Network (NN) accelerators are widely used in many scenarios, including computer vision, speech recognition, automatic driving, etc. The trained weights of NNs are considered confidential properties due to their commercial value, thus becoming the potential targets for hardware reverse-engineering intruders. In this paper, we propose a low-cost, high-security, easy-to-deploy countermeasure for convolutional neural network (CNN) systolic array against weight reverse-engineering through power side channels. This countermeasure utilizes the clock frequency-switching and phase-shifting techniques to mitigate the power side channels. We further make a design space exploration to trade-off between Side-Channel Analysis (SCA) security and performance by leveraging the non-ideality of CMOS device. The experimental results show that the proposed countermeasure achieves an 81.6% absolute correlation reduction, thus incurring more than 100× power trace consumption (more secure) while maintaining the MAC (multiply-and-accumulate) blocks work at the Pareto-optimal frequencies (Pareto faster).
- 27 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (2) , 416-420
An RRAM-Based Digital Computing-in-Memory Macro With Dynamic Voltage Sense Amplifier and Sparse-Aware Approximate Adder Tree
RRAM is a promising candidate to implement large-capacity in-memory computing on edge AI devices due to its high density. However, the efficiency and accuracy of RRAM-based computing-in-memory (CIM) works are limited by large accumulation currents and device variations. The SRAM-based digital CIM achieves superior performance and efficiency while the adder tree dominates the area. In this paper, a digital RRAM CIM macro is proposed to achieve a better trade-off between accuracy, energy, and performance by three techniques. First, a dynamic voltage sense amplifier is designed to reduce >90% read currents of low resistance state (LRS) cell. Second, an OR gate approximate adder tree is proposed to reduce the area of the adder tree by 40%. Third, a sparse-aware finetuning algorithm is proposed to reduce the accuracy loss of approximate arithmetic to 0.4% on Cifar-10 dataset. The proposed design achieves 1966TOPS/W energy efficiency and 51.2GOPS/Kb normalized throughput at 1-bit precision which is 1.2× and 1.8× higher than previous RRAM-based designs. This work demonstrates the advantage of digital CIM using RRAM devices.
- 27 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 70 (2) , 416-420
An RRAM-Based Digital Computing-in-Memory Macro With Dynamic Voltage Sense Amplifier and Sparse-Aware Approximate Adder Tree
RRAM is a promising candidate to implement large-capacity in-memory computing on edge AI devices due to its high density. However, the efficiency and accuracy of RRAM-based computing-in-memory (CIM) works are limited by large accumulation currents and device variations. The SRAM-based digital CIM achieves superior performance and efficiency while the adder tree dominates the area. In this paper, a digital RRAM CIM macro is proposed to achieve a better trade-off between accuracy, energy, and performance by three techniques. First, a dynamic voltage sense amplifier is designed to reduce >90% read currents of low resistance state (LRS) cell. Second, an OR gate approximate adder tree is proposed to reduce the area of the adder tree by 40%. Third, a sparse-aware finetuning algorithm is proposed to reduce the accuracy loss of approximate arithmetic to 0.4% on Cifar-10 dataset. The proposed design achieves 1966TOPS/W energy efficiency and 51.2GOPS/Kb normalized throughput at 1-bit precision which is 1.2× and 1.8× higher than previous RRAM-based designs. This work demonstrates the advantage of digital CIM using RRAM devices.
- 29 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 69 (7) , 3144-3148
Bit-Aware Fault-Tolerant Hybrid Retraining and Remapping Schemes for RRAM-Based Computing-in-Memory Systems
Non-volatile computing-in-memory (nvCIM) has become one of the most efficient methods to deal with increasingly complicated neural networks compared to the traditional von Neumann architecture. However, due to the immature fabrication issues, the yield of resistive random access memory (RRAM) cells is limited, which causes the Stuck-At-Faults (SAFs) and decrease of network classification accuracy. In this brief, the bit-aware fault-tolerant hybrid retraining and remapping schemes are proposed to restore the network classification accuracy caused by SAFs. The low weight bits are retrained with the fault mask and the high weight bits are remapped with column swap and zero correction methods. For those bits whose values are still wrong after remapping, the fault index is used to record and calibrate. Experimental results show that the classification accuracy of MNIST, CIFAR-10, and CIFAR-100 is restored from 12.54% to 98.13%, from 10.29% to 90.60%, and from 0.94% to 74.30% with only 10.3% energy consumption and 5.88% area overhead, respectively.
- 29 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 69 (7) , 3144-3148
Bit-Aware Fault-Tolerant Hybrid Retraining and Remapping Schemes for RRAM-Based Computing-in-Memory Systems
Non-volatile computing-in-memory (nvCIM) has become one of the most efficient methods to deal with increasingly complicated neural networks compared to the traditional von Neumann architecture. However, due to the immature fabrication issues, the yield of resistive random access memory (RRAM) cells is limited, which causes the Stuck-At-Faults (SAFs) and decrease of network classification accuracy. In this brief, the bit-aware fault-tolerant hybrid retraining and remapping schemes are proposed to restore the network classification accuracy caused by SAFs. The low weight bits are retrained with the fault mask and the high weight bits are remapped with column swap and zero correction methods. For those bits whose values are still wrong after remapping, the fault index is used to record and calibrate. Experimental results show that the classification accuracy of MNIST, CIFAR-10, and CIFAR-100 is restored from 12.54% to 98.13%, from 10.29% to 90.60%, and from 0.94% to 74.30% with only 10.3% energy consumption and 5.88% area overhead, respectively.
- 6 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 41 (11) , 5043-5056
PACA: A Pattern Pruning Algorithm and Channel-Fused High PE Utilization Accelerator for CNNs
In recent years, convolutional neural networks (CNNs) have achieved significant advancements in various fields. However, the computation and storage overheads of CNNs are overwhelming for Internet-of-Things devices. Both network pruning algorithms and hardware accelerators have been introduced to empower CNN inference at the edge. Network pruning algorithms reduce the size and computational cost of CNNs by regularizing unimportant weights to zeros. However, existing works lack intrakernel structured types to tradeoff between sparsity and hardware efficiency, and the index storage for irregularly pruned networks is significant. Hardware accelerators leverage the sparsity of pruned CNNs to improve energy efficiency. However, their process element (PE) utilization rate is low because of uneven sparsity among input convolutional kernels. To overcome these problems, we propose PACA: a Pattern pruning Algorithm and Channel-fused high PE utilization Accelerator for CNNs. It includes three parts: a pattern pruning algorithm to explore the intrakernel sparsity type and reduce the index storage, a channel-fused hardware architecture to reduce the PEs’ idle rate and improve the performance, and a heuristic and taboo search-based smart fusion scheduler to analyze the idle PE problem and schedule the channel fusion in hardware. To demonstrate the effectiveness of PACA, we have implemented the software parts by Python and the hardware architecture by RTL codes. Experimental results on various datasets show that compared with an existing work, PACA can reduce the index storage overhead by$3.47\times $ –$5.63\times $ with 3.85–9.12 average patterns, and it can improve the hardware performance by$2.01\times $ –$5.53\times $ because of PEs’ idle rate reduction.- 6 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 41 (11) , 5043-5056
PACA: A Pattern Pruning Algorithm and Channel-Fused High PE Utilization Accelerator for CNNs
In recent years, convolutional neural networks (CNNs) have achieved significant advancements in various fields. However, the computation and storage overheads of CNNs are overwhelming for Internet-of-Things devices. Both network pruning algorithms and hardware accelerators have been introduced to empower CNN inference at the edge. Network pruning algorithms reduce the size and computational cost of CNNs by regularizing unimportant weights to zeros. However, existing works lack intrakernel structured types to tradeoff between sparsity and hardware efficiency, and the index storage for irregularly pruned networks is significant. Hardware accelerators leverage the sparsity of pruned CNNs to improve energy efficiency. However, their process element (PE) utilization rate is low because of uneven sparsity among input convolutional kernels. To overcome these problems, we propose PACA: a Pattern pruning Algorithm and Channel-fused high PE utilization Accelerator for CNNs. It includes three parts: a pattern pruning algorithm to explore the intrakernel sparsity type and reduce the index storage, a channel-fused hardware architecture to reduce the PEs’ idle rate and improve the performance, and a heuristic and taboo search-based smart fusion scheduler to analyze the idle PE problem and schedule the channel fusion in hardware. To demonstrate the effectiveness of PACA, we have implemented the software parts by Python and the hardware architecture by RTL codes. Experimental results on various datasets show that compared with an existing work, PACA can reduce the index storage overhead by$3.47\times $ –$5.63\times $ with 3.85–9.12 average patterns, and it can improve the hardware performance by$2.01\times $ –$5.53\times $ because of PEs’ idle rate reduction.- 1 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 57 (8) , 2574-2585
A 65-nm Energy-Efficient Interframe Data Reuse Neural Network Accelerator for Video Applications
An energy-efficient convolutional neural network (CNN) accelerator is proposed for the video application. Previous works exploited the sparsity of differential (Diff) frame activation, but the improvement is limited as many Diff-frame data is small but non-zero. Processing of irregular sparse data also leads to low hardware utilization. To solve these problems, two key innovations are proposed in this article. First, we implement a hybrid-precision inter-frame-reuse architecture which takes advantage of both low bit-width and high sparsity of Diff-frame data. This technology can accelerate 3.2$\times $ inference speed with no accuracy loss. Second, we design a conv-pattern-aware processing array that achieves the 2.48$\times $ –14.2$\times $ PE utilization rate to process sparse data for different convolution kernels. The accelerator chip was implemented in 65-nm CMOS technology. To the best of our knowledge, it is the first silicon-proven CNN accelerator that supports inter-frame data reuse. Attributed to the inter-frame similarity, this video CNN accelerator reaches the minimum energy consumption of 24.7$\mu \text{J}$ /frame in the MobileNet-slim model, which is 76.3% less than the baseline.- 1 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 57 (8) , 2574-2585
A 65-nm Energy-Efficient Interframe Data Reuse Neural Network Accelerator for Video Applications
An energy-efficient convolutional neural network (CNN) accelerator is proposed for the video application. Previous works exploited the sparsity of differential (Diff) frame activation, but the improvement is limited as many Diff-frame data is small but non-zero. Processing of irregular sparse data also leads to low hardware utilization. To solve these problems, two key innovations are proposed in this article. First, we implement a hybrid-precision inter-frame-reuse architecture which takes advantage of both low bit-width and high sparsity of Diff-frame data. This technology can accelerate 3.2$\times $ inference speed with no accuracy loss. Second, we design a conv-pattern-aware processing array that achieves the 2.48$\times $ –14.2$\times $ PE utilization rate to process sparse data for different convolution kernels. The accelerator chip was implemented in 65-nm CMOS technology. To the best of our knowledge, it is the first silicon-proven CNN accelerator that supports inter-frame data reuse. Attributed to the inter-frame similarity, this video CNN accelerator reaches the minimum energy consumption of 24.7$\mu \text{J}$ /frame in the MobileNet-slim model, which is 76.3% less than the baseline.