Doo Seok Jeong

Input sparsity‐aware multiply‐accumulate computing is a promising solutionto reduce redundant operations and improve energy efficiency in deep neural network accelerators. However, analog computing‐in‐memory (CIM) architectures encounter challenges in exploiting sparsity due to the fixed resolution of analog‐to‐digital converters (ADCs), resulting in unnecessary latency and power consumption. This article presents an input sparsity‐aware CIM macro with novel bidirectional conversion‐skippable ADCs (BCS‐ADCs), which enables both input block‐ and element‐level conversion skipping. The proposed BCS‐ADC can skip conversion not only on the most significant bit side of multiply‐accumulate values based on input element‐level sparsity but also on the least significant bit side (equivalent to rounding down). Block‐level skipping is implemented using a population counter in the input activation driver. When designed in 45 nm CMOS at 100 MHz, the proposed CIM macro achieves up to 1.95 speedup and 2.44 energy efficiency compared to a nonskipping baseline. Compared to state‐of‐the‐art ADCs, the BCS‐ADC reduces power by up to 90.82 and improves conversion latency and energy efficiency by 14.82 and 21.02, respectively. The BCS‐ADC is designed based on the conventional successive approximation ADC with a few additional logic elements that occupy only 0.28 of the ADC area.

Computing-In-Memory (CIM) offers a potential solution to the memory wall issue and can achieve high energy efficiency by minimizing data movement, making it a promising architecture for edge AI devices. Lightweight models like MobileNet and EfficientNet, which utilize depthwise convolution for feature extraction, have been developed for these devices. However, CIM macros often face challenges in accelerating depthwise convolution, including underutilization of CIM memory and heavy buffer traffic. The latter, in particular, has been overlooked despite its significant impact on latency and energy consumption. To address this, we introduce a novel CIM dataflow that significantly reduces buffer traffic by maximizing data reuse and improving memory utilization during depthwise convolution. The proposed dataflow is grounded in solid theoretical principles, fully demonstrated in this paper. When applied to MobileNet and EfficientNet models, our dataflow reduces buffer traffic by 77.4-87.0%, leading to a total reduction in data traffic energy and latency by 10.1-17.9% and 15.6-27.8%, respectively, compared to the baseline (conventional weight-stationary dataflow).

We experimentally demonstrate the concept of combination-encoding content-addressable memory (CECAM) that offers much higher content density than any other content-addressable memory devices proposed to date. In this work, CECAM was fabricated and validated with a hafnium-zirconium oxide (HZO) ferroelectric tunnel junction (FTJ) crossbar array. The new CAM structure, which utilizes nonvolatile memory devices, offers numerous advantages including low-current operation (FTJ), standby power reduction (ferroelectric HZO), and increased content density. Multibit data are encoded and stored in multi-switch CECAM. Perfect-match searching in CECAM with a reasonable match current (lower than nA) for different sizes of CECAM has been validated from a novel CAM device. We demonstrate N-CECAM (with keys encoded into 2N-long binary arrays) for N = 3 (using 6 FTJs) and 4 (using 8 FTJs), leading to content densities of 0.667 and 0.75 bits per switch, which highlight 33% and 50% increase in content density compared to that of the conventional TCAM (0.5 bits per switch)

Deep learning (DL) has proven to be one of the most pivotal components of machine learning given its notable performance in a variety of application domains. Neural networks (NNs) for DL are tailored to specific application domains by varying in their topology and activation nodes. Nevertheless, the major operation type (with the largest computational complexity) is commonly multiply‐accumulate operation irrespective of their topology. Recent trends in DL highlight the evolution of NNs such that they become deeper and larger, and thus their prohibitive computational complexity. To cope with the consequent prohibitive latency for computation, 1) general‐purpose hardware, e.g., central processing units and graphics processing units, has been redesigned, and 2) various DL accelerators have been newly introduced, e.g., neural processing units, and computing‐in‐memory units for deep NN‐based DL, and neuromorphic processors for spiking NN‐based DL. In this review, these accelerators and their pros and cons are overviewed with particular focus on their performance and memory bandwidth.

Memristor-integrated passive crossbar arrays (CAs) could potentially accelerate neural network (NN) computations, but studies on these devices are limited to software-based simulations owing to their poor reliability. Herein, we propose a self-rectifying memristor-based 1 kb CA as a hardware accelerator for NN computations. We conducted fully hardware-based single-layer NN classification tasks involving the Modified National Institute of Standards and Technology database using the developed passive CA, and achieved 100% classification accuracy for 1500 test sets. We also investigated the influences of the defect-tolerance capability of the CA, impact of the conductance range of the integrated memristors, and presence or absence of selection functionality in the integrated memristors on the image classification tasks. We offer valuable insights into the behavior and performance of CA devices under various conditions and provide evidence of the practicality of memristor-integrated passive CAs as hardware accelerators for NN applications.

Computing-in-memory (CIM) macros aiming at accelerating deep learning operations at low power need activation function (AF) units on the same die to reduce their host-dependency. Versatile CIM macros need to include reconfigurable AF units at high precision and high efficiency in hardware usage. To this end, we propose the gradient-error-based approximation (GEBA) of AFs, which approximates various types of AFs in discrete input domains at high precision. GEBA reduces the approximation error by ca. 49.7%, 67.3%, 81.4%, 60.1% (for sigmoid, tanh, GELU, swish in FP32), compared with the uniform input-based approximation using the same memory as GEBA.

Spiking Neural Networks (SNNs) support sparse event-based data processing at high power efficiency when implemented in event-based neuromorphic processors. However, the limited on-chip memory capacity of neuromorphic processors strictly delimits the depth and width of SNNs implemented. A direct solution is the use of quantized SNNs (QSNNs) in place of SNNs with FP32 weights. To this end, we propose a method to quantize the weights using constrained backpropagation (CBP) with the Lagrangian function (conventional loss function plus well-defined weight-constraint functions) as an objective function. This work utilizes CBP as a post-training algorithm for deep SNNs pre-trained using various state-of-the-art methods including direct training (TSSL-BP, STBP, and surrogate gradient) and DNN-to-SNN conversion (SNN-Calibration), validating CBP as a general framework for QSNNs. CBP-QSNNs highlight their high accuracy insomuch as the degradation of accuracy on CIFAR-10, DVS128 Gesture, and CIFAR10-DVS in the worst case is less than 1%. Particularly, CBP-QSNNs for SNN-Calibration-pretrained SNNs on CIFAR-100 highlight an unexpected large increase in accuracy by 3.72% while using small weight-memory (3.5% of the FP32 case).

The brain is the perfect place to look for inspiration to develop more efficient neural networks. The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. This article serves as a tutorial and perspective showing how to apply the lessons learned from several decades of research in deep learning, gradient descent, backpropagation, and neuroscience to biologically plausible spiking neural networks (SNNs). We also explore the delicate interplay between encoding data as spikes and the learning process; the challenges and solutions of applying gradient-based learning to SNNs; the subtle link between temporal backpropagation and spike timing-dependent plasticity; and how deep learning might move toward biologically plausible online learning. Some ideas are well accepted and commonly used among the neuromorphic engineering community, while others are presented or justified for the first time here. A series of companion interactive tutorials complementary to this article using our Python package, snnTorch , are also made available: https://snntorch.readthedocs.io/en/latest/tutorials/index.html.