H. Ekin Sumbul

Recent advancements in large language models (LLMs) have shown promise for human-like machine intelligence. Enabling these models in wearable augmented reality (AR) devices can elevate the user experiences with reduced latency and improved privacy, seamlessly blending advanced machine intelligence (artificial general intelligence) into daily life. However, deploying these models on wearable AR devices is hindered by the limited memory capacity and high energy/latency costs associated with off-chip memory access. This article assesses the recent memory technology innovations, including 3D-stacked memories, computing-in-memory (CIM), and processing-in-memory (PIM), and it explores how to utilize their potential to enable LLMs on AR SoCs. Using roofline modeling, we showcase the importance of cooptimizing system architecture with model characteristics to meet power/performance targets.

Real-time Codec Avatars, which employ deep generative models for 3D reconstruction of human features, are crucial for immersive telepresence in AR/VR environments. However, deploying these avatars in real-time on AR/VR headsets is challenging due to the inability of existing devices to achieve satisfying performance within stringent hardware resource constraints. To address these challenges, we introduce Re-CATA, an innovative full-stack and flexible Codec Avatar accelerator design framework. Re-CATA is designed to deliver real-time throughput (greater than 120 FPS) for the complete Codec Avatar processing pipeline within an edge-level power budget of 5 W under FPGA prototyping. Our approach begins by abstracting the operation mapping and scheduling challenges inherent in Codec Avatars, which require both centralized and distributed processing to handle dynamically changing workloads. We propose a novel hardware resource and workload partitioning scheme optimized for these fluctuating demands. To complement this, we introduce an agile runtime scheduling system for efficient workload reallocation among computing units as needed, recognizing the limitations of static partitioning in rapidly evolving workload scenarios. Furthermore, our micro-architecture design incorporates unified computing modules and efficient hardware peripherals, enabling seamless workload balancing across the Codec Avatar processing pipeline. We evaluate the Re-CATA accelerators via on-board FPGA prototyping, comparing them to various baselines, including commercial AR/VR SoCs and academic accelerators. This evaluation demonstrates a maximum speedup of up to 5.95× under similar settings.

Compute-in-memory (CIM) has emerged as an effective technique to address memory access bottleneck for Deep Neural Networks (DNN). Augmented/Virtual Reality (AR/VR) devices require running high-performance DNN inference at tight power budgets, making CIMs ideal candidates for low-power on-device acceleration. While high energy-efficiencies have been reported at the CIM macro levels, system-on-chip (SoC) level energy-efficiency of CIM based accelerators have been underexplored for realistic system integration considerations. In this work, we present a CIM-accelerator architecture comprised of 16 row-pipelined fully-digital CIM macros, and provide a comprehensive analysis of CIM energy-efficiency benefits at the SoC-level targeting representative AR/VR workloads. Two key results are: (1) Realistic SoC-level CIM-accelerator energy-efficiency may be ∼50% lower than the CIM macro-level peak energy-efficiency when additional logic, memory hierarchies, and NN-dependent sub-optimal compute utilization are considered. (2) The CIM-accelerator still demonstrates up to ∼2.1× energy savings at the SoC-level compared to a systolic-array-based DNN accelerator at iso-peak throughput.

Augmented Reality / Virtual Reality (AR/VR) devices aim to connect people in the Metaverse with photorealistic virtual avatars, referred to as “Codec Avatars”. Delivering a high visual performance for Codec Avatar workloads, however, is a challenging task for mobile SoCs as AR/VR devices have limited power and form factor constraints. On-device, local, near-sensor processing provides the best system-level energy-efficiency and enables strong security and privacy features in the long run. In this work, we present a custom-built, prototype small-scale mobile SoC that achieves energy-efficient performance for running eye gaze extraction of the Codec Avatar model. The test-chip, fabricated in 7nm technology node, features a Neural Network (NN) accelerator consisting of a 1024 Multiply-Accumulate (MAC) array, 2MB on-chip SRAM, and a 32bit RISC-V CPU. The featured test-chip is integrated on a prototype mobile VR headset to run the Codec Avatar application. This work aims to show the full stack design considerations of system-level integration, hardware-aware model customization, and circuit-level acceleration to meet the challenging mobile AR/VR SoC specifications for a Codec Avatar demonstration. By re-architecting the Convolutional NN (CNN) based eye gaze extraction model and tailoring it for the hardware, the entire model fits on the chip to mitigate system-level energy and latency cost of off-chip memory accesses. By efficiently accelerating the convolution operation at the circuit-level, the presented prototype SoC achieves 30 frames per second performance with low-power consumption at low form factors. With the full-stack design considerations presented in this work, the featured test-chip consumes 22.7mW power to run inference on the entire CNN model in 16.5ms from input to output for a single sensor image. As a result, the test-chip achieves 375 µJ/frame/eye energy-efficiency within a 2.56 mm ² silicon area.

A binary neural network (BNN) chip explores the limits of energy efficiency and computational density for an all-digital deep neural network (DNN) inference accelerator. The chip intersperses data storage and computation using computation near memory (CNM) to reduce interconnect and data movement costs. It performs wide inner product operations to leverage parallelism inherent in DNN computations. The BNN chip leverages lightweight pipelining at a near-threshold voltage (NTV) to reduce the overhead of sequential elements. It employs optimized data access patterns to reduce memory accesses for convolutional operation with pooling layers. The combination of these techniques enables the BNN chip to achieve a peak energy efficiency of 617 TOPS/W. The digital BNN chip approaches the energy efficiency of analog in-memory techniques while also ensuring deterministic, scalable, and bit-accuracy operation. Moreover, the all-digital design leverages process scaling and does not require additional memory transistors or passive devices to attain a peak compute density of 418 TOPS/mm² and a memory density of 414 KB/mm². The binary design is extended to enable bit-serial integer precision operation with a reconfigurable 1-b multiplication circuit and element-wise partial sum shift and accumulate. This technique allows for fine-grain mixed precision and retains energy efficiency by exploiting parallelism inherent in DNNs. The bit-serial binary operation allows for bit-accurate operation and high DNN accuracy that multibit analog compute-in-memory designs struggle to attain. It provides favorable energy tradeoffs compared with small-integer digital DNN accelerators.

A 10nm DNN inference accelerator compresses model size with tabulation hash-based fine-grained weight sharing and increases 8b-compute density by 3.4× to 1.6 TOPS/mm2. The compressed model DNN implements lightweight hashing circuits to compress fully connected and recurrent neural networks. Optimized shared weight address generation reduces MUX tree area overhead by 40%. Runtime hash table generation and weight mapping circuits enable a peak energy efficiency of 9.0TOPS/W at 450mV, 25 ∘C. A 128×-compressed 3-layer LSTM classifies TIMIT phonemes with 85.6% accuracy for a total energy of 14μJ/classification, with <0.5% degradation in accuracy over an uncompressed network.

A 10-nm compute-near-memory (CNM) accelerator augments SRAM with multiply accumulate (MAC) units to reduce interconnect energy and achieve 2.9 8b-TOPS/W for matrix–vector computation. The CNM provides high memory bandwidth by accessing SRAM subarrays to enable low-latency, real-time inference in fully connected and recurrent neural networks with small mini-batch sizes. For workloads with greater arithmetic intensity, such as large-batch convolutional neural networks, the CNM reconfigures into a 2-D systolic array to amortize memory access energy over a greater number of computations. Variable-precision 8b/4b/2b/1b MACs increase throughput by up to $8\times $ for binary operations at 33.0 1b-TOPS/W.

A reconfigurable 4096-neuron, 1M-synapse chip in 10-nm FinFET CMOS is developed to accelerate inference and learning for many classes of spiking neural networks (SNNs). The SNN features digital circuits for leaky integrate and fire neuron models, on-chip spike-timing-dependent plasticity (STDP) learning, and high-fan-out multicast spike communication. Structured fine-grained weight sparsity reduces synapse memory by up to 16x with less than 2% overhead for storing connections. Approximate computing co-optimizes the dropping flow control and benefits from algorithmic noise to process spatiotemporal spike patterns with up to 9.4x lower energy. The SNN achieves a peak throughput of 25.2 GSOP/s at 0.9 V, peak energy efficiency of 3.8 pJ/SOP at 525 mV, and 2.3-μW/neuron operation at 450 mV. On-chip unsupervised STDP trains a spiking restricted Boltzmann machine to de-noise Modified National Institute of Standards and Technology (MNIST) digits and to reconstruct natural scene images with RMSE of 0.036. Near-threshold operation, in conjunction with temporal and spatial sparsity, reduces energy by 17.4x to 1.0-μJ/classification in a 236 x 20 feed-forward network that is trained to classify MNIST digits using supervised STDP. A binary-activation multilayer perceptron with 50% sparse weights is trained offline with error backpropagation to classify MNIST digits with 97.9% accuracy at 1.7-μJ/classification.